This post focuses on managing the volume of change in an Agile scenario.
For more information on the components and basics of a scrum-based agile approach, see “Sliger, M. (2011). Agile project management with Scrum. Paper presented at PMI® Global Congress 2011—North America, Dallas, TX. Newtown Square, PA: Project Management Institute.“
What is Agile?
“…requirements and solutions evolve through the collaborative effort of self-organizing and cross-functional teams and their customer(s)/end user(s). Agile advocates adaptive planning, evolutionary development, early delivery, and continual improvement, and it encourages rapid and flexible response to change.”
Sounds good right? Lots of good words like collaboration, self, cross-functional, customers, adaptation, evolution, early, improve, rapid, flexible, etc. However, the main questions everyone has are;
- How exactly do I get all that good stuff?
- How much better/cheaper will this approach be?
- If it’s so good, why isn’t everyone using this method for everything?
Before we look at what “Agile” is, however, let’s look at where agile came from and what it tries NOT to be.
Specifically related to IT, iterative and incremental development methods can be traced back as early as 1957, with evolutionary project management and adaptive software development emerging in the early 1970s. During the 1990s, a number of lightweight software development methods evolved in reaction to the prevailing heavyweight methods that critics described as overly regulated, overly planned, and micro-managed.
In 2011, the Agile Alliance created the Guide to Agile Practices (renamed the Agile Glossary in 2016), an evolving open-source compendium of the working definitions of agile practices, terms, and elements, along with interpretations and experience guidelines from the worldwide community of agile practitioners.
Essentially providing an understanding that change is inevitable and rather than limit that change to reach a goal, we must provide a format and forum to encourage that change and to manifest and return that evolution into our product. This, in turn, reduces the time to reach an “MVP” (Minimum Viable Product) that is close to what the customers want “on the first go”.
Some of the basic principles of Agile are actually based on an aversion to rigidity;
- Tools and processes are important, but it is more important to have competent people working together effectively.
- Good documentation is useful in helping people to understand how the software is built and how to use it, but the main point of development is to create software, not documentation.
- A contract is important but is no substitute for working closely with customers to discover what they need.
- A project plan is important, but it must not be too rigid to accommodate changes in technology or the environment, stakeholders’ priorities, and people’s understanding of the problem and its solution.
So the founding principles are essentially the antithesis of the methods that had gone before them. This radical change, to almost the opposite of what was the “standard”, has led to a large amount of confusion. The Agile method did not grow up with university graduates or the military like the “Waterfall” method did, it hit the Software industry like a bomb. Suddenly everyone wanted to be “Agile” because we perceive it as being cheaper, faster, easier, etc. and there are many organizations and use-cases where using this Agile method is very beneficial and there are others where this is most certainly not the case. Thus caution is advised.
If you ask anyone “can you do the project the traditional way” everyone says “yes”. If you ask again, can you do the project in an “Agile” way? There is a lot of hesitation because moving from a fixed approach to a flexible approach introduces unplanned and unknown variables. And variables, by their nature, are hard to quantify and that means risk. Essentially you are now talking about moving from a linear approach, where the variables are limited and the time frame for variables to appear is limited, to an iteratively variable approach, where variables are encouraged to appear many times and your capability to respond to change becomes all important.
Agile methods are not limited to software however, the same principles can be and are applied to every industry. In fact, most people are not aware, but outside software development, the main approach IS Agile.
Let’s take an example that we are all familiar with. The train ride home.
Most of us take the same line, the same carriage, the same time, every weekday. Sometimes we even see the same people in the same seat. Now, what happens when your train is cancelled due to an accident or maintenance, your brain starts working on an “Agile” approach to getting home, at the same time you begin juggling your next activities, calling to cancel the dentist appointment, finding the best alternate route, re-scheduling dinner, etc.. A sudden and massive increase in variables, all processed quickly with mitigation response, timeline adjustment, resource re-tasking, expectation management etc… you are an Agile Project Manager already!
You can deal with this because you have experience with this scenario, you know what to expect and the number of variables is limited and the risk is low. You are adaptable in this scenario and able to cope with the degree of change required in the amount of time available.
If we, however, take an example at the extreme opposite. You are the only passenger on an aeroplane, the pilot has had a heart attack and died, you need to pilot the plane, there is an operating manual in the cockpit. You have zero experience, you don’t know what to expect, the number of variables is huge and the risk is extreme. Now, the pilot is a person, just like yourself, in theory, you both have all the same basic attributes that can lead you to a successful outcome, and you are now required to be Agile and safely land the plane. What do you do?
In this scenario, you need a method by which you can take all of this incoming change, adapt to it and command the resources needed to lead to a successful outcome.
There are many many methods for managing a project in an “Agile” way, however, I’ve not come across many articles on how to pilot the plane and what variables are important to a safe landing.
When we move this theory into the business world, the number of variables is also huge, the territory is uncharted and sometimes no one has ever dealt with a situation like this before. So what do you do?
You need to think about “controlled evolution”. Re-tasking, re-focusing resources, and to realize and accept that with Agile, the first time you produce your product, it does not need to be perfect. You will work together with your customers, iteratively, until it becomes “acceptable”. From there you can enter a pro-active improvement phase and stop being “reactive”.
Thus the need for solid methods for coping with the unavoidable necessity of change is clearly understood.
Change is both proactive and reactive
Agility is “the capability to deal reactively with unplanned change”.
The uses of an Agile approach extend beyond IT however; according to Jean-Loup Richet (Research Fellow at ESSEC Institute for Strategic Innovation & Services) Agile Methods are not limited to Software Development – “this approach can be leveraged effectively for non-software products and for project management in general, especially in areas of innovation and uncertainty.” The end result is a product or project that best meets current customer needs and is delivered with minimal costs, waste, and time, enabling companies to achieve bottom-line gains earlier than via traditional approaches.
During and after WWII there was neither time nor resources for large and innovative changes in the production of war equipment. The essence of the approach came down to improving the use of the existing workforce and technologies. The “Training Within Industry” (TWI) programs in Japan in 1951 introduced the concept of Kaizen. Titled “Improvement in Four Steps” (Kaizen eno Yon Dankai) introduced kaizen to Japan. A pro-active method for innovation and improvement to get ahead of reactionary driven change and drive the economy forward.
From this we can see that change can be achieved by a) pro-actively seeking out areas that need change and instituting that change as “continuous improvement” (Kaizen) and/or b) reactively having the capability and capacity to deal with change as a “flexible evolution” (Agile).
The forces in play
So why do we need to think about this at all? What is driving us to change and improve all the time? Why are we not just satisfied with the software or the product we have? Why go to all the trouble to create another project management method?
The second law of thermodynamics, in principle, states that a closed system’s disorder cannot be reduced, it can only remain unchanged or increased. A measure of this disorder is entropy. As a system is modified, its disorder, or entropy, always increases.
Essentially any systems, products or services that we build have a use-by-date and we must “improve” them or build new ones. Unless we do so, dissatisfaction increases until the system is useless or change is initiated.
In our IT world, this is known as software entropy. It is these forces of entropy that the Agile method aims to mitigate. Our Agile method gives us a path to reduce the effects of entropy either pro-actively by continuous improvement programs (Kaizen) or reactively by flexible evolution (Agile).
We can replace the word “system” in these laws with any product or service;
- A system must be continually adapted or it becomes progressively less satisfactory to users and departs from its target usefulness.
- As a system evolves, its complexity increases unless work is done to maintain or reduce it.
- System evolution processes are self-regulating and often unplanned leading to further complexity.
- The average effective global activity rate in an evolving system will vary over the product’s lifetime.
- As a system evolves, all associated with it, developers, sales personnel, and users, for example, must maintain mastery of its content and behaviour to achieve satisfactory evolution. Excessive growth diminishes that mastery. Hence the average incremental growth remains invariant as the system evolves.
- The functional content of a system must be continually increased to maintain user satisfaction over its lifetime.
- The quality of a system will appear to be declining unless it is rigorously maintained and adapted to operational environment changes.
- Evolution processes must be treated as multi-dimensional to achieve significant improvement.
With traditional methods of product delivery, our end product is well defined and entropy does not start until the delivery is complete. Often this leads to rapid signs of entropy as the consumer does not feel aligned with the product. There is no emotional attachment to the product or at some point, the builders of the product depart from the “continuously evolving needs” of the consumer.
What the Kaizen and Agile methods do is accept that entropy is inevitable (or already in play) and seeks to delay or smooth the effects of entropy by involving the consumer in the creative process.
This does not actually reduce entropy or complexity per se, however on some level, this is a form of cognitive bias where individuals involved in the creative process create their own “subjective social reality” from their perception of the effect that their input has had on the establishment of the end product. i.e. “Because I made it, therefore it is good”.
This has a real effect of reducing the possibility of rejection or failure of a project deliverable too early and thus a very valid tactic to be aware of in the Agile development process.
Entropy is going to subconsciously influence your consumers, therefore pro-actively engage them in improvement programs and consciously provide for evolution. Thus reducing the effects of entropy.
A truly advanced approach is to involve the consumer in a “continuous creative process” thus increasing their emotional alignment and giving rise to a truly evolving consumer product or service. In this approach, the Agile start to the product or service leads to Kaizen or continuous improvement where the system is able to stay ahead of the inevitable entropy.
Can you think of a product or a service that you used as a child that you still use today? They are very rare, but if you can it will be extremely simple in its operation and thus retain relevance.
If software was capable of envisioning its own entropy, in theory, it would be able to maintain its own simplicity and relevance, essentially becoming continuously relevant and therefore indispensable. A very good state indeed as long as the driver for change is reactionary. We tend to distrust autonomous systems where the motivation for change is unclear. In many end-of-the-world movies, systems become exponentially entropic yet engaged with their consumers. This is unlikely as shown in the movie HER where the operating system’s entropy outgrows its original use case.
Now we understand more about the need for proactive and reactive change, and why it’s necessary; to avoid entropy, we need to be able to apply mitigating principles in any industry, onto any product or service, and get those benefits we imagined at the beginning. We need tools in our bag to know if we are on the right Agile path.
The No.1 complaint I hear from project managers new to Agile is
“I didn’t realize that there would be so many people wanting so much change!”
The No.2 complaint is “this is a never-ending story and going to cost me a fortune!”
Before you begin your Agile project, take an inventory of the factors influencing the change.
- Define your Agile Change Goals
It is easy to become overwhelmed by the flood of “new features” and “product requests” from every direction. Once you unlock the “creative process” it’s hard to put the cork back in. Consumers will be excited to be involved, you need to maintain that excitement, yet also keep your costs in check.
- Define your original goals clearly and communicate them effectively.
- Make sure you have a backlog or “parking lot” for good ideas on the horizon. This will keep your consumers engaged and reduce the symptoms of entropy.
Publish this list on a website for consumers to see the upcoming pipeline.
- Publishing the backlog and engaging the consumers in “feature voting” will also help you smooth the required resource pool and provide a constant stream of deliverable work.
- Define your Agile Rate-of-Change-Flow
- Publish the Change Timeline to help consumers visualize the future change-over-time of the system.
- Give equal priority to items that reduce entropy not only features that increase capability.
- Conduct your Agile Risk Assessment and form Mitigation Plans
- There is a noticeable lack of proactive risk management in Agile projects. Some would argue that Agile practices are by themselves mechanisms for mitigating risk, and to a certain extent, this is true. Techniques such as time-boxed iterations allow many opportunities for course correction, while demos provide regular empirical verification of a team’s output. Retrospectives provide teams with a mechanism for addressing team dysfunction and facilitate continuous improvement, and team ownership of each sprint’s commitment often provides the emotional investment needed to keep a project moving effectively. However these mechanisms, while necessary, are not sufficient to manage risk across even a medium-sized software project. For example, Agile by itself does not provide an effective mechanism for addressing structural problems within a project that are beyond a team’s control. For example, the customer has not provided access to users or important stakeholders. Conduct a traditional risk assessment and continue to review this with every Sprint. As we saw before, the variables continue to change, therefore so do the risks.
- Conduct your Stakeholder Inventory
My favourite author Chris Voss has a perfect quote “There is always a team on the other side. If you are not influencing those behind the table, you are vulnerable.” Make sure you correctly identify the humans AND systems involved in your system and map their perceived AND required impacts on your Agile delivery.
- Always maintain your Traceability of the Changes
Every idea comes from somewhere, every enhancement, every requirement, every bug, every “nice to have” and every single one is hoping to become a change, to add complexity to your system and add to your entropy. As such, someone will challenge every single one. “Why do you want to do that?” “Why does it cost so much?” and my favourite, “Who came up with that!?”
When collecting “stories” in your Agile project, always ALWAYS provide traceability back to the source.”The CEO of our biggest client thinks this is a good idea” is likely to have more weight in the prioritization process than “I was talking with some mates at a bbq on the weekend and …”. Each change should link to a customer story or entropy management item and should also be traced to each existing production item that it impacts. No good changing something if it breaks something already in use.
- Assess the Scope, Volume, and Impact of the proposed Change
Every change item that results from an approved change item solving a customer story will impact the consumers. It may be that the change cannot be absorbed by the consumer at that time, or without some other external adjustment. For example, you may not be able to deploy changes to your accounting system at year end. i.e. the capacity to absorb change is diminished. Your consumers are unable to “respond” to the change.
- Capacity to Change, Response to Change and the Timing of Change
There is only so much change that can be processed in the system before the effects of that change become diminished. This is a function of the systems ability to adjust to the change. The variables are Capacity, Response and Time.
- Change Capacity: On large waterfall projects, it is the expectation that change will come that gives rise to the analysis of the system’s understanding of its Ability to Change or its Capacity for change. The consumer expects a large singular change with a clear delivery time. Other changes are postponed to increase the Change Capacity at that point in time. As an agile project manager, with constant and regular change, one of the first tasks is to assess if the system upon which you are enacting change actually has the freedom, intelligence, experience, or other capabilities to be able to actually accept that change. A change will fail if it is not accepted into the system. Indeed the system will reject changes that could not be absorbed detracting from the usefulness of the system itself.
- Change Response:
Once the change is enacted and the Customer Story is enhanced or simplified by the change, the Customer Story will change. This feedback is called Change Response and must also be a source of further Agile activities. New variables will come out of the Change Response and must be added to the backlog or acted upon immediately if the response is overwhelmingly negative. This gives rise to the concept of the Agile Rejection. If the change is managed well, small incremental changes can be rolled in and out of the system with minimal impact to Change Response.
- Change Period:
The last variable is time. As an Agile project manager, you can set Sprint timelines, BUT timelines must always be appropriate to ensure not only enough time to develop a stable product, but also enough time to validate change capacity while keeping the period short enough to keep a low-cost MVP (minimum viable product). Of all the Agile concepts, the Sprint length is the most complex. Too short and your product will not have sufficient change to be meaningful or quality will suffer. Conversely too long and the change absorption capacity will suffer as your product is too complex to implement quickly. All of these issues impact delivery cost.
- Keep a focus on anti-entropic Simplification
Finally, the topic that we avoid like the plague. Simplification.
There is a perception that “cleaning up code” or “removing unused features” is a waste of money. The is absolutely not the case. It’s equivalent to taking old documents down from your website. Out of date information hurts you even more than new information does. The MIT Centre for Information Systems Research conducted a study on the cost of software complexity.
“In an empirical analysis of sixty-five software maintenance projects in a large IBM COBOL transaction processing environment, the impacts of correctable software complexity upon project costs were estimated. Module size, procedure size, and the use of complex branching were all found to significantly affect software maintenance costs. It was estimated that projects involving maintenance of systems with greater underlying code complexity cost approximately 25% more than otherwise identical projects dealing with less complex code. These costs are estimated to amount to several million dollars at this data site, suggesting that the aggregate cost across the industry may be extremely large.”
SOFTWARE COMPLEXITY AND SOFTWARE MAINTENANCE COSTS
Centre for Information Systems Research
Sloan School of Management
Massachusetts Institute of Technology (MIT)
Remember the introduction; “collaboration, self, cross, customers, adaptation, evolution, early, improve, rapid, flexible, etc”. So how exactly do I get all that good stuff?
We, as professionals, provide services to consumers who are typically not from the same educational discipline, yet we work for in the same system, in the same industry.
By definition, we build, maintain and enhance an “extra-disciplinary, cross-function, service delivery framework”.
We service consumers who want to purchase a very complex product in a simple package to complete their own work and objectives as supporting tools. We service consumers who, as their level of understanding increases, demand commoditization and simplification of use.
We service consumers who believe that they are purchasing a Service, not a Product, after all, they do not wish to own and operate these complex and expensive tools themselves.
Our very complex tools are meant to service individuals and teams from a different discipline who have performance requirements and continuity needs; our tools are important to them in a critical and personal way. Their success depends on our tools.
At the same time as demanding ever increasing levels of performance and relevance for their current consumed services, our customers continue to evolve, to change and to improve, adding complexity, yet demanding a lower operating cost.
Complexity continues to rise, yet cost is continually pressured lower.
These are the characteristics of an evolving consumer service.
We must first recognize this is what we do.
Agile is certainly an approach that can contribute to an evolving consumer service, only if those consumers are emotionally involved in the concept of change. Your job, as an Agile Project Manager, is to ensure that all participants are firstly on-board with this concept or you run the risk of Agile Rejection.
Criticism of Agile
If it’s so good, why isn’t everyone using this method for everything?
Despite some excellent use-cases, agile practices can be inefficient in large organizations and certain types of developments. Many organizations believe that agile software development methodologies are too extreme and adopt a Hybrid approach that mixes elements of agile software development and plan-driven approaches.
A good comparison is building a dam vs releasing a new software version.
You can’t build half the dam and ask the customer “what do you think?” but you can do that with a software version.
The increasing adoption of agile practices has also been criticized as being a management fad that simply describes existing good practices under new jargon, promotes a one size fits all mindset towards development strategies, and wrongly emphasizes method over results.
This article outlines some of the pitfalls and suggests how to select the right approach for your project as well as some best practices –> https://www.pmi.org/learning/library/agile-problems-challenges-failures-5869
Thus, using the tools we discussed, make a thorough assessment first on what method will be applicable to your situation before you begin.
Credits and References
- Lehman, Meir M. (1980). “Programs, Life Cycles, and Laws of Software Evolution”.
Second law of thermodynamics, 1850s, primarily from the works of William Rankine, Rudolf Clausius, and William Thomson (Lord Kelvin).
- Miller, G. J. (2013). Agile problems, challenges, & failures. Paper presented at PMI® Global Congress 2013—North America, New Orleans, LA. Newtown Square, PA: Project Management Institute.
- Sliger, M. (2011). Agile project management with Scrum. Paper presented at PMI® Global Congress 2011—North America, Dallas, TX. Newtown Square, PA: Project Management Institute.
- Agile Government Leaders, Agile Risk Management (ARM) Framework
- Software Complexity and Software Maintenance Costs, Center for information Systems Research, Sloan School of Management, Massachusetts Institute of Technology (MIT)