One key way Agile is far superior to waterfall and most other methodologies is the equal emphasis it puts on both product delivery and process/ systems/ organisational improvement. Unlike waterfall, improvement is built directly into the Agile process. But it’s important to distinguish between Agile tasks in which optimisation is the focus, as opposed to tasks where delivery is key. This article explains why getting this wrong is a real risk, and offers a simple model to explain how to get your team to make the right distinctions and optimise the Agile process as a whole.
One of the most appealing features of Agile is the way it positively encourages development teams to constantly improve how they work. But now that Agile is approaching the end of its second decade and is routinely being implemented in teams that perhaps aren’t exactly enthusiasts for Agile or who lack real buy-in to the underlying philosophy, is it clear to developers how they should balance out the competing claims of delivery and optimisation?
In fact for inexperienced teams there are (at least) two ways of misconstruing how Agile should work:
- Inexperienced teams import their old waterfall culture, focusing exclusively on basic development.
- This turns Agile into a grim reworking of the ‘just get the job done and go home’ attitude that pervades poorly managed waterfall projects and programmes.
- At the same time they neglect the opportunities Agile creates for improving every aspect of how the team works – either because they lack the necessary authority or they misunderstand the purpose of basic Agile ceremonies such as Retrospectives.
- Inexperienced teams obsess over the ‘continuous improvement’ dimension of Agile.
- Agile is then reduced to a relentless debate about how anything and everything can be made better all the time, leading to the disruption of any actual delivery.
- The autonomy of the team degenerates into a situation where the basic attitude to the outside world is one of defiance rather than collaboration.
In this article, the problem of steering a true Agile course between these two extremes is analysed in terms of the familiar saying that ‘The best is the enemy of the good’. It argues that, while this saying is true (especially if your focus is timely delivery), for Agile the opposite proposition – that ‘The good is the enemy of the best’– would be equally valid (especially if you regard core engineer values as sacrosanct).
The solution this article then offers is based on ensuring that these two conceptions do not come into conflict. This is achieved not by allowing one or the other to win out or by letting both operate at the same time, but by organising how they interact.
Waterfall: the good, but not the best
It’s a well-known saying: The Best is the enemy of The Good. Which is to say, continually aspiring to deliver The Best (the ideal, perfection, etc.) makes it very difficult to accept that anything less is really good enough – not even provisionally. So you carry on struggling towards perfection, which can quickly come to mean that you deliver nothing at all. So, far from delivering The Best, nothing is delivered at all – except perhaps exhaustion and despondency.
Of course, it’s not always that simple. Many years ago I was running a course at the UK Civil Aviation Authority, all about software quality. The class was going quite well, until on the afternoon of Day 3 someone put his hand up and argued (very firmly) that it was his job to create the very best software he possibly could – regardless of the cost. I replied that making it better than it needed to be or than anyone was asking for would be a waste of resources, and certainly not what his bosses were paying for.
I thought the lesson was simple: decide how good is good enough, and settle for that.
Quite often such an approach makes sense. Certainly it was waterfall’s underlying philosophy. In fact in waterfall development lifecycles, ‘good enough’ was defined very explicitly, in the form of detailed requirements, elaborate architectures and design specifications, carefully scrutinised and test cases, classes and scripts, formal authorisation and acceptance, and so on. Likewise for the delivery process, with its carefully managed scope, schedule, etc. More general concepts such as value for money played a similar role in controlling how we worked – always aiming at The Good, never at The Best. Pity it hardly ever worked…
That software quality class at the Civil Aviation Authority was a long time ago – quite a long time before Agile was even a twinkle in its forefathers’ eyes. But even then, the whole time the two of us were debating this seemingly simple point, the rest of the class remained completely silent – interested, but saying nothing. I suspect this was because more of them agreed with their colleague than with me but, being British, they were too polite to tell me I was talking rubbish. Now, if there’s an industry where quality is absolutely (indeed literally) vital, it’s aviation, so perhaps I should not have been surprised at their scepticism. In fact, I’m not sure that I didn’t agree with him, in my heart of hearts. There’s something very appealing to any decent engineer about the idea of getting a system just right, and not have to be too bothered about trivia like the insane costs they’re running up. There’s a principle at stake. Certainly if you look at code I’ve written for myself, it is quite astonishingly polished.
And this reflects one of the (many) fatal failings of waterfall-style development: that it was good at defining (if not delivering) ‘good enough’ but it seldom seemed to aim any higher. So: is good enough really good enough? And if not, when are we going to strive for The Best? Unlike defining what good meant, defining how we would get better was oddly distant from everyday work.
- On the one hand there were amazingly cumbersome and bureaucratic procedures for ‘change control’.
- On the other, a succession of consultants, management fads, over-promoted tools and (horror of horrors) team-building sessions.
Above all, the developers themselves seldom had any authority to change the methodology, tools, processes, standards, etc. that controlled how they worked. So it’s not really very surprising that the outcome of all these rather artificial-feeling add-ons was that none of them ever seemed to result in feeling we were doing a radically better job.
And then along came Agile…
Well, that was then – quite a long time ago. As I say, before the origins of Agile. Which was also an important moment, because Agile – at last – seemed to offer a way to balance The Best against The Good, while actively pursuing both. In fact from the Agile point of view, thinking that The Best might be the enemy of The Good would create a serious conflict. For if ‘good enough’ really is good enough, how are we to obey another, surely basic rule of Agile development, namely Quality First? In fact, isn’t Agile awash with principles and techniques for making sure we are constantly getting better? Didn’t someone (I think it was Mike Cohn) once say that if you aren’t better next month than you were last month, then you weren’t doing Agile?
It’s one of the most appealing things about Agile, of course – not only does it expect the way we work to improve constantly but it expects us to do the improving. We own the process, and use and change it as we see fit. No more waiting for that edict from on high, for the magical new tool that will fix all our problems, or for the consultants to roll up – it’s we who know what the problem is, and it’s we who solve it. Fantastic!
But if Agile promotes the idea that we should always be the best we can be, doesn’t that also create a conflict?
- On the one hand, there are all the little hiccups in development – mistakes, oversights, confusions, bugs, failures – that certainly should be fixed ASAP.
- Because if you wait, not only will they will take at least as long (and probably longer) to fix as if you fixed them straightaway, but the impact of delay can only be negative. Whether it’s the propagation of new bugs, confused or misinformed decision-making or any one of a hundred other possible side-effects, it’s almost never more sensible to put up with a mistake ‘for now’ than to fix it immediately.
- So yes – aim for The Best right off!
- But on the other hand, what about all the new stories that keep piling up in the backlog?
- They all represent the stakeholders’ idea of ‘The Best’ (i.e., what they really want), so why doesn’t it make sense to continually adjust our work to take in every little change in the backlog? Why don’t we reshuffle the backlog continually – daily, hourly – abandoning less valuable stories halfway through and starting other stories that have suddenly risen to the top of the priority list? In fact, isn’t that a standard reading of Agile: that as stories are re-prioritised, we should shift our focus to the highest priority items? But of course that’s a very different problem. Not only does fixing a mistake almost always pay for itself immediately, but every hour you put off fixing it makes it still more expensive to deal with.
- By contrast, every time one story is stopped so that another story can start is a kind of failure. Because effort has already been sunk into the (partial) development of the old story is effort that hasn’t– and perhaps now won’t – been converted into value for the user. And the longer you wait to restart the deferred story, the longer that useless work-in-progress (WIP) hangs over you. And the more you do this, the more the WIP builds up, and the fundamental efficiency of Agile as a process for delivering value is eroded.
Of course, one should certainly stop work on anything the users say they no longer need, but otherwise a very strong case needs to be made for diverting effort from a story that is already in progress. This is especially true if the story you need to postpone is, say, a spike for other stories or part of a larger epic that the current iteration or release is supposed to deliver in its entirety.
This problem can be summarised as follows: while constantly fixing every defect in the process tends to enhance the team’s overall productivity, constantly adjusting to every change in the backlog tends to reduce it. Or to put the same thing in strictly Agile terms: which takes priority – Quality First or minimising WIP?
The two phases of Agile quality
So what is the underlying problem? Essentially it concerns optimising delivery so that value is delivered with the maximum of efficiency and effectiveness. And perhaps I can solve this puzzle by suggesting that all these problems can be managed by dividing the Agile lifecycle into two phases – one in which The Best is the enemy of The Good, and one in which The Good is the enemy of The Best. By balancing these two phases against one another, maybe the Agile lifecycle as a whole can be optimised to deliver everything.
So what are these two phases?
When The Best is the enemy of The Good
The first is the development process itself. Here the overriding priority is optimising Work in Progress. So, once the team has committed itself to delivering the agreed Iteration Backlog, and then Iteration 0 has expended valuable effort in preparing for those specific stories, almost any diversion or delay from the Iteration Plan increases WIP, and is therefore a waste.
At the same time, as the heart of the development process is developers and users collaborating closely, the real test of quality is simply to ask the user, ‘Is this good enough?’ If they say Yes, then it is; and if No, then it isn’t. There really is no other test – and by the same token, if the user doesn’t want to push quality beyond a certain level, then that’s the level at which development should stop.
Of course, that’s not all there is to quality in the development process.
- How good are users at judging how good software really is? This raises the question of the developer’s responsibility for the quality of the software they write. Fortunately in a mature Agile environment that should not be a problem.
- Every good Agilist is professional, disciplined and expert, so the software should be as good as they can make it, technically speaking.
- More directly, it’s the responsibility of the developer to make sure that the user understands what is possible. After all, a perennial limitation on progress in software development is the user’s understanding of what is possible, and it’s a key part of the developer’s role to make sure that their user understands what is really available.
- But if users have a clear sense of what is possible and they are happy with what the development team has built, then that is all that is needed.
- Refactoring – a classic case of striving for The Best rather than just The Good – is central to Agile’s technical approach, and takes place right in the middle of development.
- But it only takes place as development is finishing, so it does not interrupt development, and so confuse The Best with The Good.
- Also, there is a compelling argument for not refactoring code that is unlikely to be further developed in future – e.g., throw-away webpages, or particularly obscure parts of the system.
- Backlog refinement likewise occurs during the main development cycle, but again does not disrupt it. Often it doesn’t involve the whole team, and takes place offline from the principal development process. So again, it avoids forcing the team to choose between The Best and The Good.
When The Good is the enemy of The Best
As I have already said, one of the key ways in which Agile is different from waterfall is the phases when the team is supposed to treat The Good as the enemy of The Best. This involves two things:
- Taking very clear time out from the main development process.
- Making use of specific tools and techniques for improvement.
Without these, the basic mistake of waterfall development will be repeated:
- On the one hand, The Best and The Good are confused, leading to friction, conflicts, loss of velocity and disrupted cadence.
- On the other, improvement initiatives become much more likely to fail, re-igniting all the frustrations waterfall creates in the minds of capable professionals.
So, which are the Agile tasks in which change, improvement and basically becoming The Best we can be should be prioritised? I have already mentioned how Iteration Planning and Iteration 0 are used to focus and prepare for individual iterations, and how Backlog Refinement and Refactoring optimise the product by taking brief time-outs from the main development process. Here are a few more opportunities for improvement built into the iteration itself.
- Technical Debt
And so on. All of these activities take place during the iteration, of course. And sometimes you really do have to break into the development-and-delivery phase and insist that some improvements really do have to be done, even if the burn-down chart has to be subjected to some major surgery!
What has all this to do with methodology?
One last point: if the conflict between The Good and The Best is to be managed, you need to include at least a minimum of formal organisation that separates the two out. Hence the importance of your team and your methodology of:
- Distinguishing clearly between:
- The core tasks of development and delivery; and
- The equally important tasks for responding to opportunity and crisis.
- Organising these tasks so that The Good and The Best are never prioritised at the same time.
In short, although Agile can be thought of as a process of ‘continuous everything’, that doesn’t mean ‘continuous everything all at the same time’. Strike a balance between ensuring that your team doesn’t wait when delay would be costly and making the overall process orderly enough to ensure that their work does not descend into a succession of arguments about what they should be doing. Otherwise it would be hard to ensure that ‘continuous everything’ didn’t rapidly collapse into ‘continuous nothing’!
So how do you make sure that these two phases are clearly identified and implemented, and so avoid the dithering and disputes that can so easily arise from trying to balance The Best against The Good?
Well, that of course is half the point of having a robust, well-constructed methodology. Although it should never be adhered to slavishly, a good methodology will always define where the emphasis lies, and always guide you through the specific risks and issues that are present at each point. That’s why we have built Agile201 as it is.
So: The Best really is the enemy of The Good, and The Good really is the enemy of The Best. The trick is to know how you get the best of both worlds. And to achieve that, you need to understand both the tensions between these two priorities and how to shift, quickly and seamlessly, from one phase to the other.