Most methods for implementing Agile – training, coaching, consultancy, tools – are short-term and the benefits start to fade (or be diverted) as soon as they stop being directly applied. A well-defined methodology, by contrast, provides permanent support for Agile. Not only does a methodology help teams to deliver more effectively but it also provides a single, unified organisation-wide solution, creates a robust foundation for scaling Agile and actively builds the culture to support Agile across the board. So build – or subscribe to – a proper Agile methodology.
Looking around the Agile universe, it is striking how widely established training, coaching, consultants and tools are, whereas many organisations have only a sketchy approach to methods and practices. Often they have no more than a basic diagram of a sprint or an iteration and then simply provide elementary training and leave teams to their own devices. Even with initial support from consultants and good-quality coaches, it is easy to understand why it’s still hard to make Agile work as well as it should. In particular there seems to be a widespread problem with achieving full sustainability.
At Agile201 we tend to think that this is at least partly because, while introducing Agile is quite simple, making it sustainable is not only a completely separate activity but also much harder. Training is good for kicking the process off but seldom enough to really embed a new approach; coaches and consultants eventually go away; and tools, invaluable though they may be, do not really cover Agile as a whole and they are extremely easy to subvert to non-Agile purposes.
Hence, we think, the importance of robust, agreed and widely shared methods and practices: they fill in a number of very wide gaps – in knowledge, skill and experience – that none of the others can. And properly done, they are simple, thorough, flexible, always there when you want them, and (in the case of Agile201) cheap. We also think methodology is likely to be the most fruitful source of new insights into how to make a success of Agile.
What is the problem?
When most organisations decide to adopt Agile, what are the tools they typically choose to help them? Going by the current market, the answer seems to be straightforward enough, and essentially the same as when making most other changes: send the team on a few days training, buy a tool and maybe employ a coach for a while. Bigger companies and government bodies also tend to call in consultants.
These are sensible steps from the point of view of introducing Agile, but it is very debatable as to whether it is the right recipe for making Agile sustainable. And that, of course, is the key. For if Agile is not sustainable, then the initial investment will be quickly lost without the organisation gaining any lasting benefit.
As we often note here at Agile201, Agile always tries to be simple, but that doesn’t make it easy. As countless pieces of research confirm, there’s no shortage of barriers to introducing Agile. But sustainability is an equally critical issue.
Facing the Chasm
The problem of sustainability can be summarised by the so-called ‘Chasm’ model:
As this diagram suggests, before the Chasm – when the transition to Agile begins – everything’s fine. Developers are keen. Quite a few already understand Agile. Often they were the ones who wanted it in the first place. They’re willing to go the extra mile, they aren’t put off by early difficulties and they don’t demand much support.
So the rest will be just as easy. Won’t they? Unfortunately not. As you cross the Chasm, most people are barely aware of Agile and not eager to learn. Those who do know about Agile are pretty cool about the idea of Agile, and some (the ‘laggards’) are likely to be actively hostile. And the group after the Chasm represents probably 85% or so of your organisation.
So they’ll need active support if they’re going to make it to Agile – not only to provide them with basic knowledge and skills but also the resources they need to sustain their commitment to Agile even when they’re under pressure.
But why is this such a problem? Well, in general terms:
- Agile is often initiated very locally (e.g., by an individual Product Owner or delivery team) and the basis for that change is often very informal (“We’ll just do Agile”). The enthusiasm of the leaders is not converted into something that is equally acceptable to the followers.
- In most organisations, change management tends to focus on the start of the change process – getting teams trained and equipped and providing them with basic coaching. They seldom go on to build the core management culture, tools and resources needed to make the change sustainable, or even to recognise that that could be a major problem.
- Such a top-heavy approach is particular risky with Agile, as the scale and complexity of making the transition to Agile is often much greater than managers expect. For most organisations it is a genuine revolution. As well as acquiring the necessary team-level skills and tools, huge areas of corporate organisation and culture also need to be uprooted and transformed.
- Not only is all this intrinsically hard but, for many of those on the receiving end, it is threatening, disruptive and difficult to understand or accept – and also easy to reject, resist and subvert. However well the transition to Agile starts, it demands a lot, lot more to make sure that it ends well too.
Although there’s little hard data on this subject, at Agile201 we’ve had quite a lot of experience of change programmes, and here’s our understanding of how effective the training, coaching, consultancy and tools are likely to be, not just to start with (on the left hand side) but in the long run (towards the right). The picture is in two parts: above the horizontal is the likely impact of each approach on delivery teams, measured over time, and below the line is the likely impact on the organisation as a whole. A more detailed assessment is given in the table after the diagram.
Although the overall message is clear – none of these approaches will make Agile sustainable – this is a rather complicated picture, so here’s a slightly more detailed explanation for the problems with each approach (please add your own):
Here’s a brief example of the sort of issue that arises from a one-track approach to implementing Agile. We are all familiar with what happens to training:
- The team spends a few days in class. Excellent.
- They come out with a very professional-looking slide deck. They can remember quite a lot of what they’ve been taught.
- The course was strong on principles. There were exercises and workshops. But somehow they are still not sure how to use Agile.
- A month after their training, they’ve lost the training decks and/or have made up something else. It’s like they’re playing Chinese whispers with themselves.
- They start to be more interested in the tools (useful) than the real Agile process (essential, critical, fundamental, indispensable).
- After 6 months they’re telling everyone they’re ‘really doing Agile’. They aren’t.
- Agile starts losing credibility across the organisation. People start sending each other links to articles titled ‘Agile is dead’.
- By the end of the year, a third of the original team has left. Their replacements haven’t been trained. But that’s OK – they can pick it up from the others.
- They start to gravitate back to what they used to do. It feels more comfortable.
Yes, it’s a parody, but only just. Something similar could be said about any of the other approaches outlined above. They all start well, and they should certainly be used, but they are not self-sufficient. If not supported by more durable elements, their impact is generally transient – or, in the case of tools, they add very little if you don’t already know how to do Agile. As a result, they are easily diverted from supporting Agile to being used by the wider organisation to force Agile teams into the ‘right’ sort of process and visibility.
Sustaining Agile at the corporate level
Sustainability is hard enough at the delivery team level; at the corporate level it can quickly become nightmarish.
- If not properly prepared, Agile creates an instant problem in many companies. All organisations need visibility and reliability if they are to maintain strategy and direction, and without very careful briefing and preparation the different (and generally superior) visibility and reliability offered by Agile can look like chaos. But if Agile teams are forced back into non-Agile patterns, that’s the end of Agile.
- So a balance needs to be struck between the local independence of Agile teams and the visibility and control needed by higher levels of the organisation. Or rather, Agile needs to be ‘sold’ to the organisation as a whole
- That visibility and reliability – especially when it has to be sustained indefinitely – generally relies on building at least a minimal management framework that is both compatible with Agile and gives the rest of the organisation what it needs.
- This presumes extensive work across the organisation as a whole – just working with delivery teams is never going to be enough.
- So even where delivery teams are happy with Agile and prepared to work on its sustainability, the organisation as a whole may not be content to go along with this.
As the following diagram shows, this means that Agile is frequently under pressure from the very start (or even earlier) from a wide range of powerful stakeholders.
It’s an array of powerful stakeholders that have the ear of even more powerful executives, so any pressure they bring to bear needs to be met with very powerful measures – which again, training, coaching and tools do not provide, and even consultancy finds hard to resist once the consultants have gone.
So again, if real effort is not put into making Agile sustainable, the risk of rapid decay is very high.
So what is the answer? What do you need to do to make Agile sustainable?
Well, there are at least two other factors that need to be introduced if Agile is to be kept going, and which tend to be neglected:
- A community of Agilists who are collectively capable of defining and stabilising what the organisation’s delivery teams mean by Agile.
- A shared methodology that provides an enduring definition of what Agile is and how it is to be used.
What does a real methodology look like?
Although Agile201 provides the environment for a real community of Agilists (at team, company and public levels), we can’t make them happen without a good deal of serious effort from within the organisation. So we’ll focus here on methodology. The following are some key requirements:
- An industrial strength, end-to-end, full-spectrum process model that flexibly defines what Agile is and how it is to be used, including not just the basic (but too limited) sprint/iteration but the full Agile method, from product strategy, vision and planning right down to daily stand-up meetings.
- A definition of the many disciplines that underpin Agile – collaboration, quality-first, pairing, refactoring, and so on.
- This should include a comprehensive (but not excessive) suite of practices for individual tasks, such as iteration planning, refining the backlog, conducting a retrospective, and so on. These practices need to be defined in a way that hey work for experts and novices alike.
- The methodology should also define the tools and techniques needed to apply Agile effectively. And these in turn need to be defined in a way that makes them flexible and adaptable to your working environment.
- And it should identify and specify roles and responsibilities at a level real individuals need to understand their part in the organisation, explaining who does what, where, when and how. This in turn should explain not just how the Core Team responsible for producing working software works but also how to that team engage and manage the Extended Team of experts, authorities and other stakeholders on which Agile relies.
Note how far these requirements go beyond a quick sketch of an Agile sprint or iteration. Yet that brief outline is what many organisations seem to be happy with – no wonder then that they find Agile itself under pressure.
Of course all this is pretty much a description of Agile201, and naturally we believe that practically any organisation would benefit from adopting Agile201. However, in order to avoid the rest of this paper turning into a blatant plug for Agile201, we’ll continue in general terms.
But where is the evidence?
So, where is the evidence that a robust methodology is key to making a long-term success of Agile?
Well, year after year VersionOne’s Annual State of Agile surveys have asked experienced executives, managers and developers (recently almost 4000 of them!) what makes Agile projects fail, and year after year the results are surprisingly consistent. For example, the 10th Annual State of Agile Report found that the top answers to the question ‘What causes Agile projects to fail?’ were (in descending order of importance):
- Company philosophy or culture at odds with core agile values.
- Lack of experience with agile methods.
- Lack of management support.
- Lack of support for cultural transition.
- Inconsistent agile practices and process.
- External pressure to follow traditional waterfall processes.
- Ineffective management collaboration.
- A broader organizational or communications problem.
- Unwillingness of team to follow agile.
- Inability to continuously prioritize work.
- Insufficient training.
- Ineffective collaboration.
VersionOne’s survey doesn’t say exactly what its respondents meant by any of these answers, but it’s clear how many of these would benefit from the presence of a proper methodology:
|Cause of failure||What methodology adds|
|1. Company philosophy or culture at odds with core agile values.|
|2. Lack of experience with agile methods.|
|3. Lack of management support.|
|4. Lack of support for cultural transition.|
|5. Inconsistent agile practices and process.|
|6. External pressure to follow traditional waterfall processes.|
|7. Ineffective management collaboration.|
|8. A broader organizational or communications problem.|
|9. Unwillingness of team to follow agile.|
|10. Inability to continuously prioritize work.|
|11. Insufficient training.|
|12. Ineffective collaboration.|
In short, a methodology empowers both delivery teams and the organisation as a whole to apply Agile not only to individual releases and projects but one a scale and level that makes Agile much easier to sustain.
Why not build your own?
First things first. If a methodology is fundamental to making a success of Agile, why don’t you just build your own? Unfortunately it’s harder than it looks.
- Building a robust methodology demands some very specialised skills and a lot of experience.
- Few organisations have the ability (knowledge, skill, experience, resources) to construct the disciplines, processes and practices that allow Agile to work effectively.
- Instead they either overpower the delivery teams with paperwork or give the delivery teams such a degree of ‘freedom’ that they soon start to disrupt the rest of the organisation, leading to a rapid backlash from all sides.
- Building a methodology is also time-consuming. Can you afford to divert your best people for days, weeks – months? And then are you willing to bring them back again, over and over again, to fix, support, extend and improve it?
- Finally – have you ever actually built a lifecycle before? And did it work?
OK, so you probably shouldn’t build it yourself. How about hiring a consultancy to build it for you? Well, before you take that route, ask yourself –
- Can you see what you will really be getting – right now?
- Do you know what it will really cost – in the end?
- And who will maintain it? The same consultants? At that price?
And no matter what they tell you, this isn’t a field where most consultancies have any real experience. Few consultancies have the resources, skills or experience needed to build a proper methodology.
In short, many organisations start to build their own methodology, but not many finish.
In many ways building a methodology is like building a tool: the knowledge, skills and techniques you need to build a tool are completely different from using one. Recently we witnessed a major American bank squander six months and the skills of half-a-dozen otherwise perfectly capable engineers on failing to build a relatively simple time-tracking tool; we hesitate to think what would have happened if they’d try to build their own Agile toolset too. There are reasons why tools are built by specialists, and wise organisations buy their tools instead of trying to build their own. The same reasons apply to building your own methodology.
Or perhaps an even better metaphor is a car. Literally a billion people in the world can drive a car, but the number who could design and build one can be measured in the thousands (no, that doesn’t include assembly line workers, who know how to operate their machines, not build a car).
Agile201 personnel have observed all too many organisations trying to build their own methodology. It’s painful to watch, the result is typically an unholy alliance of crushing bureaucracy and unjustifiable whim, and the result is usually universally loathed by its users.
Levels of sustainability
Sustainability isn’t just one thing. There are many tests of just how likely your Agile methodology is to last, and how well your methodology passes one test isn’t always a good guide of how well it would do up against others. Formal definitions of how robust your methodology is (such as maturity models) can be useful, but that is too complex a topic to be addressed here.
A more critical question for anyone thinking about adopting an Agile methodology is perhaps, how will it benefit attempts to build an Agile organisation, from empowering the individual delivery team right up to transforming the culture of the organisation as a whole. The stages in this process – though they are unlikely to unfold in quite as linear a way as this – are suggested below.
These stages are reviewed in the next few paragraphs.
Empowering the delivery team
The group that benefits most obviously from having a strong Agile methodology is the delivery team. Especially when they are relatively experienced, a methodology sets out a clearly articulated flow of information, decisions and deliverables – exactly the things that tend to go wrong where a methodology is missing. Team members not only know who does what but also have a proper arsenal of procedures, tools and techniques with which to do it.
At the same time, a methodology also gives the delivery team tools to resist the progressive decay and erosion of Agile itself:
- The knowledge and advice made temporarily available by training and coaching is turned into a permanent resource anyone can access at any time.
- The level of detail in a methodology is always far greater than any training, or even most coaching, so you avoid the gradual loss of detail Agile often suffers from – and the resulting tendency to replace Agile with non-Agile alternatives.
- A shared methodology is a vital reference point for distributed teams. Without an agreed methodology, coordination and alignment is an unending process of negotiation and debate; with an agreed methodology, the great majority of ‘why’ and ‘how’ questions are answered automatically, leaving the team as a whole to decide the ‘what’ – which is exactly as it should be.
- Because a comprehensive methodology defines not just the team’s day-to-day work but also how it engages and relates to other groups of experts, authorities and stakeholders, it is easier to manage and resist any attempt by these groups (and other, more distant players) to change how the team operates. In particular, the many respondents to VersionOne’s surveys who have said Agile projects failed because of ‘External pressure to follow traditional waterfall processes’, ‘Ineffective management collaboration’ or ‘A broader organizational or communications problem’ should find managing their management much simpler.
Unifying the organisation
Unified, cross-team methodology
Although it is perfectly possible for each delivery team to have its own methodology, it rarely makes sense: it’s much better to share the effort and resource expended in building and maintaining a methodology across as many teams as possible.
(In fact that’s a large part of the business case for Agile201 – we expend a lot of resource on building Agile201, but because it is shared across many organisations, its cost to each one is minimal – certainly a lot less than any equivalent training, consultancy, coaching or tool approach.)
At the same time, one key benefit, even for the individual delivery team, of adopting a consistent methodology all across an organisation is that, as personnel move up, down and around the organisation as a whole, they carry with them a shared understanding of how Agile is done. As they move from team to team, or even into other stakeholder groups altogether, they hit the ground running.
For example, many years ago, when our Chief Architect was building methodology for Accenture, he was evaluating a major programme in the north of England. The programme was suddenly short of resources, so they flew a team in from Kuala Lumpur in Malaysia. When our Chief Architect questioned how effective this could be, given the urgency of the problem, a very senior Accenture partner informed him that, as all Accenture consultants – worldwide – were all trained to use the same methodology (Accenture’s famous Method/1), the would be no problem. To quote the Accenture partner’s exact words, “At Accenture we have two common languages – English and Method/1”. The Malaysian team were duly inducted and working at full speed in only a few days.
Looking at Agile from the opposite point of view from the individual delivery team, every organisation of any size has a large group of important and powerful stakeholders who probably don’t care very much about how delivery teams work, just as long as they actually deliver. In practice, what these senior executives are concerned about is not Agile itself but whether Agile (ideally) provides them with the same information, decisions and results as the non-Agile approach Agile supersedes, or (at least) some kind of equivalent. If not even equivalents can be delivered, the problem Agile faces becomes much greater, and its opponents are far more powerful.
The principle sources executive management relies on to know whether its aims are being achieved are internal administrative functions such as finance and human resources. These functions can only assure senior executives that Agile is working if they in turn have a consistent flow of information both to and from delivery teams. Of course, Agile is rather notorious among the non-delivery functions for a rather chaotic approach to this flow, but at the same time, if delivery teams are forced back into pre-Agile methods of tracking, analysing and reporting progress, Agile itself – and all the many benefits it is capable of delivering – is unlikely to survive.
There are also quite a few non-delivery groups who do care very much how the delivery teams do their work, because these are groups who have to align their work directly with delivery. These include:
- Business management.
- Pipeline management teams.
- Support teams (environment management, facilities, tools, corporate data, etc.).
- Regulators and auditors.
But with a shared methodology that need not be the case. On the contrary, a well-structured methodology always provides the necessary ‘hooks’ – interfaces and flows – needed to satisfy the organisation’s administration. A shared methodology and its supporting tools (especially robust product management and the various levels of backlog) also ensure that, when they see a report, everyone knows what it means, and they know that reports from different teams are all in the same language.
Even from the point of view of your auditors, a methodology like Agile201 defines such a complete suite of processes, procedures, standards, tools, techniques, roles, responsibilities and so on, and leads to such clear delivery of business value that they should be more than happy to work with it.
Some adaptations are usually needed – for example, if product teams replace matrix management, then various parts of the administration will need to make adjustments. Likewise, the structure of budgets and financial reporting is likely to change. But these changes are unlikely to be any more radical or difficult than many past changes (e.g., the original introduction of matrix management). Some examples are suggested in this diagram:
In summary, a good methodology provides full support for most non-delivery functions, and therefore for senior executives. When properly implemented, it offers at least equivalent and often identical information and decisions, deliverables and deliveries.
Perhaps the single most important challenge many Agilists face at the moment is scaling. Without the ability to coordinate Agile to support multiple teams across company-wide delivery programmes, it is unlikely to complete its long-sought graduation from small-scale to industrial-scale software delivery platform.
In a sense there seems to be no problem. According to the VersionOne State of Agile Report mentioned above, the preferred recipe for 72% of respondents for scaling is a Scrum of Scrums. Although specialist scaling methods such as DAD, LeSS and SAFe are gaining ground, the continued widespread use of Scrum of Scrums suggests that Agile is well able to handle itself.
That is not to say that scaling is a safe bet. So when VersionOne’s respondents were asked for their ‘Top Tips for Success with Scaling Agile’, it is interesting that the clear Top Tip (with 43% of respondents recommending this) was ‘Consistent process and practices’. Methodology outstripped not only tools (almost the default response to any problem among software engineers!) but also consultants and trainers and even executive sponsorship and an internal Agile support team! Evidently the lack of methods has bitten deep. In fact, ‘Consistent process and practices’ is repeatedly the #1 Top Tip.
Building an Agile culture
As various people have been credited with saying, culture eats strategy for breakfast. If your company’s culture is firmly grounded in waterfall, or otherwise conflicts with the basic tenets of Agile, you will struggle to make a success of Agile. And you cannot realistically challenge an entire organisation’s culture head-on. However, most non-Agile organisations are based on a hierarchical command-and-control system led by large-scale planning and a radical dissociation between IT and the rest of the organisation, which is exactly the opposite of Agile.
This isn’t just a theoretical issue either. Back with the VersionOne survey, the single most widely noted cause of Agile project failure was ‘Company philosophy or culture at odds with core agile values’, and if that isn’t enough, number 4 was ‘Lack of support for cultural transition’. And then, when asked to identify the ‘barriers to further Agile adoption’, again number 1 was ‘Ability to change organizational culture’. There’s no getting away from company culture.
So what part does methodology play in the transition from an Agile-hostile to an Agile-friendly culture?
Well, culture itself is grounded in patterns of behaviour and beliefs and in the progressive institutionalisation of different patterns that collectively allow you to build up an alternative way of looking at delivery. There are reasons why most organisations favour a top-down, management-driven approach that have little to do with the success of this approach and a lot more to do with threats to embedded statuses, comfort and inertia, and above all a set of beliefs about employees that assumes that the self-disciplined professionalism on which Agile depends is beyond them.
Hence the value of methodology, which identifies the key disciplines and supports them with Agile-specific definitions of success, end-to-end processes, defined roles and responsibilities, routine ‘how-to’ practices and robust, effective and efficient tools that, taken together, define a different approach. Pursued with high-level authority, persistence and resources, and encouragement, feedback and active support that is sensitive to why change is so difficult, this can, in the long term, even change the organisation’s culture.
Methodology isn’t the answer to every question, but it is surprising how many of the outstanding questions in the Agile universe are at least significantly clarified and resolved by the introduction of a proper methodology. There are plenty of ways of starting out with Agile, but few ways of making it stick. As both a practical analysis of the problem and research among users suggests, methodology is one of them.
From our point of view at Agile201, this is of course excellent news. Well, not exactly news – we have observed the benefits of methodology over and over again, and now we’re offering literally every organisation in the world an opportunity to subscribe to a truly full-spectrum, end-to-end, industrial-strength methodology.