As we all know, Agile is big business. It takes up pages in Forbes and the Harvard Business Review, it’s galvanised huge consultancies, it has even spawned what has been called (with varying degrees of enthusiasm) an ‘Agile Industrial Complex’.
How much is the global Agile ecosystem worth? Atlassian alone is valued at $10 Billion so the Agile tools sector must top $100B. A single transformation costs millions, and there are hundreds of those in hundreds of cities every year. Training, certification, coaching, consulting, conferences … the global total may be more than a trillion. (Peter Merel)
But is Agile real? Or just another fad? Well, it would certainly be very unwise to deny Agile’s importance. So how do we deal with it? What is Agile? What difference does it make? What do all these Agile buzzwords – ‘sprints’, ‘stories’, ‘showcase’, ‘retrospectives’ and all the rest – actually mean?
It’s tempting to jump straight into the detail, and if you succumb to that temptation, there’s plenty of good material to feed it. Lots of people –consultants, trainers, coaches – would be only too happy to tell you what all the above terms mean and how to ‘do’ Agile. However, the most important thing to understand about Agile is not the detail but the sheer magnitude of the change it represents.
Hence this article, whose purpose is to introduce CIOs and other executives tasked with implementing Agile to the profound change Agile represents and the many obstacles that litter the path to Agile. And what this article says, in a nutshell, is that although Agile itself is really simple, getting to Agile is by no means easy. Because Agile represents a way of working, and even of doing business, that is radically different from – and perhaps completely at odds with – how you work now.
What this article doesn’t say is equally important – that the traditional corporate environment is quite fundamentally incompatible with Agile – and that, if you want to achieve the success only Agile offers, you’re going to have to deal with it.
The Biggest Risk
The single biggest risk to Agile comes before you even start. It’s there in your most basic assumptions about Agile, change and your own organisation. This is the (possibly unspoken) notion that you’ll be able to implement AINO – Agile in Name Only. If you take a superficial look at Agile and decide you can pick and choose the parts you find convenient, it won’t work.
Although it has many flavours and is by no means ‘all or nothing’, Agile is genuinely radical. It goes to the very roots of how you and your organisation operate. As Dave West, CEO of Scrum.org has put it:
The principles and the tenets of the agile manifesto are straightforward and easy to understand, but difficult to integrate. It has to be a complete mindset shift in the way an entire organization, from the bottom to top, from executives to the entry-level developers, approach their careers, their jobs and their roles.(Quoted in CIO magazine)
This is the first, most important, and unfortunately the hardest thing CIOs need to understand about Agile. Agile isn’t a change, it’s a genuine transformation, after which nothing will be the same. So if you don’t treat it as fundamental, the results will be trebly negative:
- You won’t get to Agile.
- You will have wasted a lot of resources that could have been used on something else.
- Your organisation will believe that little bit less in your ability to lead real change.
Exactly what this idea that Agile is genuinely radical entails will become clear in later sections.
(For more basic assumptions that will derail Agile from the very start, try here.)
Which brings me to the second Big Risk: that you will treat Agile like any another change. Adopting Agile isn’t like installing a system or tool or even like creating and reorganising departments. Of course, it’s those things too, but it’s true core is change to all the big-ticket items in any serious change management process – your culture, politics, organisation, management style and all.
So it’s safe to say that many people in your organisation won’t like Agile,and many will resist – some very actively. This isn’t just because many people don’t like change. It’s also because Agile is positively threatening to many people – not least some of your leadership team.It threatens their authority, their most basic notions of how an organisation should be run, and even their sense of identity. So you will need to champion Agile vigorously.
And that’s by no means the only problem. Consider this diagram – the so-called ‘Chasm’ model of change. You’re probably familiar with it already. Where do you think your organisation is on this model, and what does your answer mean for your attempt to implement Agile?
Agile has been around for about two decades. So if you’re only thinking about adopting Agile now, your organisation is probably in the right-hand section of the chasm model – maybe quite a long way to the right. And as you move farther to the right –
- Increasingly, your teams are cool about Agile. Indifferent. Maybe even hostile.
- In fact they may be barely aware of Agile – and not eager to learn.
- So your teams will need active encouragement and support to make it to Agile.
So it’s possible that the people you’re expecting to adopt Agile simply don’t want it. It’s not like installing a new time-and-expenses system – and, precisely because Agile itself aims to empower your teams as fully as possible, Agile itself will provide ample opportunities to subvert your efforts.
So before you start:
- Poll your staff for their views on Agile. Make sure you include your leadership and management teams. Take the answers very seriously.
- If your staff are hostile – or just not positive – they will need a good deal more than training to turn them into eager Agilists.
- If your management regard Agile as something they can take in their stride and/or do not recognise how serious a change it will be, they will need disabusing of this belief.
In short, recognise that Agile is a serious, organisation-wide issue. For most organisations, it’s usually the biggest single change they have ever undertaken, often by a mile.
Read – And Digest – The Agile Manifesto
So what is the big deal about Agile? Well, the basic principles of Agile are expressed in the Agile Manifesto, which you can find at agilemanifesto.org. Here it is in its entirety– yes, all 68 words of it!
“We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:
Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan
That is, while there is value in the items on the right,
we value the items on the left more.”
It’s these four values– Agile’s valuing of “this over that” – that are what Agile is really about. There have been remarkably few attempts to expand on or even interpret these values, they seem so obvious to Agilists. From the point of view of the executives of a non-Agile organisation, though, they can seem very strange. In fact the Agile Manifesto’s values challenge pretty much everything about traditional software development and delivery.
So here’s a brief explanation of each one.
‘Individuals and interactions over processes and tools’
In traditional IT delivery, the role of developers is pretty ant-like: they are given requirements and, by following a set of prescribed processes and using a set of fixed tools, turn them into packages of code for delivery to operations, users, customers, and so on. The skills they are expected to apply are narrow, the discretion they can exercise is minimal, and their contact with other groups is frequently zero.
Even within the delivery process, the individuals and teams performing the individual roles – analysts, designers, architects, programmers, testers, tool support, configuration management, and so on – generally work alone, on strictly individual assignments, in isolated silos, separated from one another by high walls and ‘stage boundaries’.
It’s a smart solution for ants but for human beings – very unsmart. Two very fundamental things are being lost here: the all-round experience, knowledge and creativity of individuals, and the synergistic power of teams – especially when they include users and customers and operations as well as developers – to innovate for themselves, without waiting for ‘official’ experts to provide the right answer or for authorities to permit them to use their often enormously rich experience.
Hence the Agile view: especially in the context of modern software development, where change, speed and creativity are paramount, individuals and interactions should take precedence over processes and tools.
It’s not just that coders and designers sometimes need to talk; it’s more that a skilled tester can tell you in a few minutes ten things about your requirements that will make them unverifiable. Or that an experienced developer will be able to tell a user ten things they could have from their software that they’d never imagined. Or that a user could explain nuances to a coder that requirements could never convey. And so on. And on. And on!
And that’s not all. Agile’s also pretty keen on various things that go with interactions – things like collocation and collaboration, and making sure everyone the team needs is right there in the team. When you can simply turn to the person whose knowledge or thoughts you want to tap, you just do it. You don’t need to set up a meeting, think of an accounting code they can charge their time to, or write minutes of the meeting. Just talk, decide, do.
(Interestingly, one of the authors of the Agile Manifesto, Martin Fowler, recently noted that the authors didn’t care about the order the Manifesto’s four values were stated – except for this one: they wanted ‘Individuals and interactions over processes and tools’to come first.)
‘Working software over comprehensive documentation’
‘Working software’? Isn’t that what we produce right now? Yes, hopefully it is. Although often it’s software that meets requirements no one has updated in 12 months, and the users don’t need that now. But more importantly, the traditional process of building software makes you do a lot before anyone gets to writing any actual software, and a lot more before it ever lands on the user’s desktop: a requirements catalogue, architecture, design specifications, programme specifications, code, data, multiple levels of test materials, and so on.
But why do we have to do all this? Mostly it’s because of three things:
- We define deliverables in terms of huge monolithic systems that can’t be described in simple terms.
- Development and delivery skills are partitioned into specialised roles organised into isolated silos that cannot communicate spontaneously or without significant overhead.
- The resulting distance– physical and functional – separating the many participants in the delivery process.
Without these, a huge proportion of the non-code items – usually most of them – would be wholly unnecessary.
Hence Agile’s strategy (already described in part):
- Break monolithic systems into small, manageable ‘stories’, each representing a minimum unit of value-adding functionality, and focus on delivering these stories as independently as possible.
- Bring all the skills needed to understand and deliver what the organisation needs, including users, into a single team, and encourage them to collaborate and share skills.
- Collocate teams so that they can maximise the opportunities for collaboration and synergy, and minimise the causes of delay, loss of information and ossification.
As Agile people like to say, the fastest way to do anything is not at all.
‘Customer collaboration over contract negotiation’
What is your organisation’s response to a customer (or user) changing their mind? In a traditional organisation, it’s something like:
- Lots of internal debate on the customer/user side, followed by hand-wringing and protests from Finance.
- A formal change request is painstakingly drafted, reviewed, revised, reviewed, approved and submitted.
- An impact analysis is performed.
- Contract negotiation (or, in the case of internal developments, endless debate and argument about budgets).
- Yet more elaborate reviews and authorisations.
- Disruption and delay.
- Bad feelings.
This whole sequence is based on the assumption that change is abnormal. It isn’t. Nothing is more normal to software development than change – of scope, of requirements, of schedule, of priorities, and of practically everything else.
So that’s exactly how Agile addresses the problem of change:
- By taking it for granted.
- By expecting to have to adapt – frequently and substantially.
- By working so closely with their customers/users that change never comes as a shock.
- By creating an environment in which change is as easy as possible.
It’s a shocking paradigm to a contract-oriented organisation: things have changed, so we should do something different. Just like that – no formalities, no documentation, legalities. Just Do It. (Yes, Agile is one of the few places where the old Nike slogan makes sense).
But from Agile’s point of view, the business case is robust:
- Give customers what they want and everyone will be happy, satisfied and back for more.
- Let developers just get on with development and they’ll be happy, productive, and still here at the end of the financial year.
‘Responding to change over following a plan’
The single most universal tool of traditional software development and delivery is almost certainly the plan. But the way plans are traditionally created, used and maintained is strictly bureaucratic:
- Creation: Template-based, comprehensive detail, long-term (often full project/programme) view, detailed assignments, formal review and approval, controlled distribution.
- Use: Formal tracking and reporting processes (e.g., milestone-based),
- Maintenance: Formal change control, including change requests, review and reapproval.
And so on. And why is all this control necessary? For essentially the same reasons as traditional projects need such comprehensive documentation:
- We define deliverables in terms of huge monolithic systems that can’t be described in simple terms.
- Development and delivery skills are partitioned into specialised roles organised into isolated silos that cannot communicate spontaneously or without significant overhead.
- The resulting distance – physical and functional – separating the many participants in the delivery process.
Now, of course, formal plans are so standard that it’s hard to imagine any way of delivering substantial results without using such methods, tools and techniques – until, that is, the creation of Agile. And once the traditional norms have been replaced by small-scale, rapid, highly adaptive development and delivery, ‘responding to change’ ceases to demand elaborate planning – or, indeed, any ‘plan’ in the traditional sense at all.
But when individual stories are delivered within days or weeks, when the resources needed to do the work and represent the business are within the team and everyone is collocated within a single space, the ‘plan’ can be reduced to a single list of stories ( the ‘backlog’) and a simple ‘burndown chart’ showing how many we’ve delivered so far.
Similarly, when the situation changes – opportunities arise, priorities shift, and so on – a single brief meeting between the team members is usually enough to decide what needs to change and how to change it. As everyone is present, there’s no need for formal reviews or controls.
What does ‘over’ mean?
But what about the very last sentence in the Manifesto – ‘That is, while there is value in the items on the right, we value the items on the left more’? This may seem rather vague, but its significance for Agilists is profound.
The items on the right of each value – ‘processes and tools’, ‘comprehensive documentation’, ‘contract negotiation’ and ‘following a plan’ – more or less define how IT delivery is done in non-Agile organisations. But in Agile, they are deprecated as occasionally-useful-but-generally-not addons.
Of course, there are situations where they are useful, even indispensable. But for Agile, the simple fact that managers and administrators want them is not a good enough reason to do them. Agile will only give them to you if you can make out a compelling case for them: if you can’t you won’t get them at all.
How is this justified? The details will become clearer below, but in general terms Agile believes that the professionalism, expertise and self-discipline of teams are more reliable than traditional hierarchy, command-and-control, and bureaucracy. Sometimes bureaucracy is useful (and occasionally unavoidable) but by and large it is an overhead that, as far as Agilists are concerned, seldom adds any value, is often actively obstructive and really should not be tolerated.
So as you read the Agile Manifesto, ask yourself how much your current management style depends on the items on the right – and how hard it would be to give them up.The impact will certainly be radical, sometimes to the extent of removing whole functions. And then ask the same of the other managers and executives – not least the one’s whose roles will disappear under Agile – who’ll be affected by any implementation of Agile.
… And The 12 Agile Principles
The Agile Manifesto doesn’t stop there.In support of its four values, the Agile Manifesto’s authors also proposed 12 Agile Principles:
- Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.
- Welcome changing requirements, even late in development. Agile processes harness change for the customer’s competitive advantage.
- Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.
- Business people and developers must work together daily throughout the project.
- Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.
- The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.
- Working software is the primary measure of progress.
- Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.
- Continuous attention to technical excellence and good design enhances agility.
- Simplicity – the art of maximizing the amount of work not done – is essential.
- The best architectures, requirements, and designs emerge from self-organizing teams.
- At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.
No more will be said about the individual principles here, though for more on what each one means, why it’s hard and what to do about it, see here. But just reading the list makes it clear how different – again – Agile is from traditional approaches to development and delivery:
- Actual delivery of value-adding product begins as soon as possible and continues to be as frequent as possible. Big Bang implementations are out.
- Change is good. If a user or the business changes its mind about something, that’s progress -so why would we not want to change with it? As Mary Poppendieck – one of the leaders in the related Lean movement – put it, ‘A late change in requirements is a competitive advantage.’
- The silos are all gone – even the larger silos of ‘business’, ‘IT’, ‘operations’ and ‘customers’. We’re all aiming for the same goal, so we should act like we’re one team, not like a swarm of disconnected, mutually indifferent and sometimes even hostile groups.
- Only do things that add value. If they don’t, drop them – even if they are part of ‘standard’ development and delivery processes. The ultimate aim is to deliver benefits to the user, not to comply with ‘the rules’.
- The whole is much more than the sum of its parts – but only if it’s allowed to act as a whole. Interaction across the whole delivery lifecycle should be the norm.
- Development is product-, not project-based (more on this later).
- Being the best is not just an ideal but also a fundamental business goal. Not only are poor quality engineering and demotivated engineers among the major causes of delay and failure but the return on investment in technical excellence is high.
And so on. As this partial list makes clear, Agilists don’t think like traditional developers or managers. This isn’t because Agile is some strange new fad going around – it’s because Agile has successfully learned from the decades of overruns and failures that had previously been the norm in IT delivery.
Some of these principles have been challenged. For example, it’s been suggested (not least by myself) that number 7 – ‘Working software is the primary measure of progress’ – isn’t really ambitious enough. If development teams are to include users and other business stakeholders, they know enough to go beyond ‘working software’. So what they should aim to deliver is ‘maximum value’ (however value is defined by the wider organisation). But this isn’t really a criticism of Agile. Firstly, the Agile Manifesto is specifically about software development, so working software is a reasonable placeholder for the bigger idea of ‘value’. And secondly, it only extends Agile still farther beyond traditional development in ways that are consistent with Agile itself.
But even the 12 Principles aren’t enough to define Agile fully.Around these principles a whole raft of disciplines has evolved with names like Quality First, Trust the Team, Incremental, Pairing, Transparency, Adapt Continuously, and so on. All of these disciplines follow from Agilists’ attempts to translate the Agile Manifesto and the 12 Agile Principles into practical ways of working.
These disciplines won’t be explained here, mainly because, although it’s tempting to jump down to that level, they are really secondary compared with the 12 Agile Principles, or indeed anything else referred to in this article. If you try to make use of them without understanding the higher-level Agile concepts, your implementation of Agile will almost certainly fail.
Professionals, Not Employees
Agile is by no means limited to high-level manifestos and principles. It penetrates right down into the smallest details of the organisation. But of its organisational implications two in particular stand out: the radically changed status of individuals within the organisation, and the contrast between how Agile and traditional delivery are organised.
Regarding the status of individuals, Agile assumes an organisation of professionals, not employees. To understand why, consider the following points:
- An employee is an individual you pay so you can tell them what to do. A professional – doctor, lawyer, teacher, priest, engineer, etc. – is a person you pay so they will tell you what to do.
- This is the basic change to how people work under Agile: a world of employees is transformed into a world of professionals.
- The basic challenge this creates for organisations is not whether or not Agile ‘works’ but whether filling your organisation with professionals is compatible with hierarchical command-and-control systems, organisation and management.
- In an employee-based organisation, authority lies with whoever is more senior. In a professional organisation, it lies with whoever has the necessary expertise – not only to decide how something should be done but also whether it should be done at all.
- This is a critical issue for any organisation moving to Agile because, where the basic ‘transaction’ in command-and-control hierarchy is to tell, in a professional organisation it’s to ask.
- And the basic risk to management in a professional organisation is that the person you ask will openly challenge what you want or how you want it. They may even simply say No.
Not everyone is suited to this model – not just staff but management. So, before implementing Agile, you need to ask yourself these questions:
- Am I running an organisation in which people are afraid to say No to their manager? (And remember, your personal opinion is not a valid answer to this question – you need independent research.)
- Can I bring myself to recognise that sometimes (often?) ‘junior’ people know better than me about what I’m asking them to do?
- Am I ready to give up my authority over them in exchange for a better organisation delivering better results?
- Am I willing and able to change people and the basic structure of my organisation?
- Am I willing to remove people – including some of my ’best’ executives – if necessary?
Don’t even think about implementing Agile until you can answer these questions – and you know what to do about the answer.
Networks, Not Hierarchies
If Agile isn’t to be treated like just another corporate add-on, you’ll need serious reorganisation. But it won’t be like adding a new department or managing a merger. What Agile assumes is that you will replace your existing hierarchy of functions with a network of networks.
What does that really mean? Well… In short:
- End silos.
- Flatten the organisation.
- Above all – self-organisation.
At greater length:
- Management and delivery functions are organised as isolated silos. They are inherently rigid, with fixed and impermeable boundaries. A network is inherently fluid: it has no fixed or impermeable boundaries.
- These functions are organised into a hierarchy and defined in terms of the roles and resources needed to carry these functions out – for example, a design function, test function, a project management function, administrative functions, operations functions, etc. A network is defined in terms of its ability to deliver specific value and the capabilities it needs to do this. For example, teams for Product A, Product B, help desk support, and so on.
- Hierarchies are permanent, created by command from above and independently of the detailed content of the work they do. (That’s why they’re ‘functions’.) Networks consist of self-organising groups that set themselves up around, and are specifically self-designed for, particular products, problems, opportunities, etc. They last only for as long – and only in the form –the team needs to deliver the value they are self-designed to create.
- In a hierarchy, higher functions have automatic authority over lower functions, based on position alone. By contrast, each network has full authority over its own area, based on its demonstrated capability. Networks of networks cooperate as equals.
- A hierarchy’s functional silos are populated by specialists, from which resources are recruited to carry out the actual work needed to achieve the organisation’s goals.Network teams either include or have rapid access to all the knowledge, skills and authority they need to carry out their own work.
- A hierarchy is managed by top-down command-and-control system of direction. The function of management is to identify, organise, direct and control work. Networks are self-directing. Work is identified by teams in liaison with stakeholders in the organisation as a whole, while management’s role is to enable and support network teams and the network of networks as a whole.
- A hierarchy uses a formal bureaucracy of procedures, records, authorities and so on to manage and coordinate its functions. In a network of networks, formal controls and records are regarded as occasionally useful tools whose use should be minimised. Informal methods – especially self-organisation and face-to-face discussions within collocated teams – are preferred.
- A hierarchy’s values are authority and conformity, and it operates through formal controls (procedures, records, authorisations, etc.). Everything else – e.g., whether or not something can or should be done – is subordinate to this. So an action that doesn’t ‘follow procedure’ is wrong (‘invalid’) even if it works. A network’s values are actually delivering value and sustaining the pace of delivery (i.e., if the way you ‘do Agile’ today stops you being Agile tomorrow, you are not doing Agile).Everything else is subordinate to this. So if an action does ‘follow procedure’ but doesn’t deliver value, it needs to be changed.
- In a hierarchy, authority is static, based on prescribed positions. In a network it’s completely dynamic, constantly shifting to whoever needs to take the lead to achieve the network’s goals.
- Within individual hierarchical functions, authority is also strictly top-down, from manager to team, and is assigned by a yet higher function. Within an individual network, authority is normally replaced by collaboration between equals, but where an authoritative decision or knowledge is needed, it’s based on expertise and ability, as acknowledged by the rest of the network’s members.
- The corollary of the tell vs ask dichotomy is authority vs trust. In a hierarchy, you do what you’re told; in a network, teams collaborate based on confidence in one another, which is in turn based on shared experience.
- A hierarchy ignores the knowledge, skill and experience of individuals except in terms of their official authority and the specialised skills they bring to their assigned tasks. A network strives to mobilise all capabilities that can contribute to success by encouraging wide-ranging debate by anyone with relevant knowledge, skill or experience.
- Likewise, hierarchies isolate individuals both by assigning them work to perform in isolation even from other members of the same function and team, and by evaluating their performance almost exclusively as individuals. Networks avoid both these shortcomings by dividing work among themselves based on individuals selecting the work they will do; by using collaborative techniques such as pairing (and even ‘mobbing’) and generally acting as teams; and by evaluating teams rather than individuals.
If you work in a traditional hierarchy, the very definition of a network of networks will probably make you feel either liberated or sick. If it makes you feel liberated, there’s a fighting chance you’ll be able to deliver Agile; if it makes you feel sick…
Again, the difference between Agile and more traditional approaches to development and delivery couldn’t be more stark. In the longer term there is also a serious issue about the compatibility of hierarchies of functions and networks of networks: unless they can be isolated from one another, it’s easy to imagine the strains having both will generate. As isolation is likely to be very difficult, there is perhaps a truly strategic choice to be made here: is the long-term outcome going to be a completely Agile organisation or the restoration of a bureaucratic hierarchy? And if the latter, how is your future performance likely to compare with that of your Agile competitors?
As a CIO, you’ll quickly find that it’s hard to isolate Agile parts of your organisation from the rest. The reason for this potentially disconcerting fact is that, even if you start adopting Agile in its traditional homeland – i.e., software development – the way Agile works means that it will quickly start overflowing into other spaces. This is because, compared to traditional software development,Agile assumes a radically different relationship between IT and its neighbours.
This has already been alluded to in previous sections, when I mentioned that:
- An Agile organisation replaces all functional and organisational silos with transient, multi-functional teams.
- Mature Agile teams incorporate all the resources they need to carry out their work.
The areas falling under ‘all the resources they need’ has expanded over the years, and just looking at the simplest diagram of the delivery process suggests just how wide the net could be thrown:
Creating Agile relationships between IT and these other areas will take (at least) the following:
- Equality and collaboration between areas in deciding what work will be done and how.
- Directly co-opting personnel from both sides into Agile teams.
- The delegation of more and more authority and control to these teams, as opposed to any residual management functions that may still exist in the organisation.
What is especially key to Agile relationships is the fact that they are all relationships between equals. Each side needs to treat its colleagues as fellow professionals – as people with whom they make decisions, not people to whom they give instructions.
This is likely to be especially difficult for business stakeholders, who are very used to having general authority over the delivery process. In Agile, the business can no more tell IT what to do than you’d tell your doctor how to diagnose and treat your illness.The business can tell IT what it wants to achieve, but it really is for IT to lead any conversation about how that should be accomplished. This includes fully expecting push-back on whether what they want to do makes sense.
If this seems shocking, the Agile perspective is that, taken as a whole, an experienced development team has an enormous range and depth of problem-solving experience. So the idea that they (should) have nothing to say about the business’s goals is as absurd as the idea that I could go to my doctor and simply announce what is wrong with me. If I’ve had such symptoms before, this is important information to convey to the doctor, but it doesn’t mean that the patient can sensibly diagnose themselves.
Products, Not Projects
Quite a lot of debate about Agile is couched in terms of slogans and hashtags: ‘No more [this]’ and #No[that]. By and large they’re not so much advice on what to do when first adopting Agile as goals for very mature organisations.
One recent example of this not always happy trend is #NoProjects – which must sound strange to anyone involved with traditional IT development and delivery. But #NoProjects follows directly from a central element of Agile, and you need to aware of it from the very start of adoption.
Agile’s replacement for the project is the Product Team. Whereas a project recruits resources to implement a single change or group of changes, after which its resources disperse to other work, a Product Team consists of a permanent team that continues to work on the same product for its entire lifetime. What’s more, it not only develops the product but also supports it, and (in more advanced versions) handles the entire lifecycle from original bright idea (many of which quickly originate with the Product Team itself) to updates arriving on the user/customer’s devices.
Product-based development is the only model developed so far that’s really compatible with Agile, because it’s the only model that emphasises the key elements of Agile itself such as:
- Intense self-organisation and collaboration.
- Deep knowledge, skills and expertise.
- Technical excellence.
- Ways of working massively adapted to the task at hand.
And so on. So if you’re still operating a traditional development and delivery lifecycle, Product Teams represent a truly radical change.
But there’s more to product-based development than simply setting up the team. The aim of a product-based approach is not only to reorganise resources around the product. That’s just a vehicle for achieving a still higher goal – optimising the delivery of value.To achieve this, the ‘product’ itself needs to be defined and structured so that the Product Team controls as much as possible at every level – not just IT. To achieve this, the product needs:
- An unambiguous position within the organisation, including a product-specific vision and strategy.
- Clear decision-making, culminating in a unique authority operating high in the organisation as a whole.
- A clear product space and boundaries within the overall enterprise architecture and product roadmap.
- Autonomous resourcing processes (including recruitment), so that the team get the people they need when they need them.
- Well-defined and unobstructed links to its own work pipeline (i.e., not complicated by external or overlapping investment and work allocation cycles).
- To share as few as possible of the organisation’s wider components with other products and projects, so that resources, organisation, methods and techniques can be adapted as quickly and as freely as possibly to changing opportunities and conditions.
- To control clearly delineated elements of the supporting development and operational facilities, systems, tools and operating data, including systems integration and deployment.
- Assured rapid access to a wider ‘Extended Team’ of supporting experts and authorities, so that areas such as bottlenecks to change in business procedure, training and so on are avoided.
- Clear and unambiguous relationships with all the other functions and groups needed to deliver the product’s success.
- A ‘Product Owner’ who operations as a ‘mini-CEO’ for that product, with all the control over strategy and resources that suggests.
Successful Product Teams don’t need absolutely all of these factors to be in place, and they can be dealt with progressively. In fact, as with almost everything else Agile requires, it’s a good idea if your organisation proceeds incrementally, by experimentation rather than trying to leap directly to a fully mature approach to product management. But the more completely you can realise this model of product-based development, the more easily success is likely to follow.
No More Requirements
For half a century, the traditional starting point for software development was always the requirements specification. Carefully elaborated, reviewed and approved, a fat document of detailed requirements would land on the project/programme manager’s desk and development would begin.
And then a couple of years later, a massively over-budget, over-schedule programme would deliver the remaining 40% of the original scope, and then the users could start being unhappy instead of the development team. Assuming, that is, that the project/programme hadn’t already been cancelled completely.
So what’s the Agile alternative? The Agile development process demands continuous user participation – not just ‘write requirements, fire and forget’. Instead two closely interconnected changes are introduced.
Firstly, requirements are replaced by ‘stories’. A story is different from a requirement in many ways, of which a few are relevant for understanding just how radical a change replacing requirements with stories really is:
- Stories describe complete units of value – not a detail in a larger solution. So they can be delivered in isolation from one another. This means that your stakeholders start benefiting as soon as the first story has been implemented – normally only a few weeks into the overall process. Requirements, by contrast, usually have to be delivered in large batches for any value to be realised, so realising any value typically has to wait months, if not years.
- Stories are short and easy to read, with just enough detail to estimate and prioritise. They aren’t elaborate exercises in exhaustive detail. This is possible because of the way development itself proceeds – by direct collaboration between the developer and an expert user who is perfectly capable of saying what the business needs – and also of explaining all the nuances and background that a requirement catalogue simply can’t communicate.
- Stories are on constant view to – and under constant review by – the whole team, not to mention passing stakeholders, visitors and neighbouring teams. So new ideas, unexpected impacts, creativity, innovation – all are constantly being maximised.
- Stories are always negotiable, right up until final delivery. Change is good. And they’re easy to change even while being implemented, without formal change controls, because everyone who needs to agree to the change is already ‘in the room’.
- The real status of development is always transparent, not obscured by bureaucracy. Just look at any Agile ‘Story Board’: no milestones charts, no formal reports. Just a rapid passage from ‘To do’ to ‘In progress’ to ‘Done’. Couldn’t be simpler.
- The ultimate test of whether a story has been implemented properly is approval by an expert user working directly with the developer. There is no elaborate acceptance testing or extended sign-offs.
- Even when they’re complete, stories are only as defined as well as the user and developer need them to be. They’re not exhaustively documented according to a formal standard that prescribes a mass of documentation no one will ever use. So enormous amounts of unproductive overhead are cut away.
- Stories are delivered as part of a rapid implementation process – not buried in a massive project or programme. So the bottlenecks and integration crises of pre-Agile lifecycles are replaced by a smooth flow of actual value.
Plainly this is a million miles from the usual requirements specification and tracking processes pre-Agile organisations are subject to!
Agile’s second innovation in this area, already alluded to here, is that the whole requirements cycle, from requirements to deployment, are replaced by active user participation in the development process. That is, instead of issuing formal requirements that a completely separate team of developers must meet, a very brief statement of the problem or opportunity (the ‘story’) is written, and then suitably expert and empowered users work directly with the development team – as part of the team – until the story has been implemented to everyone’s satisfaction.
This approach can be very disruptive if you’re not ready for it, so here’s why you should want it too:
- It gives developers a much better insight into the context of development – what users really do with the system. So the delivered solution reflects realistic usage, and has guaranteed usability.
- Needs are communicated directly from user to developer, without a complex, unreliable and instantly obsolescent requirements document standing between them. The developer validates their interpretation of what the user needs directly with the user.
- Because the user is actively reviewing the product with the developers as it is built, mistakes and omissions that are normally only discovered in test are prevented from happening in the first place.
- Compared to written documents, the bandwidth of person-to-person communication is enormous. So communication is no longer a game of Chinese whispers. Direct, multi-modal interaction brings out issues, mistakes and omissions that would otherwise be overlooked (until, that is, the system fails in production).
- Because the developers are working directly with the user, they’re always aware of how user thinking is changing. So adaptation happens quickly, easily, reliably and authoritatively, and without the overhead of extensive formalities.
- Because they’re actively discussing user needs, problems, ideas and circumstances directly with the user, the developer is often able to offer options and alternatives of which users are unaware.
- When stories are only starting points for a more detailed one-to-one discussion between user and developer, costly and specialised requirements management tasks, documents, organisation and procedures are eliminated.
- Estimating becomes much easier and more accurate, and prioritisation is more likely to reflect the delivery of maximum value.
- The team’s ‘Definition of Done’ is better attuned to the user’s real needs, preferences and priorities.
- Direct interaction with users is professionally and personally satisfying, and motivation rises on both sides.
Of course, there are limits to such an approach:
- Detailed technical or regulatory specifications aren’t well communicated by informal discussion.
- Stories affecting multiple customer or user groups may need extensive agreement. It probably isn’t realistic to do this during the development process.
- There are situations where the real users cannot speak for themselves.
And so on. So it’s crucial to remember that Agile does not exclude the use of formal materials or controls – it only prefers to avoid them wherever possible. So if a series of user workshops or detailed technical documentation is the right way to go, Agile will happily embrace it.
Moving to intensive user participation in development is a radical change, and one of the hardest parts of Agile. After all, users have day jobs: will you backfill them while they’re working on an Agile project? They’re your best people (or so they should be, if they’re to speak for your organisation within the team), so are you willing to assign them to the development team, even temporarily? Agile has been killed by smaller decisions.
No Big Bang
Getting your adoption strategy right is as crucial as anything else to making a success. I said earlier that adopting Agile isn’t like installing a system or even like reorganising existing departments and functions. Firstly, it isn’t a one-off change, and secondly – and most important of all, it isn’t a matter of making your organisation ‘be Agile’.
To understand this, consider a popular but fatally flawed approach to Agile: hiring one of the large consultancies to manage it for you. Their negative impact on Agile adoptions is clear:
The Agile consulting firms perpetuate the Agile Industrial Complex… These service firms know that force, coercion and “push” is what well-meaning execs assume will actually work, and what actually sells.
Put another way: these service firms know that awkward conversations about respect, self-management and self-organization do not really sell very well. And so it’s always best to avoid these conversations.
The Agile Industrial Complex, Daniel Melzick
But top-down imposition of Agile through large-scale re-organisation, formal processes and the like isn’t only dangerous because it’s incompatible with a ‘pure’ approach to Agile: it’s dangerous because it doesn’t work.You don’t get creativity and adaptability out of people by ordering them to be creative and adaptable.
So what’s the alternative? In fact Agile is best adopted – most efficiently and most effectively – by asking how you would adopt Agile if you were using Agile itself to do it. The answer is simple and straightforward:
- Implementing Agile needs to be a ‘pull’, bottom-up process, not a ‘push’, top-down process.
- You must invite people to adopt Agile, not order them.
- Work incrementally. Start small, prototype, learn, adapt and spread. You can start from many different points at once, but there’s no Big Bang option.
- Act the part. This cannot be overstated: if you want others to ‘be Agile’, acting in a non-Agile way sends a clear message that you don’t really mean it. (more on this later.)
- Manage your ambition: Agile at scale (e.g., for large programmes) is not possible early on.
- Trust the team.
Trust the team? What does that mean? Perhaps I can explain this idea by citing a tweet I sent a little while ago, relating an experience from three decades back:
I was recruited by a huge consultancy. One day (presumably because I have a degree in psych & sociology) I was shown into the CEO’s office & asked how it was they hired the best but killed all creativity. I was too scared to answer. Which was the answer. But he didn’t hear it.
It takes time for the idea that senior management really do want staff to speak up – and maybe even longer for them to truly believe that expressing a contrary opinion won’t quickly turn into a Career-Terminating Move.
All this takes time. Of course, this means that Agile can only be implemented slowly, iteratively and patiently, taking full account of – in fact actively inviting – the opinions and feelings of the people who are doing the real work.
In The Country Of The Blind…
As I’ve already said more than once, Agile isn’t a ‘system’ you get ‘experts’ or ‘consultants’ to install and then just train up the users. It requires everyone in your organisation to look at work in a different way, and to be willing and able to adapt an entirely different approach. By the time Agile has matured in your organisation, anyone involved in Agile itself will have completely different ways of working, including dramatically transformed responsibilities and authority.
- Those at the top will have relinquished a great deal of the authority they’ve probably taken for granted for decades.
- … but they’ll be doing work that adds real value to the organisation, and not just perpetuating counter-productive ways of working
- Those at the bottom – if it can be still described as ‘the bottom’ – will have taken on the principle responsibility for the very core of work.
However you feel about this revolution – and you should definitely put off adopting Agile until you feel fine about it – it plainly can’t succeed until everyone in your organisation is intimately aware of – and committed to – its success. So everyone in your organisation needs to be actively engaged in the process too, right from the very start.
And remember, in this context ‘engaged’ means ‘take responsibility for’. The very essence of Agile is trusting your staff’s ability to take responsibility for deciding what to do and how to do it. But for that to work, everyone needs to be part of the decision to adopt Agile in the first place – not just the fine details of how it’s done. If your staff just don’t want Agile, you cannot force it upon them. Or rather, you can force it on them – it just won’t work.
The natural corollary of this is that everyone in your organisation needs to understand the points this article makes. So make sure they do – from the top down and from bottom upwards. Explain Agile, invite your staff to consider Agile, to discuss Agile and its consequences for themselves, for their work and for your organisation as a whole. And then ensure that it is they who decide – by themselves – whether Agile is what they want.
So this is your first Agile leadership task: spread the word. Remember, in the country of the blind, the one-eyed man is incomprehensible. It is not enough for the leaders to see, choose and judge – so must the led. Because if you (collectively) decide to go Agile, they won’t be the led for long.
Are You The Leader They Need?
So: Agile takes time, it’s hard work, and it really can’t be delegated.Let’s take for granted that you know how to manage a major change in your organisation (or at least you know someone who does). Is this change like any change you’ve ever led before? Certainly not.
This is especially true if you’ve been highly successful in a traditional corporate environment – which, if you’re one of the CIOs this article is aimed at, certainly means you. Because a corporate environment specifically creates and strives to perpetuate many, many of what many Agilists would call anti-patterns of Agile, which is to say, patterns of organisation, culture and politics that aren’t just different from Agile. No, they specifically contradict Agile’s most fundamental principles.
Hopefully even a cursory reading of this article will have already made this point clear: Agile is full of words like ‘empowerment’ and ‘collaboration’; what it lacks is words like ‘authority’, ‘power’ and ‘command-and-control’. In fact Agile doesn’t just lack these terms. As you will have gathered by now – Agile is emphatically opposed to, say, hierarchical ideas of organisation.
So at some point between now and when you complete your implementation of Agile, you will have stopped being the person in charge of your organisation and started to be a unique but no longer quite so privileged node in a large network of networks. You will still be its overall leader, but you will no longer be the person in charge.
Hence the key question: How do you – the person currently in charge of your IT organisation – feel about this? And if the answer is not, ‘Very comfortable’, you also need to answer the obvious follow-up question: Are you actually the right person to lead this transformation?
It’s a difficult but vital question to face.
So what sort of person do you need to be to shepherd Agile into the world? Within the Agile community itself there is a considerable consensus about the answer to this question. The key concept here is that of the ‘servant-leader’. According to Larry Spears (a leading thinker on servant-leadership), the 10 characteristics of servant-leaders are:
- Commitment to the growth of others.
- Building community.
Other virtues often mentioned when experienced Agilists get together:
And so on. Are you this sort of leader? More to the point, do your team think you are? If either you or they think you’re not, are you willing to become this sort of leader? And if not, are you willing and able to get out of the way?
Why does a CIO needs to think and know so much about Agile before getting down to the details of how Agile itself works? Why can’t implementing Agile simply be delegated while the CIO gets on with more important stuff? Because Agile really is a revolution. Or if it isn’t, it will fail. And everyone in your organisation needs to be part of that revolution, or else they’ll become part of the reaction.
Plainly implementing Agile isn’t simple, but if there are any simple takeaways, they are these:
- This is going to be the biggest change of your career.
- The same goes for everyone else in your organisation.
- It will only work if you all work to make it succeed together.
Fortunately there’s a lot of help available, in the form of skilled Agile consultants, trainers, coaches and (in the unique case of Agile201) full-blown Agile methodologies you can subscribe to. And if you’re going to make your Agile implementation sustainable, you could do worse than look here.
However, if you simply rely on outsiders implementing Agile for you, without you understanding the real significance of what they are doing and without engaging your organisation as a whole, then it’s unlikely that Agile will work for you at all. Fortunately, if you’ve read this article with care and attention, you understand just how radical the Agile transformation really is, and so will not be too shocked at what it demands.
Finally, good luck. Hopefully if this article has done its job, you won’t need any. Or at least, a lot less than you would have otherwise.