There comes a point, especially in very large organisations, where Agile stops working. That isn’t to say that it doesn’t work at the scale it was designed for – the local team – and it should not be abandoned. But its limitations need to be acknowledged. On the other hand, the alternative is not the typical modern bureaucracy, which is almost as bad at doing proper bureaucracy as it is (via waterfall) at delivering software.
Tl;dr? Invest in your management systems like any other important investment. Maybe even more so, because in the long run it may well be your most important investment.
Why the answer isn’t always Agile
Agile is extremely attractive to many software developers: it’s flexible, non-bureaucratic and respects their professionalism and abilities. It has also, to a very large extent, proved its worth.
Unfortunately, Agile is often a lot less attractive to other people, especially in a large, complex corporate environment. This is partly because the Agile model is little understood outside IT and partly because Agile routinely defines itself in terms that are radically unfamiliar to non-IT people. But partly it is also because specific Agile methods such as Scrum and XP simply either ignore or take for granted many of the key elements of a large, complex organisation. There is simply no way that Agile as it is normally understood will work at the corporate level.
This is not because many of the basic lessons of Agile don’t apply elsewhere. Many organisations are senselessly bureaucratic, preferring top-down command and control to allowing teams and networks of seasoned professionals to just get on with their job – even in environments where such controls have no obvious superiority over letting professionals just get on with their work. True, large organisations have massive resources that could provide such professionals with invaluable support, but that isn’t the same as telling their staff how to spent every minute of their working day. In fact every large organisation would benefit from reviewing the Agile Manifesto and considering how it might be most usefully applied. Certainly almost every organisation I have ever worked for squandered immense resources in micro-managing skilled professionals.
No, the real problem with Agile, even in a software development environment, is that the span of control needed to manage a corporation is simply too large to be left to teams based on interpersonal collaboration and communications and the informal, bottom-up percolation of information, decisions and results. Although it’s tempting to appeal to Agile models such as ‘scrum-of-scrums’ as a way to progressively build up large-scale organisations, such a strategy would soon run out of steam if this scrum-of-scrums was itself only a speck in the organisation as a whole.
For example, I recently spent some time working with a very large global bank, rolling out new systems to 160,000 staff. Assuming (to simplify the calculation) that the average Agile team included 8 people, that’s 20,000 scrums for the organisation as a whole, or 2,500 scrums-of-scrums. Or 300+ scrums-of-scrums-of-scrums. Or 35+ scrums-of-scrums-of-scrums-of-scrums. Or 4-5 scrums-of-scrums-of-scrums-of-scrums-of-scrums. The last which, presumably, is the board of directors, huddling together, wondering how on earth they will ever manage to create a strategy for such a beast.
Ah, but this is surreptitiously smuggling in the profoundly unAgile idea of hierarchy! And that’s true. But unfortunately, if you prefer fluid communications between all these scrums, scrums-of-scrums, etc, then the number of communication channels between them quickly becomes astronomical, and the situation actually gets much worse.
Also, note that if you’re already into scaling Agile, even an apparently huge scaled Agile programme is minute compared to an organisation even with a few thousand people, let alone a modern business or government behemoth.
In short, much like the human body cannot work without system-wide components such as the nervous and circulatory systems, so an organisation with tens or hundreds of thousands of members cannot work by simply allowing teams to jostle together in cost but inevitably localised interactions.
That’s not to say that we should prefer a typical modern bureaucracy to Agile after all. Waterfall long since put paid to that fantasy. And in fact the average modern bureaucracy is pretty inefficient on its own terms. Whatever the relative merits of Agile versus a formal bureaucracy, most modern organisations are pretty bad by both standards! A human body that reverted to the management methods most corporations seem to prefer would be dead in minutes!
Hence the present essay, which tries outlines the basics of a management that would plug the gaps in an all-Agile corporation, but at the same time be much better at the bureaucracy to which most contemporary corporations seem to be indissolubly wed. Few organisations really provide such a system, so teams or managers are seldom as efficient or effective as they could be. Instead, both individual managers and entire organisations operate in a half-light of incompetence, false assumptions, politics, evasion and falsification, personal disappointment and barely concealed crisis management. Propaganda levels are high, but real accomplishments and expectations are low.
What is a management system?
A ‘management system’ consists of the totality of organisations, functions, processes, roles and mechanisms teams need to carry out their work successfully. In a nutshell, it’s everything the organisation needs that the individual team can’t provide for it. The upper limit of the Agile team’s scope is probably the Release, so the rest of the management system needs to define whatever happens outside that space. This means that it needs to:
Create a system for creating, communicating and maintaining the corporate ‘Big Picture’:
- Create a vision for the organisation as a whole.
- Translate the vision into goals, objectives, targets and Products.
- Define how Products are identified, owned, funded and resourced managed, tracked and evaluated, and retired.
- Fully map the flow of information, decisions and outputs connecting the start and end points of each Product’s lifecycle.
Define how Products are managed:
- Specify the core Products that define what the organisation does.
- A ‘Product’ can be anything from a saleable commodity to an internal service (e.g., training, finance management, etc.).
- The sum of these Products should describe the organisation as a whole.
- The map of these Product should show how the whole organisation works.
- Define Product Plans for each Product that collectively deliver all of the organisation’s goals, objectives, etc.
- Set up the Product Backlog of concepts, epics, stories and themes that identify and define how the Product Plan will be delivered.
- Create the policy, governance, management and administrative model needed to execute, track and evaluate progress on the Product Plans, implement the Product Backlog and so realise the (corporate) vision.
Define the strategic purpose of individual Releases:
- Identify the Releases needed to implement the Product Plan.
- Build the work environment for integrating and aligning individual Releases.
- Assign Product Backlog items to individual Releases.
- Define the relationship between these Releases and the company’s overall strategy, including organisation, communications networks, shared business, systems and management architectures, shared repositories, and so on.
Define the processes for carrying out an individual Release:
- Define both management’s authority and responsibilities and the autonomy of individual Release teams.
- These should define a continuous, end-to-end sequence for the entire workflow, including continuous pipeline management, development and delivery, and deployment of working solutions.
- Define a range of generic, cross-team lifecycles and methodologies for executing Releases of different kinds.
- These define the ‘what’ and (in functional terms) the ‘how’ of detailed functions and tasks needed to carry out a Release.
- E.g, the iteration cycle, including specific tools and techniques such as prioritisation, estimating, managing velocity charts, and so on.
- Set processes, responsibilities and resources for adapting and improving the lifecycles and methodologies themselves.
- Regularly confirm that range of options and alternatives within any single process cover its users’ and the Product’s needs.
Provide the technical resources and materials needed to carry out Releases:
- Appropriately expert, disciplined, professional people.
- Tools and systems (pipelines management, computer-assisted development tools, test tools, etc.).
- ‘Delivery vehicles’ (environments, templates, storage, controls, etc.) and their corresponding support teams for common technical activities.
- Indirect technical support (R&D, standards management, quality assurance, tool development, etc.).
Create a working environment that actively supports Release management:
- Team support (direct technical support and SMEs, recruitment, training, repositories, tools deployment and expertise, coaching and mentoring, etc.).
- Administrative services (clerical support, data and record management, finance, analysis and reporting tools, etc.).
- Work facilities and infrastructure (space, hygiene, communications, security, etc.).
- Storage, configuration management and version control for interim and products and deliverables.
- Processes and mechanisms for reassigning facilities once the assignment is complete.
Create and manage generic standards and procedures:
- Standards for management and technical methods, tools and techniques.
- These should define functional outcomes that ensure that interfaces between Iterations, Releases and Products are effective, not detailed technical steps.
- Reference metrics.
- Processes for defining, creating and operating the team’s relationship to:
- Stakeholders, executive management, administration and internal compliance.
- External regulatory authorities.
- Third parties such as contractors, suppliers and consultants.
Looks like a checklist to me…
Managing the management system
Building a management system is not a one-off activity! In fact a management system should also be defined as a Product, and be invested in, evaluated and managed in the same way.
- The system would be defined and managed like any other Product.
- The organisation, processes, mechanisms and support of both the management system as a whole should be fully owned, funded and resourced.
- The owner should be accountable for the continuous improvement of:
- The usefulness of the system to its users.
- The value the system delivers to the organisation as a whole.
But the second and equally important element of how a management system is to be managed is to try to include as many Agile principles as possible, and to resist the idea that the only alternative is bureaucratic. For example, even is a scrum-of-scrums-of-scrums-of-scrums could not possibly be as usable as a single scrum or even a scrum-of-scrums, how often does it actually need to come into action? If it’s daily or even monthly, then no it won’t work. But quarterly? After all, how often do existing boards of directors convene to make such genuinely strategic decisions?
Conversely, there are plenty of tools that are capable of dynamically aggregating and sharing high-level data and presenting it in ways that inform local teams of at lest mid-range issues and risks, to which they are perfectly capable of adapting.
How the management system should work
A management system should be adaptable to the needs and preferences of individual Products, Releases, Iterations and teams. Very many current management practices already include components such as quality plans to deal with this situation, but a more sophisticated system would be truly systematic:
- The system should default to allowing the teams that use it to work as they want, and only introduce specific methods or constraints where there is a definite risk that would not otherwise be easily recognised or managed by local teams.
- The system’s processes should be directly traceable to (in fact directly interfaced with) business objectives, critical success factors, business constraints (product/service quality, operating costs, time-to-market, etc) and operational conditions.
- The system should match system processes and components dynamically to each team, Product and Release’s functional needs, not statically and according to the formal management system’s own rigid structure.
- E.g., by defining their entry and exit criteria, and setting parameters and tools teams can configure to meet their unique objectives, preferences and needs.
The performance and outcomes of individual team’s work should be recorded (e.g., via Retrospectives) in formats from which other assignments can benefit.
- This requires global repositories for organising and distributing timely and reliable information and decisions, accredited subject matter experts and information mining tools.
- Such an approach would also enhance scalability and the integration of multiple teams and Releases into strategic work programmes.
Out of such a system and the experience that it generates the management system itself would provide the organisation, methods, tools and techniques for abstracting and systematising a comprehensive model of the factors and forces affecting the organisation, and a system for their management and further development. Such a structure would:
- Allow both individual delivery teams and senior management to be as precise as they need to be to look forward and backwards over any strategically meaningful timescale.
- Would optimise the parameters by which existing management methods, tools and techniques were defined.
- Deal with any meaningful and credible future scenario.
And the answer is…
The keys to building a management system that is capable of correcting Agile’s shortcoming without crushing Agile itself are, I think:
- Default to Agile. If someone wants something else, let them make their case.
- Conversely, assume that you have recruited good people and make positive assumptions about what they are capable of.
- Assume you can get more out of investing more and more in intelligent, creative, dynamic people than in dumb procedures and mechanisms.
- Recognise that factors such as throughout, efficiency and productivity depends on the pleasure, satisfaction and contentment humans beings get out of their lives – not least at work – every bit as much as they depend of technical factors.
- Make the management system into a tool its users can use, not a sausage machine through which they are forced.
- Localise the management of the system as much as possible.
- Conversely, even if you do need to build a quasi-hierarchical system, never allow that to translate itself into unchallegeable power. Power without genuine authority is despotism, not management, and no one ever ran an efficient despotism.
- People really are your greatest asset. Well, that and money.
Of course, even the most sophisticated management system cannot, on its own, create the vision needed to see where an organisation should be going, but the kind of system that is described above would surely be able to integrate complexly interacting strategies, goals, processes, systems, and so turn any rational vision into reality. It would be a lot happier place to work in.