Why is planning a problem?

A major difference between Agile & traditional (e.g., waterfall) lifecycles is their approach to planning.

The traditional approach

Traditionally, a plan is a formal description of the detailed activity needed to deliver an agreed outcome.

It typically includes:

  • Outputs.
  • Tasks.
  • Schedule.
  • Dependencies.
  • Effort.
  • Resources.
  • Status.

… plus (often) a number of organisation-specific fields.

Agile does all that, but with few of the conventional control structures.

The traditional plan is also typically owned by the project manager (not the team) and under some level of change control. That is:

  • It can only be changed with the authority of a formal authority such as a steering committee or project board.
  • The change process follows a strict procedure, often quite slow, that requires specific meetings, analysis, etc.
  • The changes are imposed on the team, often without any opportunity (and very seldom with any right) to comment before the change is finalised.

The real nature of software development

The traditional approach to delivery treats software development like building a bridge.

  • You can – and must – define what you’re going to build in advance.
  • Once you’ve defined that, it won’t change while you build it.
    • And no-one will tell you to move the half-finished bridge 50 metres upstream!
  • If change is unavoidable, the change process is:
    • Thoroughly understood.
    • Rigorously controlled.
    • Fully accepted.
    • Based on proven procedures & careful analysis.
  • In such an environment, it’s realistic to expert comprehensive planning to work.

Unfortunately software delivery isn’t like building a bridge. In fact, in many ways it’s exactly the opposite of other kinds of engineering.

  • It’s extremely hard to define what you’re going to build very far in advance.
  • Even if you do that, the requirements will certainly change while you are implementing them.
    • This is a good thing for the organisation, but disastrous when weakly managed.
    • And someone will tell you to ‘move the half-finished bridge 50 metres upstream’!
    • So ‘scope creep’ is absolutely the norm.
  • The change process itself is seldom rigorous, and equally seldom followed.
    • There is often no shared culture of rigour or control between IT & the business.
    • IT’s poor record of delivery on time or on budget undermines any attempt to create such a culture.
  • In such an environment, it’s wholly unrealistic to expect detailed end-to-end planning to work.

It’s also comforting to senior management to think they can see so far ahead, of course. Planning is always a highly political activity!

Why so much change?

Change is often a good thing. But why is change so common, especially in IT?

  • There are many business reasons for change to happen.
    • Markets change.
    • Competition creates different pressures.
    • Priorities change.
    • Ideas mature.
    • Organisations (& stakeholders) change.
  • The very power of information technology also makes change more common.
    • The flexibility of IT is widely recognised.
      • Even if it isn’t well managed.
    • The invisibility of code & data means that onlookers (e.g., stakeholders) do not appreciate how complex or large-scale a software system is.
      • Or, therefore, how hard, risky & expensive it is to change.
    • New technologies emerge quickly & unexpectedly.
      • Creating new opportunities – and threats.
  • The relationship between business & IT is also an issue.
    • The relationship is typically one-sided –
      • The business defines the requirements,
      • The business owns the budget.
      • The business typically has much more authority and prestige.
    • So its decisions & demands are hard to resist.

The evidence for planning

Over the last decade or so, a great deal of empirical evidence has been gathered about planning for software and systems delivery. In summary:

  • The farther things are away…
    • E.g. a 12-24 month project or programme.
  • …the harder they are to define…
    • E.g., detailed requirements, business value, designs, costs…
  • …and the more likely they are to change before we get to them…
    • E.g., changing markets, technology, organisation, business processes, architectures…
  • So as we look farther ahead, detailed planning becomes:
    • Unlikely to be effective or reliable.
    • Equally unlikely to still be relevant when the time comes to do the work.
    • But very likely to create a dangerously false sense of control.

Based on this experience, Agile emphasises ‘just-in-time’ (JIT) planning. That is, Agile argues that we should plan only when planning is needed – and where our plans will be believable.

Recent trends in planning

Over the last decade or so, even waterfall developments have begun to adopt a more realistic approach to planning. Broadly speaking , maximising accuracy is no longer being approached by aiming for an illusory precision. Rather:

  1. Overall goals & objectives are set.
    • Overall outcomes for the organisation.
    • Schedule & approximate cost/benefit budgets.
  2. But detailed planning is limited to the next phase.
  3. As each phase ends, a new investment decision is made, strictly limited to the next phase.
    • That is, investment boards decide only whether it’s worth executing the next phase.
  4. This is supported by:
    • A clear definition of what business result is needed to justify further investment.
      • That is, cost & benefit.
    • Best-and worst-case estimates for actual outcomes.
    • A clear decision-making process based on these numbers.
  5. And the entire investment is reviewed, adapted & potentially cancelled at the end of each phase.

Modelling the problem

There are various models of the problem planners face.

The Cone of Uncertainty

This model is the so-called Cone of Uncertainty, as applied to investment.

Cone of Uncertainty

That is:

  • The axes represent time (e.g., a complete release) and money (i.e., the cost of delivery or the value of the benefits delivered).
  • The horizontal line marked ‘Return required to justify investment’ shows how much value must be delivered if the investment is to meet its minimal target.
    • This will normally be considerably more than simply covering the cost of development.
    • It is also likely to include a profit margin, an allowance for risk, and other items that define the target ‘return on investment’ (ROI).
  • The two curves show how unpredictable the cost of delivery is likely to be. That is:
    • If the original estimates of the best and worst case for costs at the start of the project are as shown on the far left (‘Original plan’), experience shows that they are very likely to grow as time goes on.
    • So, at ‘Current minimum & maximum estimates’, the slightly expanded range of estimates is shown.
    • But the evidence – which is very well established and spans the whole of software development – suggests very strongly that the final cost is likely to come anywhere in the very wide range shown on the right-hand edge of the graph.
  • Comparing these lines to the line marked ‘Return required to justify investment’, it is clear that, even if the current costs are still well inside the target, in many situations they will eventually rise to a point where the investment is no longer valuable.
    • If you believe that they eventually fall into the zone marked ‘Continue’, this means that you expect the investment to pay off.
    • If you believe that they eventually fall into the zone marked ‘Cancel’, this means that you expect the investment to fail.

The Arrow of Confidence

A second model is the Arrow of Confidence.

The Arrow of Confidence

In this model:

  • The arrow points from ‘Now’ to the future – an unspecified date simply marked as ‘Eventually’.
  • At the far end of this arrow is the completed Product Vision – the definition of what, ultimately, the Product Team will build.
    • The Product Vision is seldom static and even more rarely final.
  • Along the arrow there are four levels of visibility:
    • The predictable level, generally limited to the current iteration, and providing us with reasonably reliable knowledge of the future.
      • At the outer limit of this level lies the ‘horizon of confidence’ – which is to say, the limit to our ability to speak confidently about the future.
    • The uncertain level, extending over the rest of the current release, over which period we may have realistic expectations, but little real knowledge.
      • At the outer limit of this level lies the ‘horizon of prediction’ – which is to say, the limit to how far it makes sense to even try to predict the future.
    • The unknown level, consisting of the early part of the rest of the product’s life (i.e., the next few releases), about which we may have plausible beliefs but few credible expectations and no true knowledge.
      • This area represents our ‘field of aspiration’ – that is, the area where it is realistic to imagine what we might possibly do, but nothing more.
    • The more distant future, beyond the next few releases, is not so much unknown as actually unknowable, and any claims we may make about it are little more than guesses and aspirations.
  • There is also a vertical dimension to this model. This distinguishes between:
    • The top layer – marked simply with ‘What’ – refers to the starting point of any development, namely the story.
    • The story represents a kind of ideal, a theory of what will be built – but it is not yet the practical result.
    • The proof of our story-theory is the working software, of course – the ‘How’ of our diagram. This is the real test of our work, and until it is actually built and delivered to users  and those users are happy with the results, this is another dimension of confidence – or lack of confidence, of course.
  • As the image suggests, we can think about these spaces in terms of the conventional Red-Amber-Green colours for status and progress.
    • At the start of work, the only thing we can be truly confident of is the story – and even then it is only in the sense that we all recognise that every story is provisional!
    • Uncertainty extends in two quite different directions – not only into the future, but also towards the problem of actually building working software.

Presented with this model, it is easy to see why it is that, beyond the current iteration or, at best, release, planning is a difficult and risky activity. Agile’s response to this is not (as it often is in a more traditional planning environment) ‘plan harder’ but to accept the uncertainty and plan only as much as makes sense.

The Agile approach

The Agile approach to planning takes account of all these lessons.

The Agile model of planning

Agile planning is based on matching the level of plan granularity and commitment to how close the events that need to be planned really are. That is, the depth & detail of Agile plans reflects the proximity & stability of what is being planned. Anything too risky or uncertain to plan is simply excluded from the work backlog.

The overall Agile approach is summarised in the following diagram:

Agile Planning Model

The model as a whole is recursive.

  • The top-level Product Plan operates over the longest timescale, though it is seldom more than 9-12 months. Beyond this time, all projections tend to be increasingly speculative and unrealistic.
  • Apart from the ultimate Product Vision (and other things), the Product Plan sets out the overall Release Schedule. The releases this schedule identifies are then defined individually in Release Plans.
    • Each Release Plan typically extends over 1-3 months, and identifies the overall scope and goals of the iterations in that release and defines the tasks that need to be carried out to prepare the release (‘Iteration 0‘).
  • At the lowest level, each iteration is defined by its own Iteration Plan.
    • This is seldom a separate document, but rather consists only of an agreed list of stories to be delivered during that iteration, plus a number of additional activities for setting up and supporting the iteration.

So at any given moment the Iteration Plan is being executed with the Release and Product Plans providing the context and defining the overall programme of work related to this product.

Agile vs traditional planning

To summarise the different levels of planning within the Agile model:

  • The Product Plan defines a schedule for the start & end of whole releases.
    • Typically no more than 3-4 releases.
  • Each Release Plan breaks a single release into small, easily understood iterations.
    • This provides a high-level ‘plan’.
  • At the lowest level, the Iteration Plan then specifies the detail that is needed for the realistically foreseeable future.
    • This is the only level at which very detailed planning occurs.
    • … or is credible.

How then is the level of control achieved by traditional planning also achieved by Agile? The answer is summarised in this table:

Traditional controlAgile approach
  • Defined in stories.
  • Detailed for current iteration only.
  • Tracked via the task board.
  • Start/end goals summarised in Release Plan.
  • Detailed for current iteration only.
  • Implied by story or Task Board.
  • Tracked via burn charts.
  • Agile encourages independent stories.
  • Remaining dependencies tracked by Task Board, etc.
  • Dependency Log, if required.
  • The Core Team is fixed, though the Extended Team may vary.
  • Each Core Team member is 100% committed to each release.
  • The team commits only to what it can deliver within its own resources.
  • Detailed tasks are agreed by the team themselves.
  • Daily tracking of results.
  • Frequent Showcases.

So a detailed plan isn’t needed, yet the same level of control is achieved, without sacrificing Agile’s principles of flexibility and adaptability.

The planning process

Product, Release & Iteration Planning

For details of each level of management and the corresponding approach to planning, click below:

Product ManagementProduct Planning
Release ManagementRelease Planning
Iteration Management Iteration Planning

Who plans?

In traditional environments, the project manager, PMO or even a professional planner creates & owns the plan. In Agile, this would be unusual, if not counter-productive.

For Agile, the best people to do the planning are the team.

  • They have the knowledge of the work.
  • They will understand the risks & dependencies.
  • They are motivated to make it realistic.
  • They have to accept the plan as credible if they are to commit to the release.

The Release Manager and Iteration Lead (or a professional planner) are useful resources in this exercise:

  • To provide experience on what is likely to be needed.
  • To increase awareness of the external environment with which the team has to interact.
    • Showcases, events & dependencies.
    • PMO, stakeholders, etc.
    • Corporate events, requirements, etc.
  • To perform the mechanical tasks – e.g., maintaining the Backlog & Story/Task Board.

Planning tools

The Gantt chart may be the single most powerful management tool of the last century. But if you’re working with Agile, its value is reduced.

  • A high-level Gantt chart is a useful tool for summarising a Product or Release Plan.
    • Start & end.
    • Dependencies.
    • External events.
    • Pilots.
    • Release dates.
  • But at iteration level:
    • It aims to define & control rigorously factors that Agile believes should be left to the team to decide as work proceeds.
      • E.g., resource allocation, tasks, etc.
    • So it tends not to represent what the team is really doing.
      • And so creates an illusion of control.
      • Agile is strong on control – just not this type of control.
    • A Gantt chart is relatively inflexible.
      • Especially at the hour-by-hour scale at which Agile projects can change.
    • It is usually inaccessible to most of the team.

At iteration level, Story & Task Boards are generally more effective.

NB: A Gantt chart can be useful:

  • As a record of what actually happened.
  • To support & refine velocity calculations.

Leave A Comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Want to do more than just build systems?