How good is Agile really?

How good is Agile really?2017-10-12T22:05:26+00:00

Perhaps the most fundamental question one can ask about Agile is, How good is it really?

If you want the facts – or need to make the case for Agile – this page of assorted data and claims provides more wide-ranging evidence.

Note that much of this data comes from interested parties, especially Agile consultancies and tool suppliers. None has their data been has been validated independently. Likewise terms are frequently undefined and sometimes moot.

High-level summary claims

There are plenty of general (but fairly precise) claims about how good Agile is. They aren’t usually by independent parties or at all explicit and seldom about supporting evidence, so take them for what they are worth.

QuoteSource
“During their first year of using Scrum, Salesforce.com reports releasing 94% more features, delivering 38% more features per developer, and providing over 500% more value to its customers compared to the previous year.”Mike Cohn, Mountain Goat Software (Agile consultants to Salesforce.com).

The value of Agile

Does Agile give people what they want?

Perhaps the simplest question one can ask of any lifecycle of methodology is, ‘Does it give people what they want?’. That is, if you compare what organisations are aiming for when they adopt Agile with the areas they think Agile has contributed most, are they the same? Judging by VersionOne’s research, yes, they are.

Their respondents’ aims and what they felt was actually achieved are shown in this table.

PriorityAimAchieved
Accelerate product delivery59%77%
Manage changing priorities56%87%
Increase productivity53%84%
Enhance software quality46%78%
Enhance delivery predictability44%79%
Improve business/IT alignment40%75%
Improve project visibility40%82%
Reduce project risk38%76%
Improve team morale26%79%
Improve engineering discipline25%72%
Increase software maintainability22%68%
Better manage distributed teams20%59%

This graph maps how well their aims corresponded to Agile’s achievements:

Agile gives people what they want
Clearly, the more organisations wanted something from Agile, the more likely they were to get it.

Perceived benefits of Agile practices

Murphy et al. report that Agile and non-Agile development teams expect the following perceived benefits from adopting Agile practices:

Agile - Perceived Benefits

Source: Murphy, B., Bird, C. Zimmerman, T., Williams, L., Nagappan, N., and Begel, A. Have Agile Techniques Been the Silver Bullet for Software Development at Microsoft?, International Symposium on Empirical Software Engineering (ESEM) 2013, Baltimore, MD, pp, 75-84.

Perceived problems with Agile practices

Murphy et al. report that Agile and non-Agile development teams expect the following problems from adopting Agile practices:

Agile - Perceived Problems
Source: Murphy, B., Bird, C. Zimmerman, T., Williams, L., Nagappan, N., and Begel, A. Have Agile Techniques Been the Silver Bullet for Software Development at Microsoft?, International Symposium on Empirical Software Engineering (ESEM) 2013, Baltimore, MD, pp, 75-84.

DSDM

DSDM is an early flavour of agile, specifically designed to deal with corporate environments. For data on how well it performs, see here.

Agile practices & disciplines

Collocation

To quote the abstract from a 2002 IEEE study of collocation:

In a field study conducted at a leading Fortune 100 company, … the collocated projects had significantly higher productivity and shorter schedules than both the industry benchmarks and the performance of past similar projects within the firm. The teams reported high satisfaction about their process, and both customers and project sponsors were similarly highly satisfied… [B]eing ‘at hand,’ i.e. both visible and available, helped them to coordinate their work better and learn from each other. Radical collocation seems to be one of the factors leading to high productivity in these teams.

Source: ‘Rapid software development through team collocation’, IEEE Transactions on Software Engineering, Volume: 28 Issue: 7, July 2002, pp.671-683.

Based on self-reported ‘success’, this table illustrates more precisely how effective collocation can be. (NB: These projects are ‘still in the pilot project phase’.)

Success rate
Everyone, including stakeholders, in the same work area.83%
People in different cubes, on different floors or in different buildings, but could easily get together.72%
Part of the team at a significantly different location, such as offshore.60%

Source: Ambler, S.W. Has Agile peaked?‘, in Dr Dobbs, May 7, 2008.

Sustainable pace

You can define productivity with a simple equation:

Productivity = (Work spent adding value) – (Effort spent injecting and removing defects) – (Effort spent removing Technical Debt) – (Lost opportunities)

One of the major causes of a) defects, b) bad design decisions and c) lost opportunities is overwork. The causes of overwork are many, varied and pretty inexcusable, with the delivery team frequently paying for mistakes made by other people. But whatever the cause of long hours, they are not, in themselves, going to solve any problems. The fact is, long hours reduce productivity – not only by making people less efficient but, if they become too much the norm, by actively reducing the quality and rate at which product goes out the door to the point where productivity actually falls.

Generally speaking, driving up the working week to above, say 60 hours, may, temporarily boost productivity slightly. But the boost only lasts 3 to 4 weeks, and then the curve turns firmly downwards – eventually to the point where you produce less than you would in a ‘normal’ 40 hour week.

Here’s a handy graph Daniel Cook constructed out of research data to illustrate this point:

Productivity vs Hours

The recovery time is generally at least as long as the period where overall velocity is boosted. That’s not to say you can’t ever raise production through extra hours; it’s just surprising how quickly the benefits turn to costs.

Cook also summarises data suggesting that game developers go through the same rise-and-rapid fall:

Productivity vs Hours among Game Developers

What you lose when you start to push up the hours are creativity and problem-solving ability – you lose these much quicker than simply the ability to do, say, manual labour. So as Cook puts it, ‘Grinding out problems by working longer on average results in inferior solutions.’

People who do a lot of overtime claim they’re more productive, and can work long hours without losing productivity. Unfortunately one of the first things you lose on overtime is your sense of how much work you’re doing! This is a more realistic picture:

Productivity, Perceived and Real

Again, as Cook puts it:

“This behavior is fascinating to observe. Zombies stumble over to their desk every morning. Tempers flare. Bugs pour in. Yet to turn back would be a betrayal.”

Hence the value of setting a sustainable pace.

Historical research

Early studies – to 2000

Weill (1998) (Harvard)

  • 20% improvement in communication, quality, and cycle time.
  • 13% increase in strategic alignment, sales, and revenues.
  • 54% reduction in product and service development costs.

Thomke (1998) (Harvard)

  • 50% reduction in development effort.
  • 55% improvement in time to market.
  • 925% increase in the number of changes allowed.

MacCormack (1998) (Harvard)

  • 48% productivity increase over traditional methods.
  • 38% higher quality associated with more design effort.
  • 50% higher quality associated with iterative development.

Fichman (1999) (Boston)

  • 38% reduction in time to produce products and services.
  • 50% time to market improvement.
  • 50% more capabilities delivered to customers.

Source: Rico, D. F. (2008). The Business Value of Using Agile Project Management for New Products and Services.

Studies 2001 – 2010

Improvement with Agile
StudyRespondentsProductivityCostQuality
Johnson (2003)13193%49%88%
Barnett (2006)40045%23%43%
Begel (2007)49214%16%32%
Rico (2007)25081%75%80%
Ambler (2008)64282%72%72%
Wolf (2008)20778%72%74%
Hanscom (2008)3,06174%38%68%
Average67%49%65%

Source: Rico, D. F. (2008). The Business Value of Using Agile Project Management for New Products and Services.

“In 2008, the University of Maryland developed a database with over 153 data points on the costs and benefits of agile project management from 72 studies.”

“Results:

MetricsAgileTraditionalDifference
Cost Reduction29%20%9%
Schedule Reduction70%37%33%
Productivity Improvement117%62%55%
Quality Improvement74%50%24%
Customer Satisfaction Imp70%14%56%
Return on Investment2,811%470%2,341%

“A similar study of the costs and benefits of agile project management was conducted by a market-leading firm. Data was gathered from 23 agile projects and compared to a database of 7,500 traditional ones. The vendor reported that agile project management lowered costs by 61%, reduced schedules by 24%, improved quality by 93%, and improved productivity by 39%.”

Source: Rico, D. F. (2008). The Business Value of Using Agile Project Management for New Products and Services.

Self-report on ‘How effective agile teams are in practice’:

FactorBetterSameWorse
Productivity82%13%5%
Quality77%14%9%
Stakeholder Satisfaction78%15%7%
Cost37%40%23%

Impact of Agile

Source: Ambler, S.W. ‘Has Agile peaked?‘, in Dr Dobbs, May 7, 2008.

Leave A Comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Toggle Sliding Bar Area
Want to do more than just build systems?