Continuous Delivery

Voke report: Agile delivers higher customer satisfaction and quality

Published 25 July 2012

Translations: 한국말

There's been a lot of controversy generated by Voke's Agile Realities report. SDTimes asked me to comment for their article covering the report, and so I got to read it in full. Obviously Voke want your money to read the whole thing so I am constrained by fair use as to what I can reproduce in this review, but my general conclusion is that the report represents a hatchet job on agile whose most important conclusions aren't sustained by the (interesting and revealing) data they have gathered.

From the press release, we can discern that Voke draws the following conclusions from their research:

  • "The average cost of software projects is rising dramatically, in spite of smaller teams working much shorter durations" - it is implied that this is due to agile adoption.
  • "Survey participants shared their experiences with Agile, expressed confusion, and identified more real world challenges than benefits"
  • "Given the impact of catastrophic software failures on the brand, we should be witnessing a movement toward increased quality." - the implication being that Agile produces lower quality software.
  • Executives are buying agile because they think it means business agility. Voke, as we will see, think that this is nothing less than a bait-and-switch exercise by the Agile community.

In the actual report, Voke make three further high-level claims. First, they say Agile is effectively a Trojan horse for developers to gain control over the software delivery process, so they can avoid doing boring things like documentation, design, and planning. Second, they say it is primarily a vehicle for consultants like me to sell services, training, and books (curiously, they omit that analysts do quite well out of it too). Finally, they claim that the useful core of Agile is nothing more than rapid prototyping - which is nothing new. So it's fair to say Voke aren't keen on Agile (and while I am paraphrasing here, I am attempting to faithfully preserve the tone of the original).

However, while they say - correctly in my opinion - that companies are confused by Agile, Voke certainly aren't in the business of helping. Voke make basic mistakes (or misrepresentations) in their opening discussion of Agile, Lean and Devops. The most egregious of these is around the role of quality in these movements, which they claim exclude QA and thus represent a move away from increased quality. In support, they cite the fact that the Agile Manifesto and the name "Devops" nowhere mention quality1. This ignores the fact that building quality in is central to every good discussion of lean and agile development. Indeed not only do agile methodologies emphasize the importance of cross-functional teams (including business, QA and operations): Agile practitioners pioneered engineering practices such as test-driven development, continuous integration and refactoring which (as the report acknowledges) substantially reduce the cost of discovering and fixing defects.

Let's address another claim: that the cost of software development has increased substantially since 2008 - the date of an earlier "Agile Realities" report. What their numbers in fact show is that the average2 project cost has stayed the same whereas the average project duration has decreased. This subtlety aside, Voke are making the same mistake as many enterprises that treat IT as a cost center: focusing on development cost instead of return on investment. As noted by Douglas Hubbard, author of How to Measure Anything, "even in projects with very uncertain development costs, we haven't found that those costs have a significant information value for the investment decision... The single most important unknown is whether the project will be canceled… The next most important variable is utilization of the system, including how quickly the system rolls out and whether some people will use it at all." Thus getting a quality product that incorporates customer feedback for the same price as getting a mediocre one, but in a third less time (the numbers reported by Voke) represents an incredible deal. According to Voke's own numbers, technology companies using Agile methodologies - uniquely - had zero projects abandoned after development.

The real news - which Voke hide away towards the end of the report on p32 - is that technology companies and enterprises both report higher levels of customer satisfaction and lower levels of unfixed critical defects when their projects use Agile methodologies. So much for Agile's "quality problem". However enterprises do worse than technology companies: a large majority of technology companies report customer satisfaction when using agile methodologies, compared with a smaller majority of enterprises3.

Unfortunately the authors do not address the wider macroeconomic issues: in the period from 1958 to the present, the lifetime of an S&P 500 company has shrunk from 61 years to only 18 years. This period is contemporaneous with the rise of software development and the fall of traditional manufacturing. Today enterprises are being challenged by technology-powered start-ups which are able to leverage Lean and Agile methodologies to create a competitive advantage (most VCs wouldn't consider funding a non-Agile start up). Thus the failure of many enterprises to implement Agile at scale, noted by the authors, is indeed a serious problem. However this problem cannot be addressed by recycling FUD from ten years ago, such as claims that Agile methodologies don’t believe in any documentation or design.

The data in the report thus makes for interesting reading - although I would have liked to see more statistical analysis, correlation, and some graphs. Unfortunately the analysis is mostly worthless, except as an exercise in twisting the data to suit the authors' agenda.

1 Although the word "quality" doesn't appear in the Manifesto, the 9th principle states that "Continuous attention to technical excellence and good design enhances agility." One of Voke's claims is that the Agile Manifesto is an inadequate guide to implementing Agile. While this is true, it was never the intention of the Manifesto to define how to implement Agile, or even to be a complete guide to Agile. The intention was to abstract out the core message of a number of "lightweight" methodologies that were gaining traction in 2001 (Martin Fowler and Jim Highsmith provide some history on the Manifesto, including a discussion of quality, here). It is of course these methodologies that provide the level of detail that Voke's authors are missing. As Martin Fowler says, "Agile is a mind-set, described by values and principles - not what most people would understand as a methodology that would include practices and deliverables" - more in his RigorousAgile bliki entry.

2 Voke don't say what kind of average they have used, nor do they provide standard deviations or correlate the data in any way. There aren't even any graphs.

3 I'm not going to give the actual numbers here because I think it crosses the line of fair use - you should buy the report if you want the full scoop.