You’ve probably heard of Gene Kim, Kevin Behr and George Spafford before. They are the three amigos responsible for The Visible Ops Handbook, which can be found in the book pile of every good IT operator. Their new book, The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win, follows the format of Eliyahu Goldratt’s classic, The Goal.

Told from the perspective of newly-minted VP of IT Operations Bill Palmer, it describes the turnaround of failing auto parts company Parts Unlimited. This is to be achieved through the delivery of the eponymous Phoenix Project, a classic “too big to fail” software project designed to build a system which will revive the fortunes of the company.

To quote (p51):

The plot is simple: First, you take an urgent date-driven project, where the shipment date cannot be delayed because of external commitments made to Wall Street or customers. Then you add a bunch of developers who use up all the time in the schedule, leaving no time for testing or operations deployment. And because no one is willing to slip the deployment date, everyone after Development has to take outrageous and unacceptable shortcuts to hit the date.

The results are never pretty. Usually, the software product is so unstable and unusable that even the people who were screaming for it end up saying that it’s not worth shipping. And it’s always IT Operations who still has to stay up all night, rebooting servers hourly to compensate for crappy code, doing whatever heroics are required to hide from the rest of the world just how bad things really are.

Part One of the book describes in loving detail the enormous clusterfuck pie that is baked from these ingredients. The pie is spiced with an internal Sarbanes-Oxley audit which reveals 952 control deficiencies, an outage of the payroll processing system, and various other problems that conspire to deepen the woe of the operations group, all of which are clearly drawn from the deep well of the authors’ real-life experiences.

Apart from the main characters – our hero Bill, his boss Steve, and the evil villain Sarah – The Phoenix Project features a delightful rogues’ gallery which anyone working in an enterprise will recognize:

• Brent Geller, the boy wonder whose encyclopedic knowledge of the company’s Byzantine IT systems means that his involvement is necessary to get anything done.
• Patty McKee, the Director of Support who runs a change management process so bureaucratic that everybody bypasses it.
• John Pesche, the black binder wielding Chief Information Security Officer whose constant meddling under the guise of improving security has turned him into a pariah.

The second part of the book details how the IT group is reborn from the ashes of the Phoenix Project into a high-performing organization that is a strategic partner to the business. This is achieved through the application of a heavy dose of lean thinking (including continuous delivery) administered by Erik, a mercurial IT and manufacturing guru Steve is courting to join the board. The book does an excellent job of showing – as well as telling – how to apply the concepts (and the effect of doing so) in an enterprise with plenty of technical debt. Perhaps the most eyebrow-raising part of this section is the way in which John has his soul mercilessly crushed to the point where he goes on a multi-day drinking spree before he is rehabilitated towards the end of the book (he is a phoenix too).

John’s narrative arc is just one example of how the book also succeeds as a novel. It’s gripping, with moments of drama and high emotion, as well as some great one-liners. There was even one point when I teared up (bear in mind that I also cried during Forrest Gump – unlike the book’s central characters, I did not serve in the armed forces).

Nobody who has read The Goal will miss The Phoenix Project’s similarity in terms of style and plot. Perhaps my favourite thing about the book’s pedagogical style is the way Erik (like Jonah in The Goal) uses the Socratic Method to give Bill the tools to solve his problems by himself. Of course this learning process is fictional, but it means you get to see Bill struggling with the questions and trying things out.

It remains to be seen whether readers of the book will be able to apply these techniques as successfully as Bill without a real Erik to guide them. But of course, this is a limitation of any book. If I had one criticism it’s that unlike real life, there aren’t many experiments in the book that end up making things worse, and it’s this process of failing fast, learning from your failures, and coming up with new experiments that is instrumental to a real learning culture.

One important point worth noting if you are working in an organization like Parts Unlimited is this: the IT department’s rebirth is only possible because of the Titanic proportions of the disaster that unfolds in Part One. For management to truly embrace change, a compelling event or a teachable moment (i.e. a Charlie Foxtrot) is required. Unless your organization faces the same existential threat that Parts Unlimited does, you’ll have a much harder time convincing people they should adopt the tools described in the book.

Overall, The Phoenix Project is a fantastic read. It’s entertaining, cathartic, inspirational and informative. If, like me, you have an enormous backlog of books (and more work in process than you’d like) I suggest giving yourself a break and putting this one to the top of your list. It’ll only take you a day or two, and despite its conceptual density it will leave you feeling refreshed and energized with a bunch of new ideas to try out. The Phoenix Project deserves to be read by everyone who works in – or with – IT.

Why Continuous Delivery Works

Taleb shows why the traditional approach of operations – making change hard, since change is risky – is flawed: “the problem with artificially suppressed volatility is not just that the system tends to become extremely fragile; it is that, at the same time, it exhibits no visible risks… These artificially constrained systems become prone to Black Swans. Such environments eventually experience massive blowups… catching everyone off guard and undoing years of stability or, in almost all cases, ending up far worse than they were in their initial volatile state” (p105)¹.

This a great explanation of how many attempts to manage risk actually result in risk management theatre – giving the appearance of effective risk management while actually making the system (and the organization) extremely fragile to unexpected events. It also explains why continuous delivery works. The most important heuristic we describe in the book is “if it hurts, do it more often, and bring the pain forward.” The effect of following this principle is to exert a constant stress on your delivery and deployment process to reduce its fragility so that releasing becomes a boring, low-risk activity.

Antifragile Systems

Another of Taleb’s key claims is that it is impossible to predict “Black Swan” events: “you cannot say with any reliability that a certain remote event or shock is more likely than another… but you can state with a lot more confidence that an object or a structure is more fragile than another should a certain event happen.” (p8). Thus we need “to switch the blame from the inability to see an event coming… to the failure to understand (anti)fragility, namely, ‘why did we build something so fragile to these types of events?’” (p136).

Unlike risk, fragility is actually measurable. How do we measure the fragility of the systems we build? We try to break them, using techniques such as game days and systems like chaos monkey. The systematic application of stress to your systems is essential – not just to ensure your systems are antifragile, but to develop the muscles of the people who create and maintain them through constant practice. After all, it’s the combination of the system and the people who build and run it that has the quality of antifragility.

In this context, an important quality of legacy systems is their fragility. Legacy systems that aren’t touched for a long time will turn into fragile “works of art”: changing them is considered risky, the number of people who understand the system decreases with time, and their knowledge atrophies from lack of exercise.

How do we create antifragile systems? Apply stress to them continuously so we are forced to simplify, homogenise, and automate.

Antifragile Organizations

We can measure the fragility of an organization by how long it takes before it liquidates its assets. Deloitte’s Shift Index shows that the average life expectancy of a Fortune 500 company has declined from around 75 years half a century ago to less than 15 years today.

Start-ups are notoriously fragile. But the ones that survive and grow turn into something potentially more dangerous – robust organizations. The problem with robust organizations is that they resist change. They aren’t quickly killed by changes to their environment, but they don’t adapt to them either – they die slowly. We see this effect all the time – changing the culture of an established organization is incredibly hard.

Antifragile organizations are those that have a culture that enables them to learn fast from their environment and adapt to it so they can take advantage of volatility. Here are some characteristics of antifragile organizations:

• Systems thinking. Everybody in the organization knows the goals of the organization and makes sure their work is directly contributing towards these goals.
• Theory Y Management. Management needs to assume employees are self-motivated and will be able to learn how to solve problems themselves. Organizations need to make sure they hire antifragile people who will thrive in this environment. As Daniel Pink’s Drive points out, giving your employees autonomy, purpose, and the opportunity to learn and master new skills is what stops them from quitting, thus increasing the antifragility of your organization.
• Continuous experimentation. As described in Toyota Kata, good management knows that the best solutions come from the workers. They create an environment in which practitioners are able to run experiments to learn as rapidly as possible. The feedback loops in command and control organizations are too slow for them to adapt effectively.
• Disruptive product development. Antifragile organizations aren’t content with stress generated by their environment. Like humans exercising, they also try and disrupt themselves (the organizational equivalent of a game day). For example, Amazon cannibalized its own business, creating the Amazon Marketplace and the Kindle. Apple is cannibalizing its Mac business with the iPad. Fragile organizations resist disrupting their own product lines, as Toshiba did at first with flash memory. If you do a good job at this you never need to worry about the competition – you’ll always beat them to it.

Fragility and Agility

As Taleb points out, “antifragility is desirable in general, but not always, as there are cases in which antifragility will be costly, extremely so. Further, it is hard to consider robustness as always desirable—to quote Nietzsche, one can die from being immortal.” (p22) Of course working out where on the spectrum you want your systems and your organization to lie is an art, and the great artists are those that know how to build systems, organizations, and products simply, quickly and cheaply so that they are antifragile with respect to our biggest enemy: time. How do they do that? Using the same heuristics described in “antifragile organizations”, above, which closely mirror the Three Ways of Devops.

As I read Antifragile, it reminded me of something I read a number of years ago: Kent Beck and Cynthia Andres’ Extreme Programming Explained. The subtitle? Embrace Change. It strikes me that the concept of antifragile is what we were aiming for with agile the whole time: building systems (including human systems – organizations) that benefit from volatility.

Endnotes

Thanks to Badrinath Janakiraman for feedback on an earlier draft of this post.

1 He is talking about financial markets, which are rather less fragile than IT systems, hence his rather generous “years of stability”

Elisabeth has held positions as a tester, developer, manager, and quality engineering director in a variety of companies ranging from small startups to multi-national enterprises. She has been a member of the Agile community since 2003, served on the board of directors of the Agile Alliance in 2007/2008, and is currently one of the co-organizers of the Agile Alliance Functional Testing Tools program. Elisabeth won the prestigious Gordon Pask award in 2010 and is the author of two books: There’s Always a Duck and the forthcoming Explore It! Reduce Risk and Increase Confidence with Exploratory Testing published by the Pragmatic Programmers. She is the founder of Quality Tree Software, runs Agilistry Studio, and is the creator of entaggle.com. You can find her at testobsessed.com and @testobsessed

In the video, Elisabeth answers the following questions:

• What’s been your experience of the evolution of the testing and QA role in the IT industry?
• Why doesn’t separation of duties between engineers and testers work? (01:49)
• What are the problems with the hierarchies you sometimes see within testing organizations? (04:30)
• What’s wrong with the approach of writing and executing manual test scripts? (07:53)
• Does record and playback help to solve some of these problems? (08:48)
• So the current paradigm for building quality in to software is acceptance-test driven development. Talk us through ATDD. (10:04)
• What does what in ATDD? (11:19)
• What proportion of time is spent on production code vs test code? (14:15)
• How do you get from a typical “enterprise” testing strategy to ATDD? (17:30)
• When implementing automated acceptance tests, where do you start? (21:31)
• How has continuous deployment changed the way you approach software delivery? (23:43)
• How do you go from releasing once every 3-6 months to releasing once a day? (26:36)

If you can’t see the content, go straight to the video on YouTube.

John Allspaw has worked in systems operations for over fourteen years in biotech, government and online media. He started out tuning parallel clusters running vehicle crash simulations for the U.S. government, and then moved on to the Internet in 1997. He built the backing infrastructures at Salon, InfoWorld, Friendster, and Flickr. He is now SVP of Tech Operations at Etsy, and is the author of The Art of Capacity Planning and Web Operations published by O’Reilly. He blogs at kitchensoap.com

In the video, John answers the following questions:

• What is devops? Why now?
• What did Flickr do differently that worked? How did you apply that at Etsy?
• How did you take Etsy from painful releases to continuous deployment?
• How about larger organizations? Does continuous delivery impose more risk?
• What’s the role of operations in an organization that wants to practice devops?
• Is there still room for specialization in the devops model?
• What advice would you give developers who want to do continuous deployment?
• How do you reduce the risk of releases?
• How do you create resilient systems on the web?
• How do you deal with databases in the world of continuous delivery?

Michael Feathers makes the following observation:

I think that, in the end, we just have to accept that developer skill is a far more significant variable than language choice or methodological nuances¹. Frankly, I think we all know that, but we seem to suffer from the delusion that they are the primary knobs to tweak. Maybe it’s an extension of the deeply held view that from an economic viewpoint, it would be ideal if people were interchangeable.

The problem is, how do we get skilled developers? Since the concept of individual productivity in IT has never been satisfactorily defined, this is a particularly hard problem to solve. Lines of code – still a popular measure – suffers from the devastating flaw that a line of code is a liability, not an asset as is often thought. Measuring number of hours worked encourages heroic behavior – but experience shows that the “heroes” are usually the same people that cause projects to become late through taking unacceptable risks early on, and working long hours makes people stupid and leads to poor quality software. There is still no generally accepted set of professional standards or chartering system for IT professionals, and recruiting good people is very much an art rather than a science.

Psychologists have at least addressed the problem of why it is so difficult to acquire and measure skill in IT. As Daniel Kahneman says in Thinking Fast and Slow, there are “two basic conditions for acquiring a skill: an environment that is sufficiently regular to be predictable; [and] an opportunity to learn these regularities through prolonged practice.”

But traditional software projects are the opposite of a regular, predictable environment. The only good measure of success of a project – did the end result create the expected value over its lifetime? – is so distant from the critical decisions that caused that success or failure that it’s rare for anybody from the original team even to be present to get the feedback. It’s practically impossible to determine which of those decisions led to success or failure (in artificial intelligence, this is known as the credit-assignment problem).

These factors make it very hard for IT professionals to acquire the skills that lead to successful products and services. Instead, developers acquire the skills that allow them to most efficiently reach the goals they are incentivized by – usually declaring their work “dev complete” as rapidly as possible irrespective of whether the functionality is integrated and production-ready – and similar problems arise in other functional areas too.

The fact that software projects are complex systems rather than regular environments leads to another problem – the extreme difficulty of gathering data on which techniques, practices, and methodologies are actually effective, and the near impossibility of generalizing this data outside the context in which it was gathered.

In his excellent book The Leprechauns of Software Engineering Laurent Bossavit executes a devastating attack on software development folklore such as the “cost of change” (or “cost of defects”) “curve”, the claim that the variance in developer productivity is an order of magnitude, the idea of the cone of certainty, and many other cornerstones of methodological lore in software development. He shows that these theories – and many others – depend on very small sets of data that are gathered either from informal experiments run on computer science students, or projects which cannot possibly have been effectively controlled. The organization of the studies that form the basis of these claims is often methodologically unsound, the data poorly analyzed, and – most egregiously – the findings generalized well beyond their domain of applicability².

As a result, it’s not possible to take seriously any of the general claims as to whether agile development practices are better than waterfall ones, or vice-versa. The intuitions of “thought leaders” are also a poor guide. As Kahneman says, “The confidence that people have in their intuitions is not a reliable guide to their validity… when evaluating expert intuition you should always consider whether there was an adequate opportunity to learn the cues, even in a regular environment.” As Ben Butler-Cole points out in his companion post, “why software development methodologies rock”, the very act of introducing a new methodology can generate some of the results the adopters of the methodology intend to bring about.

You might think that puts us in an impossible position when it comes to deciding how to run teams. But consider why software development is not a regular environment, and why it is so hard to run experiments, to acquire skills, and to measure which practices and decisions lead to success, and which to failure. The root cause in all these cases – the reason the environment is not regular – is that the feedback loop between making a change and understanding the result of that change is too long. The word “change” here should be understood very generally to mean change in requirements, change in methodology, change in development practices, change in business plan, or code or configuration change.

There are many benefits to reducing cycle time – it’s one of the most important principles that emerges when we apply Lean Thinking to software development. Short cycle times are certainly essential for creating great products: as Bret Victor says in his mind-blowing video Inventing on Principle, “so much of creation is discovery, and you can’t discover anything if you can’t see what you’re doing.”

But for me this is the clincher: It’s virtually impossible for us to practice continuous improvement, to learn how to get better as teams or as individuals, and to acquire the skills that enable the successful creation of great products and services – unless we focus on getting that feedback loop as short as possible so we can actually detect correlations, and discern cause and effect.

In fact, the benefits of having a short cycle time from idea to feedback are so important that they should form one of the most important criteria for your business model. If you have to decide between creating your product as a user-installed package or software-as-a-service, this consideration should push you strongly in the direction of software-as-a-service (I speak from experience here). If you’re building a system which involves hardware, work out how you can get prototypes out as quickly as possible, and how you can modularize both the hardware and the software so you can update them fast and independently. 3D printing is likely to make a huge impact in this area since it allows for the application of software development practices to the evolution of hardware systems. Working in cross-functional teams is more or less a requirement if you want to achieve a sufficiently short cycle time.

Software methodologies – even the “hire a bunch of awesome people and let them self-organize” methodology – suck because they so often lead to cargo-cult behaviour: we’re doing stand-ups, we have a prioritized backlog, we’re even practicing continuous integration for goodness’ sake – why is the stuff we make still shitty and late? Because you forgot the most important thing: building an organization which learns and adapts as fast as possible.

1 Although as Laurent Bossavit points out (private communication) “A developer’s skill is in part the method he/she knows and his/her reasons for preferring one language over another.”

2 I am not suggesting that we give up on running experiments to learn more about what works and what doesn’t in software development, and the contexts in which such claims are valid – quite the contrary, I’m saying we’re not trying nearly hard enough.

From the press release, we can discern that Voke draws the following conclusions from their research:

• “The average cost of software projects is rising dramatically, in spite of smaller teams working much shorter durations” – it is implied that this is due to agile adoption.
• “Survey participants shared their experiences with Agile, expressed confusion, and identified more real world challenges than benefits”
• “Given the impact of catastrophic software failures on the brand, we should be witnessing a movement toward increased quality.” – the implication being that Agile produces lower quality software.
• Executives are buying agile because they think it means business agility. Voke, as we will see, think that this is nothing less than a bait-and-switch exercise by the Agile community.

In the actual report, Voke make three further high-level claims. First, they say Agile is effectively a Trojan horse for developers to gain control over the software delivery process, so they can avoid doing boring things like documentation, design, and planning. Second, they say it is primarily a vehicle for consultants like me to sell services, training, and books (curiously, they omit that analysts do quite well out of it too). Finally, they claim that the useful core of Agile is nothing more than rapid prototyping – which is nothing new. So it’s fair to say Voke aren’t keen on Agile (and while I am paraphrasing here, I am attempting to faithfully preserve the tone of the original).

However, while they say – correctly in my opinion – that companies are confused by Agile, Voke certainly aren’t in the business of helping. Voke make basic mistakes (or misrepresentations) in their opening discussion of Agile, Lean and Devops. The most egregious of these is around the role of quality in these movements, which they claim exclude QA and thus represent a move away from increased quality. In support, they cite the fact that the Agile Manifesto and the name “Devops” nowhere mention quality¹. This ignores the fact that building quality in is central to every good discussion of lean and agile development. Indeed not only do agile methodologies emphasize the importance of cross-functional teams (including business, QA and operations): Agile practitioners pioneered engineering practices such as test-driven development, continuous integration and refactoring which (as the report acknowledges) substantially reduce the cost of discovering and fixing defects.

Let’s address another claim: that the cost of software development has increased substantially since 2008 – the date of an earlier “Agile Realities” report. What their numbers in fact show is that the average² project cost has stayed the same whereas the average project duration has decreased. This subtlety aside, Voke are making the same mistake as many enterprises that treat IT as a cost center: focusing on development cost instead of return on investment. As noted by Douglas Hubbard, author of How to Measure Anything, “even in projects with very uncertain development costs, we haven’t found that those costs have a significant information value for the investment decision… The single most important unknown is whether the project will be canceled… The next most important variable is utilization of the system, including how quickly the system rolls out and whether some people will use it at all.” Thus getting a quality product that incorporates customer feedback for the same price as getting a mediocre one, but in a third less time (the numbers reported by Voke) represents an incredible deal. According to Voke’s own numbers, technology companies using Agile methodologies – uniquely – had zero projects abandoned after development.

The real news – which Voke hide away towards the end of the report on p32 – is that technology companies and enterprises both report higher levels of customer satisfaction and lower levels of unfixed critical defects (so much for Agile’s quality problem) when their projects use Agile methodologies. However enterprises do worse than technology companies: a large majority of technology companies report customer satisfaction when using agile methodologies, compared with a smaller majority of enterprises³.

Unfortunately the authors do not address the wider macroeconomic issues: in the period from 1958 to the present, the lifetime of an S&P 500 company has shrunk from 61 years to only 18 years. This period is contemporaneous with the rise of software development and the fall of traditional manufacturing. Today enterprises are being challenged by technology-powered start-ups which are able to leverage Lean and Agile methodologies to create a competitive advantage (most VCs wouldn’t consider funding a non-Agile start up). Thus the failure of many enterprises to implement Agile at scale, noted by the authors, is indeed a serious problem. However this problem cannot be addressed by recycling FUD from ten years ago, such as claims that Agile methodologies don’t believe in any documentation or design.

The data in the report thus makes for interesting reading – although I would have liked to see more statistical analysis, correlation, and some graphs. Unfortunately the analysis is mostly worthless, except as an exercise in twisting the data to suit the authors’ agenda.

² Voke don’t say what kind of average they have used, nor do they provide standard deviations or correlate the data in any way. There aren’t even any graphs.

³ I’m not going to give the actual numbers here because I think it crosses the line of fair use – you should buy the report if you want the full scoop.

Here are some of the key points from the interview:

• Etsy deploy to production 25-50 times a day;
• They built a new, segmented PCI-DSS compliant environment for their payment systems – “we built a whole separate Etsy, essentially”;
• Segregation of duties doesn’t mean people can’t talk to each other. In fact, collaboration is an essential element of effective risk management. So everybody in the PCI environment still follows devops principles – there’s no physical separation of the teams, and Etsy have hired more people into the various roles (dev, sysadmin, dba, manager, networking) to facilitate collaboration;
• In the payments environment they “still have to follow the rules: a developer still doesn’t have access to a production database”, but they’ll have dbas working alongside them who they can ask for help, and graphs showing metrics from the database;
• They use a similar deployment process in their PCI environment to their non-PCI environment, but it includes much more logging on who did what when, and they have roles for QA pusher and production pusher;
• Developers can run most of the test framework in their PCI development environment, and they have access to production logs via Splunk in read only mode;
• They have a ticketing system that all changes have to go through, but they can push out a change from dev to production in less than an hour if necessary with their normal, non-emergency change management process;
• Final words of advice: “Make sure that the team is on board and realizes that yes, there’s going to be some constraints, but … if you all work together you’re going to be able to continuously deliver the product that you want even into a secure environment.”

• Low-risk releases are incremental.
• Decouple deployment and release.
• Focus on reducing batch size.
• Optimize for resilience.

Read the rest of this article on InformIT (free, no registration required)