Continuous Delivery

Book Review: The Phoenix Project

I am not going to do a ton of book reviews on this blog (I have one more planned for next month). I’ll only bother posting reviews of books that I believe are both excellent and relevant to Continuous Delivery. This book easily satisfies both criteria. Full disclosure: Gene gave me a draft of this book for free for reviewing purposes.

You’ve probably heard of Gene Kim, Kevin Behr and George Spafford before. They are the three amigos responsible for The Visible Ops Handbook, which can be found in the book pile of every good IT operator. Their new book, The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win, follows the format of Eliyahu Goldratt’s classic, The Goal.

Told from the perspective of newly-minted VP of IT Operations Bill Palmer, it describes the turnaround of failing auto parts company Parts Unlimited. This is to be achieved through the delivery of the eponymous Phoenix Project, a classic “too big to fail” software project designed to build a system which will revive the fortunes of the company.

To quote (p51):

The plot is simple: First, you take an urgent date-driven project, where the shipment date cannot be delayed because of external commitments made to Wall Street or customers. Then you add a bunch of developers who use up all the time in the schedule, leaving no time for testing or operations deployment. And because no one is willing to slip the deployment date, everyone after Development has to take outrageous and unacceptable shortcuts to hit the date.

The results are never pretty. Usually, the software product is so unstable and unusable that even the people who were screaming for it end up saying that it’s not worth shipping. And it’s always IT Operations who still has to stay up all night, rebooting servers hourly to compensate for crappy code, doing whatever heroics are required to hide from the rest of the world just how bad things really are.

Part One of the book describes in loving detail the enormous clusterfuck pie that is baked from these ingredients. The pie is spiced with an internal Sarbanes-Oxley audit which reveals 952 control deficiencies, an outage of the payroll processing system, and various other problems that conspire to deepen the woe of the operations group, all of which are clearly drawn from the deep well of the authors’ real-life experiences.

Apart from the main characters - our hero Bill, his boss Steve, and the evil villain Sarah - The Phoenix Project features a delightful rogues’ gallery which anyone working in an enterprise will recognize:

• Brent Geller, the boy wonder whose encyclopedic knowledge of the company’s Byzantine IT systems means that his involvement is necessary to get anything done.
• Patty McKee, the Director of Support who runs a change management process so bureaucratic that everybody bypasses it.
• John Pesche, the black binder wielding Chief Information Security Officer whose constant meddling under the guise of improving security has turned him into a pariah.

The second part of the book details how the IT group is reborn from the ashes of the Phoenix Project into a high-performing organization that is a strategic partner to the business. This is achieved through the application of a heavy dose of lean thinking (including continuous delivery) administered by Erik, a mercurial IT and manufacturing guru Steve is courting to join the board. The book does an excellent job of showing - as well as telling - how to apply the concepts (and the effect of doing so) in an enterprise with plenty of technical debt. Perhaps the most eyebrow-raising part of this section is the way in which John has his soul mercilessly crushed to the point where he goes on a multi-day drinking spree before he is rehabilitated towards the end of the book (he is a phoenix too).

John’s narrative arc is just one example of how the book also succeeds as a novel. It’s gripping, with moments of drama and high emotion, as well as some great one-liners. There was even one point when I teared up (bear in mind that I also cried during Forrest Gump - unlike the book’s central characters, I did not serve in the armed forces).

Nobody who has read The Goal will miss The Phoenix Project’s similarity in terms of style and plot. Perhaps my favourite thing about the book’s pedagogical style is the way Erik (like Jonah in The Goal) uses the Socratic Method to give Bill the tools to solve his problems by himself. Of course this learning process is fictional, but it means you get to see Bill struggling with the questions and trying things out.

It remains to be seen whether readers of the book will be able to apply these techniques as successfully as Bill without a real Erik to guide them. But of course, this is a limitation of any book. If I had one criticism it’s that unlike real life, there aren’t many experiments in the book that end up making things worse, and it’s this process of failing fast, learning from your failures, and coming up with new experiments that is instrumental to a real learning culture.

One important point worth noting if you are working in an organization like Parts Unlimited is this: the IT department’s rebirth is only possible because of the Titanic proportions of the disaster that unfolds in Part One. For management to truly embrace change, a compelling event or a teachable moment (i.e. a Charlie Foxtrot) is required. Unless your organization faces the same existential threat that Parts Unlimited does, you’ll have a much harder time convincing people they should adopt the tools described in the book.

Overall, The Phoenix Project is a fantastic read. It’s entertaining, cathartic, inspirational and informative. If, like me, you have an enormous backlog of books (and more work in process than you’d like) I suggest giving yourself a break and putting this one to the top of your list. It’ll only take you a day or two, and despite its conceptual density it will leave you feeling refreshed and energized with a bunch of new ideas to try out. The Phoenix Project deserves to be read by everyone who works in - or with - IT.