Agile Tokyo 2010 by Yoko Yoshikawa

The most surprising discovery for me was that agile is considered new and somewhat subversive in Japan. Japanese IT companies have been doing waterfall on large projects for a long time now, and it is deeply entrenched. Indeed, everybody I spoke to confirmed that waterfall worked perfectly well for them as a software development methodology. This was staggering to me, since outside Japan large software projects run with waterfall routinely go over budget and end up with either cuts in scope or poor quality – several recent reports, such as this one, put IT project failure rates in the range of 60-70%. This appears not to be the case in Japan – it is perhaps the only country that could make waterfall work reliably and repeatably. As my host, Yoshi Nagase, commented, this is deeply ironic in a country that created Toyota and lean manufacturing.

However there is certainly an interest in agile now. Several large systems integrators have started running small agile pilot projects, and apparently the results have been successful. Why the sudden interest in agile if waterfall can deliver high quality software on time and on budget? The answer is easily extrapolated from this example of a project plan I saw at one of the companies I visited – the plan has been changed somewhat to protect the guilty, but not in any essential details – it is a pretty standard plan following the V-model.

For me, the most egregious problem with this plan is that the planning stage for phase 2 of the project comes before the testing stage for phase 1. So anything discovered during the testing and user acceptance stages cannot be fed in to phase 2 – it has to wait for phase 3. Effectively, that means that the feedback loop from discovering a problem in your requirements for phase 1 will have to wait until the end of phase 3 to be delivered to users – a cycle time of 1 year. This makes it extremely hard to respond to customer feedback. Even in the best case scenario, whereby you discover a requirement during the planning stage of a phase, you must wait 6 months for it to be delivered to users.

Japanese companies seem to be realizing that they simply cannot maintain a competitive advantage in the current economic climate with this kind of cycle time. They must get valuable software into the hands of users, and incorporate feedback from them, much faster. This of course is the value proposition of agile methods – the first principle of the agile manifesto, which also inspired the title of our book, is this: “our highest priority is to satisfy the customer through early and continuous delivery of valuable software”.

The risk is that Japanese companies simply have very little experience doing agile delivery at all, let alone on large projects. Adopting agile in an effective way involves serious organizational change. Cross-functional teams including developers, testers, analysts, testers, operations staff, and project managers must be created. Developers must be trained in practices like test-driven development, continuous integration, refactoring, and incremental development. Testers will have to start collaborating more closely with analysts and developers. Managers will need to be trained in agile delivery methodologies – and they will need to make mistakes and gain experience. Teams will need new tools.

However if there is any country that can make such a large change in an organized and efficient manner, it is Japan. I’m really excited to see what agile methodologies do for Japanese IT, and also to see what the Japanese software delivery community does to agile – no doubt we will see many new ideas and evolutions. It’s an exciting time to be in IT.

Automation is key to enabling collaboration. If deploying software to testing environments, managing infrastructure, testing, building and releasing your software are manual activities, they are of course terribly error-prone. Worse, delivery teams then typically spend a large proportion of their time fire-fighting to get branches merged and new builds created so that they can be deployed into a production-like environment and tested. This inevitably means that the feedback loop from testing is incredibly slow, and doing things like load testing on staging environments gets left till late in the process, rather than being done from the beginning when problems can be fixed cheaply. Automating as much as possible of the delivery process ensures a much tighter feedback loop, and frees people to focus on high-value activities like evolving an appropriate architecture, exploratory testing, and making deployments and releases low-risk, push-button processes.

There is also an organizing principle for all of these activities: the deployment pipeline³. The description and elaboration of this pattern forms the core of the book. The idea behind the deployment pipeline is to model the part of your project’s value stream that goes from check-in to release, and then to automate it. Every check-in triggers a new build, which then passes through the deployment pipeline as shown below (inspired by Chris Read).

The deployment pipeline allows everybody involved in delivering software to get fast feedback on the status of every change that is introduced to an application. It makes it simple to trace which builds have been through which environments, what the results were, and what’s currently in each environment. Finally, it allows testers and operations people to self-service the build of their choice into the environment they control. The upshot is faster feedback, better collaboration within delivery teams, and – because each process from check-in to release is automated – fewer errors, more reliable releases, and shorter cycle times.

The book is still in Rough Cuts, which means we’re actively seeking out feedback to help us improve it. Please let us know what you think: the book has a feedback section, and we’ve created a group to discuss the issues we cover in the book. We’ll be posting outtakes, code examples, and further material from the book here, as well as providing regular updates on the state of the art in this space. We look forward to seeing you here!

¹ We got the idea for the book’s name from the first of the Twelve Principles behind the Agile Manifesto: “Our highest priority is to satisfy the customer through early and continuous delivery of valuable software”.

² There’s a whole movement devoted to increasing collaboration between developers and operations types.

³ Dave, along with Mark Rickmeier, came up with the idea of deployment pipelines back in 2004. The idea then circulated within ThoughtWorks for some time, and was first documented publicly in 2005 (in a blog entry by our colleague Sam Newman). Dave and I have both written papers on the subject. More recently, there has been a new variation on the theme in the shape of continuous deployment.