Read the rest of this article on InformIT (free, no registration required)

Branch by abstraction: a pattern for making large-scale changes to your application incrementally on mainline.

The example Paul Hammant provides in his original blog entry on this technique is moving from Hibernate to iBatis. As it so happens, Go, the continuous integration and agile release management platform I work on, is presently moving from iBatis to Hibernate, and has been doing so for over a year now. We are also slowly moving our UI over from Velocity and JsTemplate to JRuby on Rails.

Both of these changes are being done slowly and incrementally, at the same time as developing new features, while checking in to mainline on our Mercurial repository multiple times a day. How do we do it?

Moving from iBatis to Hibernate

The team decided to move from iBatis to Hibernate for two reasons: first, we were able to use its ORM efficiently since we had control of our database schema, which saved us writing lots of custom SQL, and second, because its second-level cache helped performance.

Of course we didn’t want to move the whole codebase over at once. So as we started adding new functionality that required new calls to the database, we added these new calls using Hibernate, moving over old calls that used iBatis as required.

It is relatively straightforward to update the persistence logic incrementally because the Go codebase uses a standard layered architecture, with controllers using services that in turn use repositories. Because all the database access code is encapsulated in repository classes using the repository pattern, it’s a simple case of incrementally changing one repository class at a time from using iBatis to Hibernate. The service layer has no idea of the underlying persistence framework.

Moving from Velocity and JsTemplate to JRuby on Rails

We also wanted to move over from a Java-based UI stack to a JRuby on Rails stack, both because it was much easier to write tests for this stack, and because it speeded up UI development. Again, this change was made incrementally. When we created a new page in the application, we would create it using the JRuby on Rails stack, linking to the new page from the rest of the application once it was ready.

We would also move pages over whenever we wanted to make substantial changes to them. Again, the new version of the page would be developed using the new stack, and then we’d switch URIs in the rest of the application to point to the new version of the page once it was ready. At this point, we would remove the old version of the page. So while most of the UI in Go is now implemented using JRuby on Rails, there are still a couple of pages that use the old Java stack.

However you’d never know by looking at the pages, because they have identical styling. You have to look at the URI. Any URI that starts /go/tab is routing through the old Velocity stack. All the other URIs are routed via Rack to JRuby on Rails, which in turn calls through to the same Java service layer that the old UI also uses.

How branch by abstraction works

Branch by abstraction involves making large-scale changes to your system incrementally as follows:

1. Create an abstraction over the part of the system that you need to change.
2. Refactor the rest of the system to use the abstraction layer.
3. Create new classes in your new implementation, and have your abstraction layer delegate to the old or the new classes as required.
4. Remove the old implementation.
5. Rinse and repeat the previous two steps, shipping your system in the meantime if desired.
6. Once the old implementation has been completely replaced, you can remove the abstraction layer if you like.

Branch by Abstraction

In the iBatis/Hibernate example, the abstraction layer is the repository layer, which hides the implementation details of which persistence framework is being used. In the JRuby on Rails example, the abstraction layer is the servlet engine, which can dispatch either to the JRuby on Rails framework (using Rack) or to standard Java servlets by matching on the URI.

While Go is a relatively small project – it has less than ten developers, and it’s only been going for a few years – the same principles apply on projects of all sizes, and teams in ThoughtWorks have used this pattern successfully even on large and distributed projects.

Admittedly, branch by abstraction can add more overhead to the development process, especially in the case that the codebase is poorly structured. You need to think hard and move a bit slower in order to make changes incrementally in this fashion. But in many cases the upside is worth the extra effort, and the larger the restructuring, the more important it is to consider using branch by abstraction.

The key benefit of branch by abstraction is that your code is working at all times throughout the re-structuring, enabling continuous delivery. That means your release schedule is completely decoupled from your architectural changes, and thus you can stop working on the restructuring at any point to do something else that is higher priority, such as putting out a release with an exciting new feature you just thought up.

Branch by abstraction compared with branching in version control

Branch by abstraction is a somewhat misleading name, because of course it represents an alternative to using branching in version control when making large-scale changes to your system. Teams often use version control branches to make large-scale changes so that they can continue to develop functionality and fix bugs on mainline. The problem of course is that the merge back to mainline is guaranteed to be painful, and the amount of pain is a function both of how big the change you want to make is, and the amount of work you do on mainline in the meantime¹.

That means that the stronger the forces are that push you towards using a version control branch, the more painful it is going to be at the end when you have to merge. If you’re also using branches for features, the situation is made even worse. In general, using branches for features or large-scale changes is a bad idea for several reasons, of which the most important are that it prevents both continuous delivery and refactoring². Martin Fowler has excellent articles on why feature branching is bad, and how to use feature toggles as an alternative.

That doesn’t mean that all branching in version control is bad. It’s OK to branch in order to spike out an idea that you’re going to throw away. It’s also OK to branch upon releasing, provided you only use the release branch for small, critical bug-fixes. However teams who are practicing continuous deployment usually don’t bother with this, since it’s typically easier to fix any problems on mainline and roll forward than it is to roll back, because the delta between releases is so small.

The only other time it might be permissible to use branching is if your codebase uses the big ball of mud pattern. In this scenario even creating an abstraction layer can be hard. In order to do this, you must first find a “seam” (typically in the form of a set of interfaces if you’re using a statically typed OO language) that you can put the abstraction layer over. If a seam was not readily available, you would normally create one through a series of refactorings, but if that’s not possible for some reason then you might have to resort to creating a branch to get into a position to do this. Of course, this is an extreme move.

Relationship to other patterns

Relationship to refactoring. Refactoring has been defined as “a change made to the internal structure of software to make it easier to understand and cheaper to modify without changing its observable behavior”. In this sense, both of the examples given above of branch by abstraction are also examples of refactorings. Crucially though branch by abstraction is effectively a programme of related refactorings, which taken together result in a large-scale change to the architecture of the application. Along with the ability to release your software at any time, the ability to refactor is perhaps the most important benefit of developing on mainline.

Relationship to feature toggles. People often confuse branch by abstraction with feature toggles. Both are patterns that allow you to make changes to your system incrementally on mainline. The difference is that feature toggles are intended to allow the development of new features, while keeping those features invisible to users when the system is running. Feature toggles are thus used at deploy time or run time to choose whether a particular feature or set of features is visible in the application.

Branch by abstraction is a pattern for making large-scale changes to an application incrementally, and is thus a development technique. Branch by abstraction can of course be combined with feature toggles, such that – for example – you could switch between the iBatis and Hibernate implementations of a particular set of data access calls to compare the performance of them at runtime. But typically, the choice of implementation is chosen by the developers and either hard-coded or baked in at build time, perhaps through your dependency injection configuration.

Relationship to strangler application. The strangler application pattern involves incrementally replacing a whole system (usually legacy) with a completely new one. Thus it operates at a higher level of abstraction than branch by abstraction, which is for incrementally changing the implementation of a component of your system. The lines between the two start to blur if you have a service-oriented architecture.

Isn’t this just good object-oriented design? Yes. Code which follows the SOLID principles makes it very easy to apply this pattern, in virtue in particular of following the dependency inversion principle and the interface segregation principle (ISP). The ISP is important because it provides a nice level of granularity at which to switch out implementations. As my colleague David Rice points out, branch by abstraction is the only sensible way to change the implementation of some particular component. Martin Fowler makes the same point by sometimes defining a component as some part of a system that can be swapped out for another implementation.

² Of course there are exceptions to every rule. Branches (other than for releases and spikes) are OK if you are working in a small, experienced team, and the branches are very short lived (less than one day).

In his 2003 article in Harvard Business Reviews, Nicholas Carr argued that IT didn’t matter – that it was simply infrastructure that didn’t offer a competitive advantage. History has proven him to be almost right inasmuch as most IT services, such as email systems and payroll, are like utilities. You need them, but you don’t want to create or manage them in-house. However in many organizations there exist software projects that are considered essential to the organization’s strategy, such as Southwest’s website. Indeed, a service that is considered utility software for one company, such as a bank’s online service, might be considered strategic for another, for example an online-only bank. Martin Fowler goes through the differences between strategic and utility software and provides some good references in his bliki entry Strategic vs Utility Dichotomy.

It’s all too common for organizations to waste a great deal of time and money doing custom development on services that should be utility for them. Usually, they’d be much better off using packaged software. Apart from political considerations, the major reason many organizations don’t do this is because packaged software doesn’t support their business process as-is and thus requires customizing. Often, vendors are all too happy to customize their products, since it means lucrative consulting revenue. But this is almost always the wrong thing to do.

Instead, organizations should change their business process to match what the product supports out of the box. Indeed, this is a good test to discover if a business process is strategic or not: if using commercial, off-the-shelf software to implement the system – and changing the process to match the software’s model – impacts a competitive advantage of their business, then guess what: the system isn’t utility, it’s strategic. There’s a great recent case study from Telstra, Australia’s largest telco, which discusses how they moved from a heavily customised service desk solution to one that ran out of the box.

There are two types of software that are often cited as being unsuitable for continuous deployment: products, and embedded systems. It’s hard to release commercial products to users very frequently because it’s too expensive to support many versions of the system. Embedded systems can’t be released frequently for similar reasons, and also because of course it’s very expensive to rev hardware. These arguments are the reason why I am very careful to distinguish continuous delivery from continuous deployment. Continuous delivery is about keeping your systems production-ready throughout development, so that they can be released to users at any time – in particular, when the business decides it’s the right time to release. It thus follows that you can practice continuous delivery even if you’re not doing continuous deployment.

Indeed there are many benefits to following the principles and practices of continuous delivery, even when working on products or embedded systems. On Go, the CI and release management platform I manage, we only release every 3-4 months, because we can’t support more than a couple of releases. However we use our tool to build itself, and upgrade it every time there is a good build. We deploy once a week to our company’s internal build grid, which is used to build our other products. We do early access builds that are available to users – although unsupported – every two weeks. This ensures our software is always production-ready, that we can get rapid feedback from our users – both internal and external – and that we can release whenever we think the feature set is sufficiently compelling. This kind of practice certainly wasn’t our idea – the Eclipse project has been producing daily and weekly public builds since at least 2002.

Embedded products can also follow continuous delivery by regularly integrating and testing their products throughout their development lifecycle, and by having everybody involved – hardware, firmware, and software – working together from the beginning. Indeed, this is exactly what great product companies already do. One of the biggest advantages Apple has – other than the primacy of design and their focus on perfection – is their vertical integration. When the first Macintosh was created, there weren’t separate, physically separated hardware, software, and OS teams: instead, the entire Macintosh team sat together in one building, and wasn’t allowed to grow past 100 people (see this fascinating interview with John Sculley for more on the early history of Apple). That meant everybody could work together to make sure that every decision – what chips to use, how to implement the operating system, how to build the applications – were made in a way that optimized the product as a whole.

Granted, the Apple team weren’t practicing the level of automation in terms of integration and testing one would expect from a more sophisticated continuous delivery set-up – which perhaps explains why they were still debugging a few hours before the operating system had to ship. But a great deal of the quality of the product was clearly due to the cultural practices of having everyone involved in the system working together from the beginning – something we emphasize in the book, and which finds an echo in the DevOps movement.

However it’s possible to follow ITIL principles and practices in a lightweight way that achieves the goals of effective service management while at the same time enabling rapid, reliable delivery. In this occasional series I’ll be examining how to create such lightweight ITIL implementations. I welcome your feedback and real-life experiences.

I’m starting this series by looking at change management for two reasons. Firstly it’s often the bottleneck for experienced teams wanting to pursue continuous delivery, because it represents the interface between development teams and the world of operations. Second, it’s the first process in the Service Transition part of the ITIL lifecycle, and it’s nice to pursue these processes in some kind of order.

Change Management in ITIL

In the ITIL world, there are three kinds of changes: standard, normal, and emergency changes. Let’s put emergency changes to one side for the moment, and distinguish between the two other types of changes. Normal changes are those that go through the regular change management process, which starts with the creation of an RFC which is then reviewed, assessed, and then either authorized or rejected, and then (if authorized) planned and implemented.

Standard changes are low-risk changes that are pre-authorized. One of the examples in the ITIL literature is a “low-impact, routine application change to handle seasonal variation” (Service Transition, p48). Each organization will decide the kind of standard changes that they allow, the criteria for a change to be considered “standard”, who is allowed to approve them, and the process for managing them – like normal changes, they must still be recorded and approved. However the approval can be made by somebody close to the action – it doesn’t need to work its way up the management chain.

Even normal changes can vary in terms of the complexity of the change management process they go through. Major service changes which impact multiple organizational divisions will require a detailed change proposal in addition to the RFC, and will need to be approved by the IT management board. Very high cost or high risk changes will need approval by the business executive board. However low risk changes can be approved by a CAB, or change advisory board.

Often CAB approval can be a lengthy and painful process. However it certainly doesn’t need to be. The ITIL v3 Service Transition book specifies both that CAB approval can be performed electronically, and that not everybody in the CAB needs to approve every change (pp58-59). Getting agreement on exactly who needs to authorize a given type of change, and making sure those people know in advance the information they need to decide whether to approve it, can mean a fully compliant authorization process that can be performed in seconds in a just-in-time fashion.

Close collaboration between the CAB and the delivery team is also important in order to make the approval process efficient. For example, my colleague Tom Sulston suggests that if you want to do a release at the end of every iteration, you should invite the relevant members of the CAB to your iteration planning meeting (via conference call if necessary). Thus they can see exactly what is coming down the pipeline, and ask questions and provide feedback to ensure that they are comfortable giving approval when the time comes for release.

We can see that ITIL provides several mechanisms which enable lightweight change management processes:

• Standard changes that are pre-approved.
• CAB approvals that are performed electronically, rather than at face-to-face meetings.
• Arranging in advance which subset of CAB members are required to approve each type of change.

How do we ensure that as many as possible of our changes can use these mechanisms?

Using Continuous Delivery to Reduce the Risk of Change

ITIL follows the common-sense doctrine that each change must be evaluated primarily in terms of both its risk and value to the business. So the best way to enable a low-ceremony change management process is to reduce the risk of each change and increase its value. Continuous delivery does exactly this by ensuring that releases are performed regularly from early on in the delivery process, and ensuring that delivery teams are working on the most valuable thing they could be at any given time, based on feedback from users.

Figure 1: Traditional delivery

In a traditional delivery lifecycle, even with agile projects, the delivery cadence looks rather like figure 1. The first release can often take some time: for really large projects, years, and even for small projects typically at least six months. Further releases typically occur with a relatively regular rhythm – perhaps every few months for a major release, with a minor release every month.

In the world of continuous delivery, and in particular continuous deployment, things look somewhat different. New projects will provision their production environment as part of project inception, and begin releasing to it as early as possible in the delivery process – potentially long before the service is actually made available to users. Further changes are made as frequently as possible, so that in effect there are no subsequent major releases, or indeed minor releases – every further release is a micro release. Project managers will focus on optimizing for cycle time, ensuring that changes to the system get put live as rapidly as possible. This process is shown in figure 2.

Figure 2: Continuous delivery

Continuous delivery has several important effects both on the delivery process and on the organization as a whole. First of all, it means operations must work closely with the rest of the delivery team from early on in the project. The combined team must provision the production environment and testing environments, and create automated processes both for managing their configuration and for deploying the nascent application to them as part of the deployment pipeline.

Because operations personnel get to see the application from early on in its development, they can feed their requirements (such as monitoring capabilities) into the development process. Because developers get to see the system in production, they can run realistic load tests from early on and get rapid feedback on whether the architecture of the application will support its cross-functional requirements. This early and iterative release process also exerts a strong pressure to ensure that the system is production ready throughout its lifecycle.

Continuous delivery enables a low-ceremony change management process by ensuring that the first (and riskiest) release is done long before users are given access to the system. Further releases are performed frequently – perhaps as often as several times a day, and certainly no less often than once per iteration. This reduces the delta of change, reducing the risk of each individual release, and making it easier to remediate failed changes. Because continuous delivery mandates that each release must be comprehensively tested in a production-like environment, the risk of each release is further reduced.

I said earlier that continuous delivery ensures that value is more efficiently delivered to the business. But what is the value of releasing so early in the delivery process, long before the anticipated business value of the system can be realized? The value proposition of continuous delivery – keeping systems in a working state throughout their development and operational lifecycle – is threefold. First it allows the customer to see the system, and perhaps show it to selected users, from its earliest stages. This allows the customer much more fine-grained control over what is delivered, and perhaps even to discover more radical changes that need to be made to the service in order to make it even more valuable. Second, incremental releases from earlier on are much less risky than a big-bang release following an integration phase. Finally, it allows project and program managers real data on the progress of the project.

Change Management and the Deployment Pipeline

The deployment pipeline supports an efficient change management process in several ways.

First of all, it provides the mechanism through which changes are evaluated for risk and then applied. Every proposed change to your systems, whether to an application, your infrastructure, your database schema, or your build, test and deployment process itself, should be made via source control. The deployment pipeline then performs automated tests to determine that the change won’t break the system, and makes changes that pass the automated tests available to be deployed to manual testing and production environments in a push-button fashion, with appropriate authorization controls.

A good pipeline implementation should be able to provide reports that let you know exactly what has changed since last time you deployed. People approving deployments can use this information, reports from the automated and manual testing that is performed through the deployment pipeline, and other metrics such as the length of time since the last change, to assess the risk of the change.

Because the pipeline is used to control the delivery process, it is also the ultimate auditing and compliance tool. Your release management tool, which controls the deployment pipeline, should be able to tell you information like:

• Which version of each of your apps is currently deployed into each of your environments.
• Which parts of the pipeline every version of your app has been through, and what the results were.
• Who approved every build, test and deployment process, which machines they ran on when, the logs, and exactly what the inputs and outputs were.
• Traceability from deployment back to the version in version control that each artifact deployed was ultimately derived from.
• Key process metrics like lead time and cycle time.

If your tool doesn’t give you this information, get one that does, such as the one that I manage: Go from ThoughtWorks Studios.

This kind of information is essential as part of the metrics gathering that needs to be performed in order to make go/no-go decisions, do root cause analysis when changes fail, and determine the effectiveness of your change control process as part of continuous improvement.

Leveraging Standard Changes

It’s my contention that standard changes should be much more widely used than they are. In particular, all deployments to non-production environments should be standard changes. This isn’t common right now for the simple reason that it’s often hard to remediate bad deployments. Thus operations people naturally try to erect a barrier which requires a certain level of effort and evidence to surmount before deployments are allowed to happen to environments they control.

Addressing this problem requires a two-pronged approach. First, there must be a sufficient level of automated testing at all levels before a build ever gets towards operations-controlled environments. That means implementing a deployment pipeline which includes unit tests, component tests, and end-to-end acceptance tests (functional and cross-functional) that are run in a production-like environment. Worried that pressing the “deploy” button might break stuff? That means your automated testing and deployment processes are not up to snuff.

Second, it must be easy to remediate a deployment when something goes wrong. In the case of non-production deployments which don’t need to be zero-downtime, that means either rolling back or re-provisioning the environment. The more projects I see, the more I am convinced that in most cases the first thing teams need to address to improve their delivery process is the ability to be able to provision a new environment in a fully automated way using a combination of virtualization and data center management tools like Puppet, Chef, BladeLogic, or System Center. Assuming you package up your application, data center management tools can also perform rollbacks by just uninstalling your packages or reinstalling the previous version.

Once all this automation in place, it is possible to kick off a positive feedback cycle. Deployments happen more frequently because they are lower risk, and in turn, the act of practicing deployments more frequently drives out problems earlier in the delivery cycle when they are cheaper to fix, further reducing the risk of deployment.

Conclusion

Continuous delivery and the deployment pipeline allow you to make effective use of the lightweight mechanisms ITIL provides for change management. However this in turn relies on building up trust between the delivery team and the people who are accountable for approving changes on the CAB. Continuous delivery helps with this by ensuring the change process has been exercised and tested many times before the service is made available to users. However as part of a continuous improvement process the change management and delivery processes must be constantly monitored and measured to improve their effectiveness – a process which is in turn enabled by the tight feedback loop that frequent releases provide.

Finally it is important to note that there is of course a cost associated with continuous delivery. Potentially expensive environments must be requisitioned and provisioned much earlier on in the delivery process (although virtualization and cloud computing can significantly reduce these costs). Everybody involved in the service management lifecycle – from customers to developers to operations staff – needs to be involved in development and operation of the service throughout its lifecycle¹. This requires high levels of communication and feedback, which inevitably incurs an overhead. While continuous delivery is widely applicable to all kinds of software delivery projects from embedded systems through to software as a service, it is only really appropriate for strategic IT projects.

So here are my recommendations:

• Create a deployment pipeline for your systems as early as possible, including automating build, tests, deployment and infrastructure management, and continue to evolve it as part of continuous improvement of your delivery process.
• Release your service from as early on as possible in its lifecycle – long before it is made available to users. This means having operations personnel involved in projects from the beginning.
• Break down further functionality into a many small, incremental, low-risk changes that are made frequently so as to exercise the change management process. This helps to build up trust between everyone involved in delivery, including the CAB, and provides rapid feedback on the value of each change.
• Ensure that the people required to approve changes are identified early on and kept fully up-to-date on the project plan.
• Make sure everybody knows the risk and the value of each change at the time the work is scheduled, not when you want approval. Short lead times ensure that CAB members won’t have time to forget what is coming down the pipeline for their approval.
• Make sure authority for approving changes is delegated as far as possible down into the delivery organization, commensurate of course with the risk of each change.
• Use a just-in-time electronic system for approving changes.
• Use standard changes where appropriate.
• Keep monitoring and improving the change management process so as to improve its efficiency and hence its effectiveness.

What I’m not suggesting is trying to bypass the CAB or get rid of face to face CAB meetings. Face-to-face CAB meetings serve an important purpose, including supervising, monitoring and managing process I describe here. And every change should be approved by the agreed subset of the CAB, who can decide to push back on the change or require that it be considered by a larger group if necessary.

Thanks to Tom Sulston for his feedback and suggestions.

However this context is important – and not just because it helps you convince your boss to implement continuous delivery. One of the big technical memes of the last year has been continuous deployment – the practice of releasing every good version of your software, often multiple times a day. But that practice came out of a business imperative, which finds its general expression in the lean startup movement. In startups, it’s essential to get a minimum viable product available to users and then iterate rapidly based on real feedback. Sometimes you need to change direction in a more radical way (known as pivoting) if you discover what you built isn’t valuable.

When building strategic (as opposed to utility) software in a non-startup environment, many of the same imperatives apply. Furthermore, continuous delivery has other important benefits: for example, it reduces the risk of each individual release substantially, and provides a true measure of project progress.

I’ve found myself talking a lot about the value proposition of continuous delivery since I wrote the book, so I thought it would be useful to write it all down. You can get hold of my essay here on InformIT for free. InformIT have also made chapter 5 of my book – which explains the deployment pipeline in some detail – available free of charge too.

Kingsley’s problem was that his team had implemented a deployment pipeline such that it was only possible to self-service a new build into his exploratory testing environment once the acceptance tests had been run against it. This typically took an hour or two. So when he found a bug and it was fixed by a developer, he had to wait ages before he could deploy the build with the fix into his testing environment to check it.

This problem results from a combination of two anti-patterns that are common when creating a deployment pipeline: insufficient parallelization, and over-constraining your pipeline workflow.

Insufficient parallelization

The deployment pipeline can be broadly separated into two parts – the automated part and the manual part. The automated part consists of the automated tests that get run against every commit, which prove (assuming the tests are good enough) that the application is production-ready. In an ideal world with lots of processing power (and near-zero power consumption) this part of the pipeline would consist of a single stage that built the system and executed all these tests in parallel.

However if you have a reasonably sized app with good test coverage, running all the tests (including acceptance tests) on a single box will take over a day. Even with a build grid and parallelization, you’re unlikely to get it down to a few minutes, which is an acceptable length of time for the development team to get feedback on their changes. Thus the pipeline gets split into two stages – a commit stage and an acceptance stage.

The commit stage contains mainly unit tests and gives you a high level of confidence you haven’t introduced any regressions with your latest change. It runs in a few minutes (ten is the absolute maximum). The commit stage also runs any analysis tools for producing internal code quality reports (e.g. test coverage and cyclomatic complexity), and produces an environment-independent packaged version of your application that is used for all later stages of the pipeline (including release). The acceptance stage contains all the rest of the automated tests and usually takes longer – an hour or two.

Both of these stages should be parallelized so they execute as fast as possible. For example it is usually possible to have the commit stage run across several boxes: one to build packages, one to do analysis, and a few to run the commit tests. Similarly, you can run acceptance tests in parallel – on the Mingle project, at the time of writing, they have 3,282 end-to-end acceptance tests that run across 53 boxes. The whole stage takes just over an hour to run¹.

The heuristic is:

• Make your pipeline wide, not long: reduce the number of stages as much as possible, and parallelize each stage as much as you can.
• Create more stages if necessary to optimize feedback.

Inflexible workflow

Even if Kingsley’s team had followed the pattern above and aggressively parallelized their tests, it’s possible they might not have got the lead time from check-in to availability to deploy to manual testing down sufficiently. But they also made another mistake. They only allowed a build to be deployed to manual testing after the acceptance test stage had passed. Presumably this was to prevent manual testers from wasting their time on builds that weren’t known to be good.

Some of the time, this is reasonable. However, often it isn’t – and the tester will know whether or not they want to see the acceptance tests pass, so he or she should get to choose whether or not to wait.

When people design pipelines, they are usually deciding in their head what the ‘ideal’ process should look like, and often they think linearly. The result might look something like figure 1, a linear obstacle course that builds must overcome to prove their fitness for release.

Figure 1: Linear pipeline

However the problem with this design is that it prevents teams from optimizing their process. For example, this design makes it difficult to manage emergencies. If you need to push a fix out quickly, you might re-organize your team on the fly, parallelizing tasks that might normally be performed in series, such as testing capacity and performing exploratory testing. This is impossible with the pipeline design above.

So the same rule applies – the pipeline should fan out as soon as it makes sense to do so. In the case of exploratory testing environments, all builds that pass the commit tests should be available. In the case of staging and production environments, all builds that pass both commit and acceptance tests. Arguably – unless you’re using the blue-green deployments pattern in which staging becomes production – builds should pass through staging before they can be deployed to production. But if you have comprehensive end-to-end acceptance tests that are run in a production-like environment, you might allow builds to be deployed directly to production.

Figure 2: Optimized pipeline

Of course all deployments can be made subject to approvals. The important thing is not to conflate your workflow with your approval process by requiring builds to go through multiple different stages serially. Instead, make it easy for the people doing the approval to see which parts of the delivery process each build has been through, and what the results were, so they have the information they need to make decisions – such as which build to deploy, or whether a particular build should be deployed – at their fingertips.

Go showing which environments a build has been deployed to

There are cases where the arrangement shown in Figure 2 is not sufficiently linear – for example, when integrating several applications and then promoting them through staging and live in a deployment train. However in general the heuristic is:

• Make your pipeline wide, not long: allow for resources to be redistributed so manual work can be performed in parallel if required.
• Prefer visibility to locking down: give people the information they need to make informed decisions, rather than constraining them.

Conclusion

As part of your practice of continuous improvement, evolve your deployment pipeline to enable teams to become more efficient. In general, aim to make pipelines as wide and short as possible through aggressive parallelization, and by avoiding chaining deployments so that builds must pass through multiple stages before they become available for release. Rather, make it easy for the person performing the deployment to judge what can – and should – be deployed.

And of course follow the Deming cycle: measure the average cycle time for builds to pass through the pipeline, and the lead times to each environment, to judge the effect each change you make has on the efficiency of your delivery process, and continue to optimise accordingly.

We’ve also created a multiple-choice assessment that grades your team’s maturity at build and release management according to the model we present in Chapter 15 of the book. The final report you get when you complete it includes a table which lays out the maturity model. There’s also a more general agile assessment at the same site.

While a multiple-choice assessment obviously can’t replace a serious investigation of your organization’s capabilities, strengths, and weaknesses, it is a useful starting point in such an investigation. I was happy to hear Dave Thomas talk about the importance of assessments in his Agile 2010 keynote – doing an assessment is step one in his list of how to do an agile transformation programme (and the rest of the list has a lot in common with the material we set out in our book).

Although they’re created by ThoughtWorks Studios, both the white paper and the assessment are vendor neutral. There is of course advertising for the book and the ThoughtWorks Studios ALM suite at the end, because hey, we’re proud of our stuff.

If you have any feedback or want any further information, please feel free to leave a comment.