Amazon (hardback, kindle)

InformIT (pdf, epub, mobi)

中文 (in Chinese)
日本語 (in Japanese)
한국말 (in Korean)

You can also see a list of all my publications and talks, including slides, on the Publications page.

Upcoming talks

Get the software

On DVCS, continuous integration, and feature branches

Translations: 中文

I like to say that feature branches are evil in order to get people’s attention. However in reality I lack the determination and confidence to be a zealot. So here is the non-soundbite version.

First, let me say that Mercurial (and more recently Git) has been my workhorse since 2008, and I love distributed version control systems. There are many reasons why I think they represent a huge paradigm shift over existing tools, as discussed in Continuous Delivery (pp393-394). But like all powerful tools, there are many ways you can use them, and not all of them are good. None of my arguments should be construed as attacking DVCS: the practice of feature branching and the use of DVCS are completely orthogonal, and in my opinion, proponents of DVCS do themselves – and the tools – a disservice when they rely on feature branching to sell DVCS.

First a few definitions. Note that some people use these terms in different ways, so you’ll need to temporarily erase any other definitions from your brain or my discussion won’t make much sense.

Continuous Integration is a practice designed to ensure that your software is always working, and that you get comprehensive feedback in a few minutes as to whether any given change to your system has broken it.

Feature branching is a practice whereby people do not merge their code into mainline until the feature they are working on is “complete” (i.e. done, but not done done1).

Mainline is the line of development – on a conventionally designated version control repository – which is the reference from which the builds of your system or project are created that feed into your deployment pipeline. Note that this definition applies perfectly well to DVCS and to open source projects, even on GitHub.

First, let’s dismiss the straw man argument. Every time you use version control you are effectively working on a branch: your working copy. On a DVCS, there’s a further level of indirection, because your local repository is effectively a branch until you push your changes to mainline. I have no problem with creating branches. What I do have a problem with is letting code that you ultimately want to release accumulate on branches.

Here are my observations. When you let large amounts of code accumulate off mainline – code that you ultimately want to release – several bad things happen:

  • The longer you leave it, the harder it becomes to merge, because as other people check in to mainline, mainline diverges from your branch. Really awesome merging tools help with this to some extent, but anyone who has done much programming has experienced situations where the code merged successfully but the application broke. The probability of this happening increases substantially – more than linearly – as the amount of stuff you need to merge, and the time between initial branch and final merge, increases.
  • The more work you do on your branch, the more likely it is you will break the system when you merge into mainline. Everyone has had the experience of getting in the zone and running with what seemed like a genius solution to your problem, only to find hours – or days – later that you need to scrap the whole thing and start again from scratch, or (more subtly and more commonly) that your check-in resulted in unintended consequences or regressions
  • When you have more than a handful of developers working on a codebase and people work on feature branches, it becomes difficult to refactor. If I refactor and check in, and other people have significant amounts of stuff on branches, I make it much harder for them to merge. This is a strong force discouraging me from refactoring. Not enough refactoring equals crappy code.

These problems go away when people regularly merge their work into mainline. Conversely, they become exponentially more painful as the size of your team increases. Furthermore, there’s a vicious circle: the natural reaction to this pain is to merge less often. As I am fond of saying, when something hurts, the solution is to do it more often, and to bring the pain forward. In this case this is achieved by having everyone merge to mainline more frequently.

However, it’s hard to do this if you’re working on a feature that involves a lot of work, or if you’re working on a large-scale change to your system. Here are some solutions.

  1. Break down your stories into smaller chunks of work (sometimes referred to as tasks). I have never yet found a large piece of work that I couldn’t split into smaller chunks – usually less than an hour and almost always less than a day – that got me some way towards my goal but kept the system working and releasable. This involves careful analysis, discussion, thought, and discipline. When I can’t see a way to do something incremental in less than a couple of hours, I try spiking out some ideas2. Crucially though, it means I get essential feedback early on as to whether my proposed solution is going to work, or whether it will have unintended consequences for the rest of the system, interfere with what other people are working on, or introduce regressions (this is the motivation for continuous integration.)
  2. Implement stories in such a way that the user-facing bits are done last. Start with the business logic and the bits further down the stack first. Grow your code using TDD. Check in and merge with mainline regularly. Hook up the UI last3.
  3. Use branch-by-abstraction to make complex or larger scale changes to to your application incrementally while keeping the system working.

How do you know when you’ve got too much unmerged stuff? Here’s a thought experiment. Imagine you’re the maintainer of an open source project, and someone you don’t know just submitted what you have locally on your branch as a patch. Would you merge it? Is the unified diff with mainline more stuff than you can easily keep in your mental stack when you read it? Is your intent sufficiently clear that someone else on your team could understand it in a minute or so without having to ask you? If you can’t answer “yes” to all these questions, then you need to stop working, stash your work, and split it into smaller chunks.

It should be clear that I’m not really attacking feature branches, provided your “features” are sufficiently small. However generally people who use feature branches overwhelmingly fail the test in the last paragraph, which is why it makes for a nice soundbite. Really experienced developers understand the trade-offs that using feature branches involve and have the discipline to use them effectively, but they can still be dangerous – GitHub is littered with forks created by good developers that are unmergeable because they diverged too far from mainline.

The larger point I’m trying to make is this. One of the most important practices that enables early and continuous delivery of valuable software is making sure that your system is always working. The best way for developers to contribute to this goal is by ensuring they minimize the risk that any given change they make to the system will break it. This is achieved by keeping changes small, continuously integrating them into mainline, and making sure there is a comprehensive suite of automated tests to verify that changes behave as expected and don’t introduce any regressions.

What about feature toggles?

See how I haven’t even mentioned feature toggles yet? Feature toggles don’t even come into play unless you have a complete, user-visible feature that you don’t want to appear in your next release. In this situation, the feature-branch alternative is to keep your feature branch unmerged until after your release. Unless you’re doing continuous deployment, or working on a small and experienced team, this is a painful and risky proposition.

However another (perhaps more important) use of feature toggles is to reduce the risk of release, and to increase the resilience of your production systems. The most important part of release planning is working out what to do when things go wrong (this is known as “remediation” in ITIL circles). Re-deploying an old version is usually what people opt for, but having the ability to turn off problem features without rolling back the whole release is a less risky approach. In terms of resilience, an important technique is the ability to gracefully degrade your service under load (see John Allspaw’s 40m talk at USI for a masterful discussion of creating resilient systems). Feature toggles provide an excellent mechanism for doing this.

For people who are skeptical about feature toggles, or interested in finding out more, I highly recommend that you look at Facebook’s video on release management (look for the section on “Gatekeeper”). Sarah Taraporewalla also just wrote an experience report on using feature toggles.

What about cherry picking?

Some people recommend keeping features out of mainline until they’re ready to be released, perhaps keeping them on a development branch that developers check in to, and then cherry-picking them in. However assuming you’re following the guidelines I provide above and your stories are small, the need to take features out is very much the exceptional case, not the normal case.

Furthermore, you then face all the problems that I mention elsewhere of getting from done to done done1 – the pain of integrating, regression testing, performance testing and so forth. With continuous delivery, you completely get rid of any integration or testing phases. In my experience, unless you have a small, experienced team working on a well-factored codebase with plenty of automated tests, these benefits massively outweigh the pain of occasionally having to take a feature out – and feature toggles provide a cheaper alternative if your analysis is done right.

Of course a key assumption here is that your stories are small and don’t spatter stuff all across the UI. I discuss this, and other considerations when analyzing stories in the context of continuous delivery, in another article.

1 A feature that is dev complete is “done”. A feature that is released is “done done”. One of the axioms of continuous delivery is that much of the pain and risk in releasing software occurs after software is “done”, particularly if your work isn’t sitting on mainline and needs to be merged. Thus “saving” your work on a feature until it is “done” doesn’t really make sense. Some people recommend not merging until after a feature is tested and showcased, but this seriously exacerbates the problems described below without provide much additional benefit, since tested, showcased features are still not “done done” (consider the need to integrate your code and run regression tests, for example).
2 Spiking is the practice of writing some code that you will throw away to test out an idea. The output of a spike is knowledge.
3 I am not trying to imply you shouldn’t prototype the UI early on.

  • James Baxter

    “Re-deploying an old version is usually what people opt for, but having the ability to turn off problem features without rolling back the whole release is a less risky approach.”

    I don’t agree that rolling back is **more risky** than switching off functionality – if the previous release worked and you can go back to that known state easily, its zero risk. Although it is certainly more convenient to degrade the service if you have changed data structures and started processing data!

    I think the primary value of DVCS is the ease of collaboration – it’s a boon for open source. But I can’t see the fundamental problem of merging branches being resolved, without version control moving beyond files and becoming aware of the actual language and higher level constructs like method names…

  • jez

    @James – agreed, if you’re using blue-green deployments or canary releasing, then “rollbacks” are very low-risk. That’s not quite the same as redeploying though. As you point out, data is usually the problem.

    • James Baxter

      Quite true, that is an important distinction :)

      On dividing up stories – as a small team we ‘get away’ with having substantial amounts of code modified locally, mainly relying on known what each other is working on. As you suggest I think we would benefit from earlier discussions of implementation details – we will give that a try, thanks!

  • fschwiet

    “if the previous release worked and you can go back to that known state easily, its zero risk”

    One could also deploy their update with the feature turned off, then once the deployment is verified as successful then toggle the new feature on. Then if the feature must be turned off you are returning to a verified state.

  • Wes Winham

    To mitigate merge pain, we have anyone developing in a ticket branch is constantly merging master back in to their branch. Before they do a pull request to get code review and a merge in to master, they’re required to have master merged in.

    To mitigate the idea of a ticket branch having broken code that we don’t know about, we have a separate set of Jenkins builds to pull in any changes to ticket branches and run our full suite of unit tests and selenium tests on every commit. We also don’t merge in a ticket branch after code review until it has passed the test suite with current master merged in.

    As you pointed out though, I think the number 1 rule for keeping feature branches in good shape is that they should always always always be small. We shoot for not having any tickets that we think will take more than a day to implement, test, review and deploy.

    • Devon M

      “To mitigate the idea of a ticket branch having broken code that we don’t know about, we have a separate set of Jenkins builds to pull in any changes to ticket branches and run our full suite of unit tests and selenium tests on every commit”

      While this appears to cover all your bases, anyone out there that has multiple feature branches that last a day or more should bare in mind that those branch-targeting test runs are not actually validating the completed releasable code.

      So while a developer will get rapid feedback on issues with branch A + mainline, issues that arise from branch A + branch B + branch C + mainline may not show up until later in the development cycle.
      As a team gets closer to a scheduled release and starts to merge their separate feature branches back in, you can end up with a lot of late-breaking problems, refactoring conflicts, and other nasty things that make the tail-end of the process much more painful.

      To me it sounds like your number one advantage is actually the fact that your feature branches always last one day or less.
      In that scenario, the benefit of having the feature on a branch is really just the ability to have the build server run the integration and selenium tests instead of having to run through them locally before a check-in.
      Using branches to carry a lot of code changes forward due to features that take several iterations to implement, trying to hop over a pending release from mainline, refactoring projects, etc. seems to be where a lot of the pain starts to show up.

  • Adam D

    I’m still not convinced that the definition of branch-per-feature is stuck in time. See my response here:

    Small features. Qa at the start. I hope you chime in.

    • jez


      The problem is that the pattern I describe, which has been called branch-by-feature for a long time, is very common and almost universally bad news. If you have another pattern which you think is different, you are shooting yourself in the foot by calling it branch-by-feature. You’ll just confuse things.

      Also, I have chimed in. This blog post is me chiming in :-)

  • Jilles van Gurp

    It’s not the practice of feature branching that is the problem but the fact that testing and continuous integration are not decentralized. In other words until your changes land on the central branch, you are not doing the due diligence of testing. Even worse, you are not making sure you have tested your changes before you add them to the main branch.

    So, the problem here is that you can’t do feature branches unless you also decentralize your testing and integration. There is no good reason why you can’t do that with git.

    Here’s a few simple practices that will address most of the issues:
    1) no change that will break CI on the main branch is allowed on the main branch. Zero tolerance on this one. In git terms: rebase against main, run ci builds on the feature branch, push. You can automate this even: jenkins pretty much supports this out of the box. If main breaks ever, somebody doesn’t get the basics: educate them with a big clue bat. Rationale, it is vital to keep main stable at all times. That way everybody on a feature branch will know it is safe to rebase.
    2) Rebase against main frequently, especially if you do big changes. Rationale: you are doing the changes, it is you that will take the pain of doing the integration work when things go wrong, not everybody else. The earlier you know about problems, the easier it is to fix. Feature branches are not about stopping rebases. If your feature branch is way behind, you have done something very wrong. Don’t ever do that without good reason.
    3) Commit frequently, keep commits as small as you can. Rationale: smaller commits are easier to analyze and fix when you have conflicts. Also, Git is really good at applying commits one by one. If you isolate the merge problems to a handful of commits, rebasing is pretty much painless.
    4) Push as early as you can. If the CI builds are green on your branch, push and don’t wait. Don’t accumulate integration work for others. Feature branches are not about hiding change but about isolating change. There’s a difference. One is a communication problem and the other is a proven strategy of divide and conquer.
    5) Communicate clearly around big restructurings. Rationale, everybody rebasing against your changes will experience some pain. You are causing people to have to do work, so tell them it is going to happen. I always ask people to push their changes before I push my big changes. That way, I can fix the integration issues on my side before I push.

    The situation you outline above, suggests you are failing on all fronts here. It is very common because people fail to adjust their central oriented way of working when they switch to Git. It is absolutely vital to using git effectively that you have as little dependencies on central branches as possible.

    • jez


      The situation you outline above, suggests you are failing on all fronts here. It is very common because people fail to adjust their central oriented way of working when they switch to Git. It is absolutely vital to using git effectively that you have as little dependencies on central branches as possible.

      Unfortunately this arrogant and patronizing attitude – “the reason you don’t agree with me because you don’t understand how awesome git is” – is rather too common amongst git-heads and makes the process of engaging with people on this issue quite tiresome. Let’s assume I know how to use git, and that I understand the process you outline. It’s really not rocket science.

      However I think that despite your bluster you are actually describing the same process as me, and in fact we agree. I have no problems with what you describe, because you’re frequently committing into mainline. What I think you have attacked is the straw-man I described just after my definitions. Correct me if I’m wrong.

      • didibus

        I got confused here, because I see what your article talks about, and what @Jilles describes in his comment as completely different.

        I got from your article that you should only have one branch, where you would do all your coding, testing and releasing from. Every programmer would work on that one branch, and would be required to commit to it even when what they are doing is unfinished (feature isn’t complete in terms of functionality). To mitigate the problem of having half done features in your product, you would reduce features to very small increment, so that in most cases, committing often will mean committing finished features. Or you would use feature toggles.

        I get from his comment that he suggests to use feature branch, but to merge mainline into the feature branches often (FI) and to merge it back to mainline (RI) when the feature is complete and that it has passed testing on it’s own branch (which would have been forward integrated with mainline before doing the tests).

        Am I understanding things completely wrong?

        • Jez Humble

          In Git (or any DVCS) every developer is *effectively* working on a branch, because until you push your changes to the repo that the canonical builds come out of, you are not integrated.

          But @Jilles nowhere says to wait until a feature is complete before merging it into mainline. He says “Push as early as you can. If the CI builds are green on your branch, push and don’t wait… Feature branches are not about hiding change but about isolating change.” Which I understand is the same as I am proposing, having incomplete features potentially going into production dark.

  • Pingback: james mckay dot net » Why does Martin Fowler not understand feature branches?

  • Leena

    I have one question regarding about this part:

    Implement stories in such a way that the user-facing bits are done last. Start with the business logic and the bits further down the stack first. Grow your code using TDD. Check in and merge with mainline regularly. Hook up the UI last.

    Isn’t it opposite of whats suggested for Outside In Development with tools like cucumber? In that you are supposed to start with UI and in the end reach at your model/business logic? Or is it that feature flagging is required whenever you are working on new features?

    • jez

      Perhaps, but it depends if you need to release before your story is complete. If your stories are sufficiently small it’s not a problem. Even when this is not true, in general, it’s best to have the UI be a thin layer with no business logic, and to write most of your acceptance tests against the service layer not the UI. Any tests you write directly against the UI will only pass right at the end of the story, but you can still write them first and then mark them so they don’t get run until your story is ready to go.

  • Leandro

    Two ways to help reduce this problem:
    1- know what each person is doing
    2- say how is your progress

  • Chris Nicola

    My thoughts on naming what we are trying to do rather than how we try to do it:

    “If I fail to build an airplane I don’t call it a chazwazer next time because it didn’t work the first time.”

    Just because attempts at BPF failed in the past doesn’t mean those who succeed at it today because of new knowledge and better tools should be forced to call it something else. This is a very geek way of thinking about things. “X is bad because it’s called X and it didn’t work for me when I tried it in the past.”

    Reposted from my twitter discussion with Adam:

    • jez

      Chris, it’s nothing to do with my leet credentials, the problem is when I’m trying to teach or coach people that A is bad and B is good. If A and B have the same name when they are actually very different practices, it makes things pretty difficult. In the situation when people are trying to work out how to do things right without someone to explain it, it’s even more confusing.

      • Chris Nicola

        It’s certainly not cut and dry, but I wonder how much more confusion we potentially introduce when a name clearly explains what we are trying to do but we chose to call it something else because it was taken by those who failed before?

        Perhaps simply qualifying the name by saying this is BPF a la DVCS/Git/Gitflow should be sufficient?

        • jez

          Why not call it feature topics?

  • James McKay

    Just a thought on what we refer to by “feature branches”:

    I always thought that “feature branch” meant a single branch for a single feature, ie a single user story. If you’re following an agile methodology such as Scrum, your features will naturally be fairly small anyway (typically less than a week or so).

    I’d have thought that if your “feature branches” last more than a couple of weeks, then either (a) you’re not breaking your tasks down enough, or (b) you have more than one feature on the branch. In either case, “feature branch” is something of a misnomer — “version branch” or “release branch” or “multi-feature branch” is perhaps a more appropriate term.

  • Pingback: On DVCS, continuous integration, and feature branches | Brent Sordyl's blog

  • Pingback: Jez Humble on continuous integration and feature branches « Hornet Dear Bernard

  • Frank Wathne

    Thanks for an interesting article. I guess every project has its unqiue set of constraints so its hard to generalize about what works an not. On my current project (30+ developers) we’ve been using feature branches for a while, and it seems to work pretty well. Doing CI on each branch separatly seems to reduce the risk of integration issues. I guess this works because the lifetime of each branch varies a lot so there is a constant flow of complete features comming into the main branch. Other branches will pull in these and run the “Branch CI”. This requires a strong gate on the main branch to reduce the risk of pushing in code that will break the integration and cause havock on other branches.

    • jez

      Hi Frank. Thanks for sharing your experience.

      You’re right that everything is governed by constraints. However although it may be hard, I’m not shy about trying generalize. Since generalization – or rather, synthesis – is a necessary condition for learning, it’s the only way for us as an industry to get better at what we do. In the case of using feature branches, the general lesson is that you have to merge regularly into trunk in order not to end up with integration hell at the end. That, I think, is well-understood, has passed the test of time, and your experience seems to confirm it.

      One note though – as I say in the blog post, continuous integration means that everybody is merging regularly onto trunk. Thus it’s a contradiction-in-terms to talk about doing CI on a branch. Running your build and unit tests every time someone makes a change on a branch is a good practice – but it’s not continuous integration. That only happens when people merge regularly into trunk.

  • Pingback: How we use environments in Chef at Recorded Future « Mansson about infrastructure

  • Steve Watson


    Reading this through and having discussed with the dev teams in my oprganisation, I can see the positives with this approach. It negates the need to undertake intergration testing once merging back code into the main trunk.
    The issue I have as a test manager is that code ccan end up being released to production when it has been checked in and build tested but not necessarily fully functionally tested. Lets say a release is due at 3pm on a given day, people are continually checking in code up until that point and build scripts are checking that the code is not broken. But there may not have been time to functionally test everything. That means you need to switch code or features off until they have been tested, which means a manual overhead.
    Is this approach best adopted when developing an application which is not already in production?

    I’d welcome feedback and thoughts on this.

    Thanks, Steve

    • jez

      Hi Steve

      Your concern is absolutely spot on. That’s why it’s essential to have a deployment pipeline that is triggered off trunk to ensure all necessary validations – including functional testing – are performed against any build before it can be considered production ready and releasable. The key point is that validations should be performed against a build off mainline. When you test against a build created off a branch, you are not testing the same thing you are going to release, which invalidates the point of the testing.

      Is this approach best adopted when developing an application which is not already in production?

      Absolutely not. The deployment pipeline, which relies on everyone merging into mainline, is a pattern used by organizations which deploy new software frequently. The whole point of continuous delivery is to release to users early on and then keep releasing new software frequently (based on business requirements).

      Just to reiterate, continuous delivery emphasizes building quality into your delivery process, and the deployment pipeline is what enables this.

  • Pingback: Perché Martin Fowler non ha capito il Feature Branching | Arialdo Martini

  • RDO


    I agree with the general spirit of your post; however, there are couple of aspects that I have issues with.

    1) “Hook up the UI last.” Even with the small print caveat, seems to me you are making a big assumption here. Many teams (even for enterprise-grade apps) are adopting UX/UI into their methodologies. While business requirements don’t change, the UI might need a different (bigger or smaller) set of data as well as underlying and related data for display. Also consider products with “fat” UIs and thin back-ends in these cases when UI functional/unit testing are more crucial to release than the back-end.

    2) Not sure if you are for or against (or on the fence) feature toggles, but I’d say these are just as dangerous, if not more so, that feature branches — specially if that feature you turned off, caused cascading changes to the data model. I personally am vehemently against feature toggles. If something is not ready for prime time or insufficiently tested, it should not go into the release. I know that it’s not always our call, but at least I’d be on the record opposing adding it to release.

    • jez

      Hi RDO

      re: 1) I am all in favour of integrating UX / UI into the methodologies. But you are making the assumption that integrating UX means you have to do big-bang feature releases. Of course you can do UX in an incremental, iterative way too: check out the new book by my colleagues on this subject. This works just fine even for rich clients.

      2) I prefer feature toggles to feature branches (subject to the caveats in my post, above). You can manage cascading changes using branch by abstraction. I am also against deploying stuff that is insufficiently tested – anything that goes into prod should have automated acceptance and unit tests passing for it. But it can still be switched off using feature toggles. Also note that you can’t really know if it’s “ready for prime time” without getting feedback from real users – the whole point of continuous delivery.

  • Pingback: Always Agile · Extent of Automation Caveat

  • Pingback: Feature branching fits the Kanban model – Branching by MMF | Rhyous

  • Dave Hounslow

    Jez, I think you are spot on when you say feature branches are evil. In my recent blog post I take a different angle and explain why I think the need for feature branching can be a smell of waste in an agile project.

  • Ian Monroe

    Is the problem that people are using features branches, or that they don’t know how to use git? I mean just merge in the mainline branch routinely. The issue of “The longer you leave it, the harder it becomes to merge, because as other people check in to mainline, mainline diverges from your branch. ” 95% disappears. I mean that simply shouldn’t be the case that not merging into mainline means mainline diverges.

    Of course I can’t really disagree with the thrust of your post since a lot of people don’t know how to use their VCS. And feature branches should be seen as a necessary evil and avoided if possible. But `git merge origin/master` every morning ain’t rocket science.

    • didibus

      Ya, I wish the author would have talked about this point. When doing feature branch right, you are supposed to merge the mainline into your branch at a minimum every couple of days. So your branch is only a couple of days behind the mainline. Once your feature is complete (the full functionality of the feature is there and devs working on it can’t find bugs inside their own environment anymore), you merge from your feature branch into mainline. This merge won’t be catastrophic, since it’ll only be a couple days old at max.

      I’d be curious to know if there are any negatives to this.

  • Pingback: On DVCS, continuous integration, and feature branches

  • YGG

    I am observing sort of deadlock situation in our company, which is delivering embedded devices with the software on them.
    We are delivering different versions to different customers, at a different time, so they get the product with different stage, and use different subset features of it in the common case.

    Once product shipped the customers are quite conservative of getting new versions, since this means quite big change and usually they will pass the new software version through their own validation process all over again. So usually they require only fix me this this and this and add this (basically cherry picking).
    Software Development team is pushing towards usage of continuous integration and delivery process with only one base tree, since we know it will lead us to more software platform and will leave us out of the load supporting as much trees and as much customers we have, and let us concentrate on continuous improvement.

    But management says: We cannot go there our customers do not want the new version, they want their old one. I tend to agree with them , since the software is often not in good condition.
    But the team struggles to stabilize it and deliver new features, and because lack of power(due to lots of support effort) we cannot stabilize.
    We often finish that conversation with the obvious absurd suggesting the managers to lie to the customer that we are delivering only that and that patch but shipping the whole new baseline.
    How does the company can exit out of this deadlock? The only way I see is that in terms of months and months, we should invest extra effort on the team to keep stable the main tree and with years gain our customer’s trust and just after that get rid of the feature/customer branches. Of course one fact that is quite “motivating” for the dev team is that each time customer problem appears on one of its special branches , this is handled with higher priority than improving the main tree. Which leads into situation that you are going to a long and exhausting war, with constantly falling moral of the army :(

    • Jez Humble

      > I tend to agree with them, since the software is often not in good condition.
      This is the root cause of the problem. And of course your customers are responding rationally.

      Over time, you need to move to a mainline development model, and require customers to upgrade in order to receive support. But for them to do that, you must first ensure that you fix the quality problems in mainline. Management has to ensure adequate resources are available to achieve this goal.

      Once this is done, you can move most of your customers onto the mainline version, requiring them to do so in order to receive support. You’ll probably have to keep some customers on older versions – but management will need to be courageous about who they allow to stay on old versions, and ensure they are always working to minimize the number of different versions being supported, with a goal of eventually moving everyone onto the mainline version.

      One absolute rule has to be that features never get backported to older versions. Hopefully, if you continue to deliver valuable features, this will act as a “carrot” to convince people to upgrade.

      HP had a similar problem with their firmware – you can read about how they moved to a mainline model in this book:

  • Luciano

    Linux has been (and it is currently) developed using feature branch. This is the demonstration that it works, and that feature toggle is a weird, dirty trick to be avoided.

    • Jez Humble


      If I had said that feature branching doesn’t work, or if were true that the Linux kernel (I assume you mean to refer to the kernel) didn’t use feature toggles, your comment might have had some kind of value. Unfortunately none of these statements is true.

      In particular, while feature toggles may be a “weird, dirty trick” (beauty is in the eye of the beholder), the Linux kernel is full of them – both compile time and run time ones. For example, the entire kernel configuration system – and kernel modules – are a set of feature toggles.

  • Pingback: (Вне)Классное чтение, Выпуск 1 | Open Space

  • Christophe

    Hi Jez,

    I like the comment about “constraints” and I am wondering, having seen a couple of companies failing to reach CI, whether they did not pay attention to any of their constraints and jumped into CI a bit blindly… Constraints were: poor test coverage (so you can build but your build is not usable), poor SW architecture (you change one bit and you rebuild the whole platform), poor CI infrastructure (a build & deploy takes hours), poor discipline (you can’t automate people), poor organisation (e.g. no build master to help initially), etc so trying to achieve CI means they stick to “everybody submit in the trunk everyday” but forget they only make money by shipping products.

    So is there a migration/transition path to CI? Could team branches be a temporary measure until most constraints are solved? In your experience, how long does it take a company to really reach CI when the “environment” is not there?

    In a nutshell, going to CI, does that require some smart planning?

    Thanks & regards!

  • Pingback: 关于DVCS、持续集成和特性分支 - 博客 - 伯乐在线

  • Pingback: Are Git and Mercurial Anti-Agile? | Evolvable MeEvolvable Me

  • Doug Shelton

    Beautifully written, concise and to-the-point, easy to understand article. I might have retitled it something more along the lines of “Why Continuous Integration into Mainline”, tho.

  • Pingback: Continuos Delivery http continuousdelivery com 2011 07 on… | iRo

  • Pingback: Using Pull Requests for commercial/private/proprietary development | Robert Daniel Moore's Blog

  • didibus

    How would this apply to software that needs to release bug fixes before new features are completed?

    • Jez Humble

      You can use a release branch for this. I have no problem with release branches, provided they are only used for critical bug fixes.

  • Pingback: Always Agile Consulting · Extent of Automation Caveat