Feature Branching Is Not a Bad or “Un-CD” Practice

Alan Kay has written that the programming community is driven by fads. There is a fad today: trunk-based development (TBD). Trunk-based development is when you merge local code changes (from your local copy of the main branch) into the shared “trunk”—aka “main branch”—of a code repository, at least once a day.

TBD is a good practice. What is not good is that some in the industry are saying that “if you don’t do TBD in all cases, then you are not doing true continuous delivery and you are not doing things the right way”. There are now self-appointed TBD police who attempt to shame people who create a branch, saying they are “not doing true CI”. Frankly, I do not care if it adheres to someone’s definition of CI: what I care about is being effective.

To me, telling people they have to adhere 100% to a very strict practice is dogmatic and pedantic; and it is incorrect. In fact, I have seen alternatives to TBD used to great benefit in many circumstances. The main alternative is known as the “feature branch” approach.

One challenge is that these techniques have many variations. For example, if one creates a feature branch, one can pull changes into it on a frequent basis, thereby staying in sync with “main”. That completely avoids the “big merge” problem that TBD advocates often point to. Doing that trades cognitive load on others working from main for cognitive load on the developer who is working in the branch: but tradeoffs are not black-and-white. There is room for agency and judgment.

Another benefit claimed for TBD is that lots of feature branches are confusing to merge. For example, Jez Humble wrote in an email to me,

“Of course you can and should sync into a feature branch, but that doesn't help with the combinatorial problem of ultimately combining multiple feature branches in order to achieve the fast flow to production of the combined work of teams of developers.”

But this combinatorial problem only exists if different people are maintaining feature branches pertaining to unrelated features, and you try to merge them all. A very different case is when a feature branch is created by one person, and then merged by that person. If they pull changes from main on a frequent basis, and run tests locally, then when they merge, they are only dealing with their own changes.

Martin Fowler has written about the problem of “big merges”:

“The problem with big merges is not so much the work involved with them, it's the uncertainty of that work.”

But if one person is maintaining their own feature branch, and they are pulling changes from main daily, then there never is a “big merge”.

There is also the case of when a product feature requires changes to multiple repos. That is actually quite common. In that case, one can create a feature branch in each of those repos, and name each branch after the feature. Then code the changes to each repo, and run some local integration tests on them to make sure that the changes all work together. Mock everything else, but use local live versions of the changed repos. When it all works, merge each feature branch into its respective main.

Net result: no “big merge”, and the integration tests that kick off in the pipeline will likely all pass!—in other words, you kept the integration test job green. If that is not CD, then what is? It seems to me that that is kind of the definition of CD, even if the main proponent of CD does not like to do it that way.

Jez has written about using “branch by abstraction”. But he writes that,

“Admittedly, branch by abstraction can add more overhead to the development process, especially in the case that the codebase is poorly structured. You need to think hard and move a bit slower in order to make changes incrementally in this fashion. But in many cases the upside is worth the extra effort, and the larger the restructuring, the more important it is to consider using branch by abstraction.”

So there are pros and cons. He says that “it is worth it”. But he also says that you have to “move a bit slower” and that “there is more overhead”. Seems like there are tradeoffs, and maybe one size does not fit all. Branch by abstraction is certainly an approach to consider—it’s a good idea—but does it really rise to the level of a rule that everyone must follow, lest they be called out for “not doing true CD”? (and I have witnessed that happen)

He also writes,

“In the simplest case you build the entire abstraction layer, refactor everything to use it, build the new implementation and then flick the switch.”

But wait: before we can use this approach, we have to perform some major refactoring on the code. So is that done locally, or in main? If it is done in main, that will result in some pretty messy code before the abstraction layer is ready, at which time we can actually start migrating to the change that we really want to make.

So I think that there are issues that are being conflated. I vehemently disagree that one size fits all. I disagree that if one creates a feature branch that lives longer than a day, that one is “not doing CD”—and that claim is often made, and in a cancel culture kind of way: “You are not doing it the way advocated by the priest so you are bad”. That is dogmatic, ideological, extremist, and irrational. It robs us of agency.

In the book The Voltage Effect, John List writes,

“the most productive workers tend to be hired first, and if you want to continue to scale once that ‘superstar’ pool is exhausted, you must hire less productive people. Yet in this case the diminishing returns were compounded because the line only moved as fast as the slowest worker…”

Make sure that you don’t turn your creative processes into an assembly line, throttled by the lowest common denominator. His point is that we should not think in terms of the “average”, but instead should recognize that people fall on a spectrum. If we manage to “the average”, then we lose insight about what List calls “the margins”, and the margins give us insight—and opportunity.

Reed Hastings has said,

“With a fixed amount of money for salaries and a project I needed to complete, I had a choice: Hire 10 to 25 average engineers, or hire one “rock-star” and pay significantly more than what I'd pay the others, if necessary. Over the years, I've come to see that the best programmer doesn't add 10 times the value. He or she adds more like a 100 times.”

Would you expect those best programmers to work the same way at the hypothetical “average” one? If you do, then you take away their agency and stifle their abilities, and dumb down the whole organization, forcing them all to use methods that are optimized for “the average”.

We need to be careful that we don’t manage for the lowest common denominator. We need to empower our best, because our best don’t just keep the wheels turning—they help us to lift off.

Update 2022-04-08:

Some people have pointed to the DORA data (the excellent book Accelerate) that shows that trunk-based development seems to predict good performance for an organization. I am very familiar with that: in fact, Nicole Forsgren and I have corresponded many times about her data. People are misinterpreting her work. Consider these two statements:

1. TBD seems to predict good performance for an organization.
2. If someone on a team creates a feature branch, then the team’s performance will be worse.

These are not the same, and people are conflating them as equivalent. In fact, circumstances matter. Individuals matter.

Also, while Google is a proponent of TBD, what people forget is that Google has a system that they developed that runs complete integration tests on every commit. Not everyone has that. And Google was very successful long before TBD became fad, and before it became a social test that if you fail, you get shamed for “not doing things in the approved way”.

Many have pointed out that feature branches are indispensable for open source development. Well yes: the situation matters—tell Linus Torvalds that the Linux kernel team (who created git) cannot create feature branches and he will quickly tell you what he thinks and it will not be good. And there are a range of situations within organizations in which it might make senses to use a feature branch.

One approach is not best for every situation. People need to use their brains, and not follow a prescription for how to do their work. Programming should not return to the PMI days of following rules without thinking and without exercising judgment.

Update 2022-04-09:

Someone pointed out that the “CI let’s us see problems sooner”. Indeed, but original CI was not designed to find defects when running the CI server. The whole point of CI, as I recall from the early 2000s, was to find problems before the CI server took over. You were supposed to get the latest changes, run your tests locally, and then - if and only if all the tests passed - you were then supposed to push your changes, at which point the CI server would run all the tests again.

If the CI server’s test run failed, you had “broken the build”, which was considered very bad behavior.

So the main element of CI was to keep the build green - i.e., keep all the tests being run by the CI server passing. To do that, you have to test everything locally first - before you push your changes. Pushing changes that were incomplete or that would break the build was considered the worst of all behaviors. The twist was that you were supposed to make small but complete changes that were fully tested before pushing them. Small is key, because you were supposed to integrate often.

But the logical inconsistency is that if your change has broad scope, then you cannot make a small but complete change. So what you have to do is hide the changes behind a feature flag until, eventually after many days, they are complete enough to turn on; and when you turn them on, you should have tested them locally first. So contrary to the claims, you are not using CI to discover problems sooner, because your incomplete feature is turned off by a feature flag. All you have done is cluttered up the main branch.

Today the TBD evangelists have flipped this, and the most outspoken of them are trying to redefine CI, by saying that CI means that you are always pushing your changes, even if they are half-baked. That is not the CI that I learned in the early 2000s, and frankly it makes little sense to me; and if someone forced me to work that way, I would consider it to be a toxic environment that was imposing a highly opinionated work approach and taking away my agency as a developer. What do you think about that? - would you feel your agency was being stolen, and that you were not being trusted as a developer? Are we seeing a return of the dogmatic XP days, now coming from the TBD camp?

Update 2022-05-19:

I just came across this excellent article about branching: Better Git branching strategy — Multi-apps, monorepos and multiple teams in focus. These nine teams really thought this through, and they did not use TBD. If they had been forced to use TBD, it would have removed their agency, and they would not have gone through this learning journey, and would not have come up with this.