A better way to handle branching and merging?

Question

We run a web development shop with ~20 developers working on ~30 different sites at any given time and we are sinking an incredible amount of time into managing our subversion repositories - there has to be a better way.

Our client sites generally have 3 separate deployment environments: development (trunk), staging (branch), and production (branch).

New features get reviewed internally on development, then merged with staging for client review and approval, and finally merged to production.

Our current workflow: Every developer who is working on a major new feature for a client will create a branch from trunk, work on their feature, while regularly updating from trunk, and then merge the changes back to trunk (development) for internal review. Developers working on minor changes or fixes, will make them directly in the trunk.

After internal sign off, the changes are then merged to staging. If changes need to be made, they will most often be made in trunk, and then merged to staging. Once approved, the changes will be merged with production and then deployed.

New features are not reviewed sequentially internally or by clients, and the whole thing gets rather messy. It seems we are using the wrong processes - there has to be a better way to do this. We are very interested in learning how to utilize version control better, but we are lacking the experience to jump start the process.

What are best practices for these scenarios? In addition to this forum, we are interested in engaging an experienced consultant, who can help us improve our processes.

Thank you!

Answer 1

You might consider changing to Git rather than Subversion. Git supports this kind of branching model extremely well and is very lightweight. It takes a little bit of getting used to, especially if you are moving from Subversion, but ultimately it buys you a lot.

A suggested Git workflow for a new feature could be:

1. Create a branch off trunk.
2. Implement feature, including iterations of internal review, on the feature branch.
3. When client signs off on the new feature merge the branch into both trunk and production and (optionally) delete the branch.

You don't really need to have a staging branch for this workflow. You also don't need to keep the feature branches once they are merged - Git will remember the change history.

This model supports asynchronous/out-of-order feature development and review pretty well, because each feature branch is essentially independent.

Branches are very cheap in Git and merging is usually pretty trivial.

Answer 2

I think branching by feature (or by team) would be better than branching by developer. The feature can be tested and previewed before merging through Development (Trunk) and into Staging branch.

My team has had a similar "Branch by Quality/Environment" structure that I spent some months researching how to best convert so developers worked together and we could reduce the merge tax. I'd suggest the following:

• Development (Trunk / Main / Root branch)
  • (FeatureName1[Version]) - replaces branch for each Developer.
  • Staging (if you are keeping all releases in a single branch)
  • Production
  • (ReleaseName)(MajorVersion)Staging - My preference to isolate different releases even if released to same environments
  • (ReleaseName)(Version)Production

Staging fixes are made directly in the Staging branch (or short-lived child branch of Staging if QA and/or risk mandate isolation), then merged fix back to Development (Trunk) as soon as fix is accepted. CAUTION: Pay close attention to any merges from Development while a Staging fix is in progress.

From QA Perspective I strongly prefer the final staged-and-tested build be the identical binaries/files released to production. The only change should be configuration files (with different configuration settings checked in to a repository). This means I typically make final build in "Staging" and the "Production" or ReleaseVersion branch is a read-only archive that nobody builds. (A label or tag may be used instead of this safekeeping branch if said lable is immutable... which is not the case for TFS 2010 :-(.)

See the cool diagrams in Visual Studio TFS Branching and Merging Guidance MSDN article from Feb 2011. To scale this out see Branching for Scrum and Parallel Feature Teams working on multiple releases in development. Monthly releases to production.. Both of these come with some excellent diagrams that apply to any version control system.

I'd also suggest looking at the TFS Branching Guide and the discussions forum in this same location for additional good patterns and guidance. (Branching methodology is largely independent of tooling with exception when avoiding tool weaknesses.)

I see at least one fundamental flaw in the original branching process:

After internal sign off, the changes are then merged to staging. If changes need to be made, they will most often be made in trunk, and then merged to staging.

If a change to Staging is made in Development and then merged to Staging again then you've just inherited all other changes made to Development branch since last merge to Staging. OR if you cherry-pick the change (leaving all other changes to be merged later). SCM tools are getting better at handling this scenario, but cherry-pick merges still introduces significant risks.

FYI: The pattern originally described is similar to "Branch By Quality" as described by Jeff Levinson. It can work, but you have to look carefully at how to manage hotfix/qfe branching and merging changes back to Trunk.

Answer 3

You don't exactly say how it get messy, but I'm guessing it's when everyone merges their feature branch back to the trunk. I would suggest this as a workflow:

1. Everyone working towards the next deployment works on the trunk and commits regularly; only features that take more than one cycle or that are likely not to be included in the next delivery should be worked on a separate branch. We call those development branches and they should be merged at the start of a development cycle - not at the end.

2. When you're ready to stage, create a release branch. Continue the trunk as the next release, merging in any development branches targeted at the next release. At this point bug fixes may have to be merged from the branch back to the trunk, but only bug fixes and not feature development should be added on the branch, so it shouldn't be too bad. In practice, to reduce unnecessary merging, we usually hold off creating the release branch until we're really ready to begin working new features.

3. When the time comes to move the code into production, simply tag the release branch and keep using it. Use this branch to maintain this version of the code with patches until all you client sites have replace them with the next version.

This model where you develop on the trunk and each release is kept on a single branch works very well for us in an environment where we sometimes have customers wanting fixes to very old versions of our software.