svn deployment strategies for multiple groups of developers (not co-located) working on different components of the same project

Question

Our project is a content management system supporting several dozen of our websites. The development group started off small and in one location, and we dealt with a fairly standard coding/deployment strategy.

We coded off of trunk and insisted on a clean trunk. Every few days we would tag trunk and deploy to the test server. If all worked out, we'd deploy to Production and move on.

That worked well for a while until the team grew. We frequently faced situations where the revision that was tagged had problems that needed to be fixed before going to Production. While the developer responsible was working on those fixes, we had other developers committing changes to trunk. Once the original developer's fixes were complete, the new commits that were added would have to go along for the ride, further delaying the build because now there's additional validation that needs to be done.

In an attempt to rectify this, we created a separate trunk used strictly for releases. People would work in the main trunk, and then ask the project manager or development lead to merge their changes into the release trunk.

This worked for a while until the team got even bigger and more disjoint. We have teams of 3-5 people working in 4 geographic locations - some on the same components, others on different components with different priorities and release schedules. This was pretty much a full-time job and became a nightmare for the person managing the builds.

In an attempt to work around that, we started creating "release branches" off of whatever the latest Production tag was. People would commit to there ONLY what is ready to be tested and go to Production. Others would commit to trunk until it was their turn to merge. That took the burden of merging and resolving conflicts off of the build manager and on to the person owning the code.

This worked for about a week until we started having to do several "high priority emergency" releases. This effectively meant that we would:

1. Create a branch off of the latest Production tag
2. Add the emergency stuff to that branch
3. Tag that branch and release to Production
4. Merge all of the changes that were made in that branch into the regular "release branch" that is sitting in QA.

This is every day. Sometimes twice a day.

I've tried to relate this a bit to an open source project where there are developers all over the place who don't even know each other and they still seem to get by... but that comparison falls apart when new stable, tested, production-worthy builds are expected for "public" consumption several times a week (or day). If Firefox's daily build is a buggy mess, for example, at least users can go back to a previous version or use the latest stable release. That's not the case for our users. If our release is not perfect, they can't work.

Backstory completed, I now pose the question:

Given an environment where...

1. Developers are all over the place and working on different components.
2. Changes to some components can wait a week before being released, others can't wait even a day.
3. The application is mission-critical and changes must be tested and stable before being released.

... what suggestions or alternative workflows can you recommend to promote a saner process where the majority of the burden is not on one person?

Answer 1

Combined maintenance- and release-branches

I think you are having similar requirements like our people. A CI would help you automate but it does not solve the underlying organisational challenge.

So you have 2 types of checkins:

• normal non-urgent code (slated for release every once in a while)
• urgent "hotfixing" code (happens after a release if something slips through without enough testing or when customer X calls because he wanted the button pink not purple and he threatens to drop the contract :P)

Both cases are different and you need to seperate them. Your approach is allready close to an answer I think but you are doing "too much"

Let me describe you what we do:

Repository Layout

trunk (is our project with components and all)
branches
|
-- 1.0-stable-week-40
|
-- 2.0-stable-week-42
|
-- 3.0-stable-week-44
tags
|
-- 1.0.0
|
-- 1.0.1
|
-- 1.0.2
|
-- 2.0.0
|
-- 2.0.1
|
-- 3.0.0

As you can see we have a trunk for all the main development work. We also create stable branches for release preparation and testing every 2 weeks and we tag all releases when they go live.

Release lifecycle (generalized)

After a new release we maintain the branch (1.0 for example) until the next major release is pushed out. Our policy is that during that time ONLY critical fixes may be checked in to that branch. They go through minimal testing only and can be released in a matter of minutes by creating a new tag from our maintenance branch.

Halfway through the maintenance period (1 week after the release) we create a new branch from our trunk called "2.0". All not-so urgent development allready in the trunk will be in this release automaticly. More things can be added "carefuly" like urgent fixes that come from the currently active maintenance-branch (merge from 1.0 to 2.0 to trunk).

After another week passed and all testing was done the 2.0 branch gets tagged as 2.0.0 and released, given no major problems arised. The 1.0 maintenance-branch will be abandoned and deleted eventually.

This way we can seperate urgent from non-urgent changes and have relatively pain-free and stable releases. What you do is pretty much the same but you branch from a tag which when you'r finished you tag again. Thats a bit much :). Branching from tags is also bad style. branch from branches.

---

branch policies

It helps the team if you write down the policy for each of your branch-types enabeling them to do more on their own without having a release-guy constantly sitting in their neck and luring over their commits ;)

Our policies could be described this way:

• trunk:

  • no commits with syntax errors
  • receives merges from maintenance
  • no direct releases from this branch

• branches/X.X-stable

  • may receive urgent fixes only
  • should be "release-ready" at all times
  • developer MUST merge down his commit from here to any younger stable branch
  • If no younger stable branch is available merge to trunk

• tags/*

  • No commits here
  • used for deploy

Merging becomes also less of a mind-training once you try to merge into one "direction" only. (You might want to google for the tofu scale but thats getting a bit OT now ). See that merges are done continuously by developers and not release managers to limit the bottleneck. While one release is live and actively maintained receiving hotfixes, we allready start to prepare the next, isolating it from possibly unstable changes in the trunk and giving it time to "ripe".

You might have different requirements for the duration of code-incubation and testing or length of release iterations of course. Adapt :)

Answer 2

I'm a big fan of continuous integration along with test driven development.

Here's some links I'd recommend checking out:

• CI from wikipedia
• CruiseControl
• Xinc - CI for PHP (my fav, it's simple)