What are some good strategies to allow deployed applications to be hotfixable?

Question

In an ideal world, our development processes would be perfect, resulting in regular releases that were so thoroughly tested that it would never be necessary to "hotfix" a running application.

But, unfortunately, we live in the real world, and sometimes bugs slip past us and don't rear their ugly heads until we're already busy coding away at the next release. And the bug needs to be fixed Now. Not as a part of the next scheduled release. Not tonight when the traffic dies down. Now.

How do you deal with this need? It really can run counter to good design practices, like refactoring your code into nice, discrete class libraries.

Hand-editing markup and stored procedures on a production server can be a recipe for disaster, but it can also avert disaster.

What are some good strategies for application design and deployment techniques to find a balance between maintenance needs and good coding practices?

Answer 1

[Even though we test a lot before we release, ] What we do is this:

Our SVN looks like this:

/repo/trunk/
/repo/tags/1.1
/repo/tags/1.2
/repo/tags/1.3

Now whenever we release, we create a tag which we eventually check out in production. Before we do production, we do staging which is [less servers but] pretty much the same as production.

Reasons to create a "tag" include that some of the settings of our app in production code are slightly different (e.g. no errors are emailed, but logged) from "trunk" anyway, so it makes sense to create the tag and commit those changes. And then checkout on the production cluster.

Now whenever we need to hotfix an issue, we fix it in tags/x first and then we svn update from the tag and are good. Sometimes we go through staging, with some issues (e.g. minor/trivial fixes like spelling) we by-pass staging.

The only thing to remember is to apply all patches from tags/x to trunk.

If you have more than one server, Capistrano is extremely helpful to run all those operations.