Release Management - releasing to a subset of users - how would it work for a public facing website

Question

I read somewhere (sorry don't exactly remember the source) that facebook has release Tuesdays. They release the new features to their internal employees first, then to a small set of external users and then to the whole world. I believe google also does something similar

I work primarily with Microsoft stack (TFS for source control, IIS, asp.net, sql server with huge data). Public facing sites of course, so they have to be up 24x7x365. Though I can envision releasing my api/dll only on one of the servers (in the webfarm) and testing it out, how would I do this if there are DB (stored proc signatures, table schema changes )? Presently we are versioning SPs (the new ones will be mySPNameV2, where as the old one will be mySPNameV1 - both taking different set of parameters hence the renaming) and the new APIs would use SP-V2 where as the old API would continue with SP-V1.

I see some design smell, but is there a better way to do it?

Edit: we release the new code to only one server and test it, what is difficult is how would you abstract (may be abstract is not the right word, but you get the idea) db schema changes from multiple concurrent versions of the application

Answer 1

At my company we almost always do any major release in a staggered fashion. We achieve this by adding a flag for every new feature in the users table. By default this flag is set to false; and as we are rolling out the feature to more users we simply switch the flag in the DB table.

This means at the application level, we have to make sure at all places we check for this flag. The code push goes to all servers. DB changes are made; but still only some users see the new feature.

At the database level, we make sure that any changes to SPs are "backward compatible". This is done by following some simple rules:

1. Any new parameters added to the SP have to go to the end of the parameter list.
2. New parameters should have a default value. This is done so existing calls to the SP does not break.
3. If existing parameters to SP has to be changed (or if order of parameters has to change), then obviously all calls to the SP is changed. But the SP is coded in such a way that it supports users that have the feature enabled as well as users that don't have it enabled.
4. Most table changes are related to adding new columns. It is really rare when we have to modify an existing column. All new columns are added with a default value or the allow NULLs.

As for the API, most of our parameters are passed as custom objects(structures). This way we can add a new parameter to the API methods and still prevent existing calls to the API from breaking.

Also, for each user we store the API version they are using. Depending on the version, users go to a different API URL. So basically when users make the first API call to authenticate, we pass a new API URL (based on API version for the user). For all subsequent calls, they are supposed to call the new URL. Salesforce.com also follows this for their API calls.

Answer 2

I was at QCon last year, when a guy from Facebook's Web Team was talking about it this approach. You are 100% correct that there will be code smells in order to implement that. But it takes a lot more than just a code smells (actually they call those code smells Gate Keepers :).

First of all - their way of representing and storing data is a lot more sophisticated, they are not using Foreign keys, or any other "advanced" features of the databases at all :), as far as I remember, their data is not that "relational" as we all want to keep ours :).

What you can do is add gate keepers (if (user is from New Zeland) { use then new version with the new classes } else { stick ot the old one }).

You can probably imagine to what kind of code duplication will this lead in case of structured model (e.g. you will need to have 2 Users, 2 Orders, 2 Order Details). Leaving that aside, I think a lot of regression will also take place if you don't be extremelly careful.

As far as I remember - they are releasing a lot more often than just Tuesdays. I think they have a continous deployment and code goes live every day, just they clean old Gate Keepers / schema changes Tuesdays as they switched all the users to the new functionality by then.

So basically:

1. You add gate keepers everywhere a new functionallity is taking place.
2. You maintain 2 versions of the Schema.
3. You add more and more users to switch to the new one.
4. You clean the old one.
5. You go to point 1 (if another team isn't already there :).

I realize this answer may not be full enough to qualify for an answer, but I heard this from their guy and thought it is worth sharing.