How does your company deploy its software?

Question

I am currently on a short research project. The company I work at has a very heavy release process that is getting worse as time progresses. We are encountering more and more issues with each release, which is starting to severely impact our delivery schedules and the quality of each release. We provide a large SAAS product that is deployed to the Internet on a very large web farm. Our deployment process is currently handled by a dedicated team, with minimal developer involvement. We are primarily a .NET shop, however we have a couple Java components as well.

I am researching how we could improve our QA and deployment process to reduce waste and bring more of the process under the wing of our dev teams. I am interested in hearing about how your company deploys your products (preferably SAAS, but not limited to such products) to production, as well as the journey through testing on its way there. I am curious what has worked, and what hasn't, and I'm sure many of you have stories to tell.

EDIT (Additional RFC):

As I have continued my research, I came across the concept of "Continuous Deployment", apparently pioneered by the IMVU 3d online community team. It sounds like an intriguing concept, if perhaps a little complex. I am curious if anyone here on SO has any experience with continuous deployment? Particularly with a large, complex project that has many parts to it. You don't necessarily have to continually deploy to production...for our short-term needs, we would only look at continuous deployment to internal dev/qa/perftest environments. If anyone has implemented continuous deployment, I am also curious to hear how you managed database schema and data changes/rollbacks.

Thanks!

Answer 1

We deploy a financial services SaaS solution to the Amazon AWS cloud environment. Our solution is 100% Java so many of the tools don't apply to you but the concepts should.

First of all, we reduce the number of surprises when it comes time to do a release by running a continuous integration process. Any time source code is checked in by any developer, the entire solution is automatically built and all unit tests automatically run. Failure notifications are emailed to the developer in question and team lead.

AWS is built around the concept of virtual machines. We leverage this by creating a virtual machine image (Amazon calls them AMI's) that contain the specific version of the OS and applications (Java, DB, etc) that we desire. The build process creates all deployable artifacts and then copies them to a running instance based on that AMI. It's the exact same process for all environments (TEST, DEMO, PROD), except for a single configuration project that encapsulates version differences (e.g. connection strings).

QA tests the result in the TEST environment. Once they approve the version, we repeat the build process targeting PROD (using the exact same version control revisions. Remember, the only difference between each environment is one config project).

At that point we create a virtual machine (instance) running on the base AMI with the PROD code base and call it STAGING. STAGING goes through a series of acceptance tests, both automated and manual. Here's the really nice part... now, we burn this environment (base AMI plus new version of our application) into a new AMI (virtual machine image). Then we create new running instances of our app servers based on this new image, update the load balancer to point to these new instances, and just kill the old instances. That's the beauty of using virtual machines (at least, that's one of the beauty's).

Whenever we need to adjust capacity (peak hours/days), we create new application server instances from the same production AMI and register them with the load balancer. When the peak is over, we just remove them from the load balancing rotation, then kill the extra instances after a few minutes (to allow for existing sessions to complete).

Answer 2

We have a SaaS solution and basically use the same process (continuous integration server, deployment scripts for test-staging-production) as Eric above, but we deploy it to our own infrastructure using custom scripts based on PSTools (http://technet.microsoft.com/en-us/sysinternals/bb896649.aspx) to do all the copying to the farm nodes.

For each deployment we evaluate if its possible to allow different nodes have different versions of the app (ie. no data integrity risks) or we have to bring the app off line for a few seconds to sync all the nodes (it usually takes about 20 seconds for the app to be back online, since it just coping the apps, from one master node); but the whole key is to have a "one-key" deployment process setup.