How does one achieve .NET Deployment Nirvana?

Question

Our company is nearing its "go live" date (and its getting a QA department date), and I'm trying to define the right operational processes to support this. A big consideration of mine is how to avoid the deployment/configuration hell that has inevitably occurred. Have any of you found a good solution for handing off builds to the non-programmers so that they could successfully install and configure it in a QA, staging, and production environment?

A full environment for us is composed of a mixture of heterogeneous scheduled tasks, Windows services, and web sites, all of which can be scaled out through parallel deployment. Thankfully, the means of configuration is consistent. Unfortunately, it's all managed through .NET web/app.config files. In my experience, QA and ops folks always mess up when trying to modify them (XML is surprisingly hard for most people to handle!)

Here's the options I'm considering:

Using machine.config files

This is something I haven't done in practice, but it looks promising. If we create a machine.config template containing every setting for every application that can vary by environment, this would allow an admin to make all changes to one file and deploy it to each machine in the environment.

Pros:

This potentially reduces the number of steps necessary to deploy a system

Cons:

Having to somehow document configuration schema changes

Unknowns:

We make use of custom config sections and other configuration extensions that reference assemblies. Would this require us to install our .NET assemblies in the machines' GACs?

Perform config file manipulations in build process

If we set up the QA, staging, and production environments so they appear identical to our software (virtual servers and LANs, etc), QA should be able to transition ready software with no configuration changes directly to the staging environment, and staging to production. With this setup, theoretically we could hand to QA pre-configured foo.config files that nobody needs to touch.

Pros:

Engineering would be more adept at ensuring that configuration files are valid

Cons:

It may be considered poor security practice for engineering to be aware of production configurations (a poor argument, IMHO)

Have a network-centralized settings repository

This one doesn't look attractive to me, because I tried this in three ways that were ultimately failures:

1. At a prior company, we had the configuration settings in the database, but of course you can't put them all in there, since you need to configure the connection string to that database. Also, it was just as difficult to ensure that the database got properly updated before deployment.
2. Another approach we'd tried was to have a networked service that worked as sort of a centralized registry. This almost worked, but there were always issues with local caching, ensuring that the URL to the config server was properly configured, and of course configuring the config server.
3. Active Directory? Ew! Need I say more?

Thoughts?

How successful have you been with using the options I'm considering? Are there any alternatives to these that have worked well for you?

Answer 1

I'd say do a couple of things:

First, use a staging server. Whether for engineering or non-engineering, have a location where you perform "mock deployments" of your code to the server, and where it's tested from there. This serves a good purpose of giving a distinct "production-like" environment to test from, and it allows for deployment training without causing everybody to go all freaky and shaky from fear of nuking everything on a deployment. It costs a bit more in terms of hardware, but it's probably worth it from the errors that it prevents.

Second, if your configuration files are truly complex and they're hard for non-engineers to construct, create a quick tool that will create your validation files for deployment. A simple website or even client-side app that just takes the basic deployment parameters, does some validation on them, and then saves everything in the right format can do wonders in terms of helping out some of the less-technical folks. The confidence that having a tool that validates your input can be really useful for those types of folks, and knowing that you're always going to have well-formed XML with validated results can save some engineer worry time as well.

Answer 2

We have an custom exe that runs after builds that we use.

Our projects have 4 config files

web.config -- development (local box)
web.integration.config -- alpha testing (runs on our alpha server)
web.staging.config -- beta testing (runs on our beta server)
web.production.config -- production (runs on our production server)

the exe simply deletes the all files except for the one needed and then renames it to web.config...

We don't allow non-developers (QA, DBA, etc) to manipulate the config files as they could change to production values (mail server, sql server) and cause some serious issues...

It works very well for us