Build Environment setup - Using .net, java, hudson, and ruby - Could really use a critique

Question

I'm trying to figure out the best way to stitch together a fast, repeatable, unbreakable build process for the following environment. I've got a plan for how to do it, but I'd really appreciate a critique. (I'd also appreciate some sample code, but more on that later)

Ecosystem - Logical:

1. Website - asp.net MVC 2, .net 3.5, Visual Studio 2010. IIS 6, Facebook iframe application application. This website/facebook app uses a few services. An internal search api, an internal read/write api, facebook, and an IP geolocation service. More details on these below
2. Internal search api - .net, restful, built using old school .ashx handlers. The api uses lucene, and a sql server database behind the scenes. My project won't touch the lucene code, but does potentially touch the database and the web services.
3. internal read/write api - java, restful, running on Tomcat
4. Facebook web services
5. A mocking site that emulates the internal read/write api, and parts of the facebook api
6. Hudson - Runs unit tests on checkin, and creates some installers that behave inconsistently.

Ecosystem - Physical:

All of these machines can talk to one another, except for Hudson. Hudson can't see any of the target machines. So code must be pulled, rather than pushed. (Security thing) 1. Web Server - Holds the website, and the read/write api. (The api itself writes to a replicated sql server environment).
2. Search Server - Houses the search api.
3. Hudson Server - Does not have permissions to push to any environment. They have to pull. 4. Lucene Server 5. Database Server

Problem

I've been trying to set this site up to run in a stress environment, but the number of setup steps, the amount of time it takes to update a component, the black-box nature of the current installers, and the time it takes to generate data into the test system is absolutely destroying my productivity. I tweak one setting, have to redeploy, restart in a certain order, resetup some of the settings, and rebuild test data. Errors result in headscratching, and then basically starting over. Very bad.

This problem is complicated further by my stress testing. I need to be able to turn on and off different external components, so that I can effectively determine the scalability of each piece. I've got strategies in place for how to do that for each dependency, but it further complicates my setup strategy, because now each component has 2 options. A mock version, or a real version. Configurations everywhere must be updated accordingly.

Goals

1. Fast - I want to drop this from a 20 minute exercise when things go perfectly, to a 3 minute one
2. Stupid simple - I want to tell the environment what to do with as few commands as possible, and not have to remember how to stitch the environments together
3. Repeatable - I want the script to be idempotent. Kind of a corollary to the Stupid Simple thing.

The Plan So Far

Here's what I've come up with so far, and what I've come looking for feedback on:

1. Use VisualStudio's new web.config transformations to permit easily altering configs based on envrionment. This solution isn't really sufficient though. I will leave web.config set up to let the site run locally, but when deploying elsewhere, I have as many as 6 different possible outputs for the stress environment alone (because of the mocks of the various dependencies), let alone the settings for prod, QA, and dev. Each of these would then require it's own setup, or a setup that would then post-process the configs. So I'm currently leaning toward just having the dev version, and a version that converts key configuration values into a ruby string interpolation syntax. ({#VAR_NAME} kinda thing)
2. Create a ruby script for each server that is essentially a bootstrapping script. That is to say, it will do nothing but load the ruby code that does the 'real' work from hudson/subversion, so that the script's functionality can evolve with the application, making it easy to build the site at any point in time by reference the appropriate version of the script. So in a nutshell, this script loads another script, and runs it.
3. The 'real' ruby script will then accept commandline parameters that describe how the environment should look. From there, 1 configuration file can be used, and ruby will download the current installers, run them, post-process the configs, restart IIS/Tomcat, and kick off any data setup code that is needed.

So that's it. I'm in a real time crunch to get this site stress-tested, so any feedback that you think could abbreviate the time this might take would be appreciated. That includes a shameless request for sample ruby code. I've not gotten too much further than puts "Hello World". :-) Just guidance would be helpful. Is this something that Rake would be useful for? How would you recommend I write tests for this animal? (I use interfaces and automocking frameworks to mock out things like http requests in .net. With ducktyping, it seems that this might be easier, but I don't know how to tell my code to use a fake duck in test, but a real one in practice)

Thanks all. Sorry for such such a long-winded, open-ended question.

Answer 1

I would recommend two things:

1. Centralize your configuration in a replicated, fault-tolerant repository, like ZooKeeper, CouchDB, etc. You can put all the information needed to bootstrap your various environments in this system, such that the only config you need for an individual component is the location of this repository. This will dramatically improve the scalability of your environment.

2. Use something like Puppet for deployment & server management. It will ensure that all your machines are in the proper state at any given time, and it will allow for easy deployment. It's incredibly powerful and will make your life much easier.