How to keep automated tests fast?

Question

Automated tests MUST be fast to reflect real time project state. The idea is that:

1. after any commit to repository automated build is performed (as fast as it can be done).
2. if build succeeded automated tests are started. MUST be fast.

This is the best way i know to find out if your changes break anything.

At first it seemed that making a build fast is hard, but we managed to keep it around 100 sec. for a solution of 105(!) projects (MSVS 2008 C#).

Tests appeared to be not that simple (we use NUnit FW). Unit testing is not a big problem. It is integration tests that kills us. And not the fact that they are slower (any ideas on how to make them faster are much appreciated) but the fact that the environment must be set up which is MUCH slower(atm ~1000 sec)!

Our integration tests use web/win services (19 so far) that needs to be redeployed in order to reflect latest changes. That includes restarting services and a lot of HDD R/W activity.

Can anyone share the experience on how environment and the work flow should/can be organized/optimized to fasten automated testing phase. What are the "low level" bottlenecks and workarounds.

P.S. books and broad articles are welcome, but real world working solutions are much more appreciated.

Answer 1

There are a number of optimization strategies you can do to improve the throughput of tests, but you need to ask yourself what the goal of this testing is, and why it needs to be fast.

Some tests take time. This is a fact of life. Integration tests usually take time, and you usually have to set up an environment in order to be able to do them. If you set up an environment, you will want to have an environment which is as close to the final production environment as possible.

You have two choices:

1. Optimize the tests, or the deployment of the tests.
2. Don't do them as often.

In my experience, it's better to have an integration environment which is correct, and finds bugs, and represents the final production environment adequately. I usually choose option 2 (1).

It's very tempting to say that we'll test everything all of the time, but in realilty you need a strategy.

(1) Except if there are loads of bugs which are only found in integration, in which case, forget everything I said :-)