How to automate integration testing?

Question

I'd like to know something, I know that to make your test easier you should use mock during unit testing to only test the component you want, without external dependencies. But at some point, you have to bite the bullet and test classes which interact with your database, files, network, etc.

My main question is: what do you do to test these classes?

• I don't feel that installing a database on my CI server is a good practice, but do you have other options?

• Should I create another server with other CI tools, with all externals dependencies?

• Should I run integration test on my CI as often as my unit tests?

• Maybe a full-time person should be in charge to test these components manually? (or in charge to create the test environment and configure the interaction between your class and your external dependency, like editing config files of your application)

I'd like to know how do you do in the real world.

Answer 1

I'd like to know how do you do in the real world ?

In the real world there isn't a simple prescription about what to do, but there is one guiding truth: you want to catch mistake/bugs/test failures as soon as possible after they are introduced. Let that be your guide; everything else is technique.

A couple common techniques:

• Tests running in parallel. This is my preference; I like to have two systems, each running their own instance of CruiseControl* (which I'm a committer for), one running the unit tests with fast feedback (< 5 minutes) while another system runs the integration tests constantly. I like this because it minimizes the delay between when a checkin happens and a system test might catch it. The downside that some people don't like is that you can end up with multiple test failures for the same checkin, both a unit test failure and an integration test failure. I don't find this a major downside in practice.

• A life-cycle model where system/integration tests run only after unit tests have passed. There are tools like AnthillPro* that are built around this kind of model and the approach is very popular. In their model they take the artifacts that have passed the unit tests, deploy them to a separate staging server, and then run the system/integration tests there.

If you've more questions about this topic I'd recommend the Continuous Integration and Testing Conference (CITCON) and/or the CITCON mailing list.

• There are lots of CI and build|process automation tools out there. These are just representatives of their class of tools.