What tool to test complex multi-processes system?

Question

We have a data acquisition system made of many programs and scripts running together, some sequentially, most in parallel, on different machines.

We use Jenkins to build and run unit tests.

We now want to test the system as a whole by using a data generator as input to the system and checking the output of the data acquisition chain.

Simplified view of the steps to test the system :

1. Restore system (database, packages in the repository, relaunch virtual machines)
2. Configure the system as needed (write config files to be used, update database)
3. Launch the few parent processes needed and wait for them to be ready
4. Start data input
5. Once done, collect metrics and results, compare with expected results, check for error messages in the logs, ...

Ideally we shall run it with different configurations and input.

What tool would you use to do it ? Or would you try to do it directly in jenkins ?

Answer 1

You don't tell us enough about your setup to give you specific answers. It strongly depends on the language/environment you are using. You probably need a combination of one or more of the following:

• Vagrant/Chef or Puppet/Veewee/VirtualBox to automatically build VMs and configure and run them. That's assuming a Linux platform. Chef + Linux containers is a lot faster, especially if you run it on a ram disk (it is supposed to be a repeatable process, so only the artifacts need to be stored)
• A package cache with version control for all the system packages you need to install on the VMs
• Some framework to run acceptance tests, Cucumber, Robot or so. If your unit tests are good, you might be able to get away with smoke tests.
• A simulation of your system, that you can feed run-time results from the log so you know how and when to scale.

You are very likely to run into the slow tests problem as you have to populate a database, and might have to split test jobs. In general, the feedback loop to get this working is way too slow, and you'll probably need to be very creative to get an acceptable cycle time.

In different environments, there are different tools for this, and having multiple stacks is not much fun. Jenkins runs on a Java stack, Puppet & Chef on a Ruby stack, and there are similar tools for the Python, c & perl stack. You'll have to decide for yourself if you want to have best-in-class with integration problems, or a single stack and coding things yourself. Both can work, and neither is trivial. Team experience is most likely the deciding factor there.

The feedback loop in DevOps is a lot worse than that of a modern CI development environment, so you'll have to do much more planning & research up front. The number of experiments you can run in a day is much lower, at least a factor 20 or so, so making things work the first time is going to be difficult. Make sure you have experienced people doing that.

Answer 2

The answer is: Robot Framework (see 1 or 2).

• Supports Data Driven Tests and Behavioural Tests
• Tests are written using user-defined keywords, in a human-like language. Easy to develop, easy to read, easy to maintain.
• Lot of libraries to access different functionality (ssh connections, database, .....)
• Can be extended with custom libraries written in python, java or other languages.
• For each test a detailed log is generated

I don't know Jenkins, but I am pretty sure you can integrate Robot Framework with it. It's as simple as running a Python script.