Automatic / templated generation of test vectors in C++

Question

I want to find a good way to generate test vectors automatically. By way of example, I am testing an audio processing module by calling a function that exercises the module-under-test with the specified test vector and in doing so makes various checks for proper operation and correctness of module output.

void runTest(const char *source, double gain, int level);

The test vector is the triplet of source, gain and level. Here is the multidimensional space I want to test against:

const char *sources[] = {"guitar.mp3", "vocals.mp3", "drums.mp3"};
double gains[] = {1., 10., 100.};
int levels[] = {1, 2, 3, 4};

Values can have other properties, for example if vocals.mp3 has a dynamic rage of 2, guitar 5 and drums 10, we could conceive a representation like:

int dynamicRange(const char *source);

I want to be able to configure various test runs. For example, I want to be able to run:

// all permutations (total 36 vectors)
runTest("guitar.mp3", 1., 1);
runTest("guitar.mp3", 1., 2);
runTest("guitar.mp3", 1., 3);
runTest("guitar.mp3", 1., 4);
runTest("guitar.mp3", 1., 1);
runTest("guitar.mp3", 10., 2);
runTest("guitar.mp3", 10., 3);
// ...

// corner cases (according to dynamicRange)
runTest("vocals.mp3", 1., 1);
runTest("vocals.mp3", 1., 4);
runTest("vocals.mp3", 100., 1);
runTest("vocals.mp3", 100., 4);
runTest("drums.mp3", 1., 1);
runTest("drums.mp3", 1., 4);
runTest("drums.mp3", 100., 1);
runTest("drums.mp3", 100., 4);

// sparse / minimal tests touching every value for each parameter
runTest("guitar.mp3", 1., 1);  
runTest("vocals.mp3", 10., 2);  
runTest("drums.mp3", 100., 3);  
runTest("guitar.mp3", 1., 4);  

// quick test
runTest("guitar.mp3", 1., 1);

I want create the above code without lots of copy and paste either dynamically or using my compiler to do the legwork, for example:

// syntax tentative here, could be class/template instantiations
allPermutations(runTest, sources, gains, levels);
cornerCases(runTest, lookup(sources, dynamicRange), gains, levels);
minimal(runTest, sources, gains, levels);
quick(runTest, sources, gains, levels);

The above looks like dynamic C but my language is C++ and I am expecting to use templates and some combination of dynamic and static techniques. Perhaps even metaprogramming.

Combinations and variations would also be interesting. For example, I might want to use only the shortest input file. Or I might want to run all sources with corner-cases for gain and level. Or gain could also be a continuous range 1 to 100 but let's keep things discrete for now.

Before I start designing types, templates, representation, etc. I wondered if this is a problem that has been solved before or, if not, would any existing libraries, e.g. Boost MPL, be useful?

Answer 1

I think it would be useful if you introduce yourself to concept of All-pairs testing, and have a quick check for QuickCheck (it is the Haskell test framework which generates test cases randomly according to the given spec, and then checks that some properties are hold; there exists C++ version of it).

Regarding Boost.MPL in particular, I don't think it would help you for this task at all: you are not dealing with list of types here, are you.

My another advise on your upcoming design: don't overgeneralize. Before you start with types, templates, etc. implement 3 (three) reasonably different implementations, and then generalize what you already have at hand.