Catch.hpp unit testing: How to dynamically create test cases?

Question

I am using CATCH v1.1 build 14 to do unit testing of my C++ code.

As part of the testing, I would like to check the outputs of several modules in my code. There is not a set number of modules; more modules may be added at any time. However, the code to test each module is identical. Therefore, I think it would be ideal to put the testing code in a for loop. In fact, using catch.hpp, I have verified that I can dynamically create Sections within a Test Case, where each Section corresponds to a module. I can do this by enclosing the SECTION macro in a for loop, for example:

#include "catch.hpp"
#include <vector>
#include <string>
#include "myHeader.h"

TEST_CASE("Module testing", "[module]") {
    myNamespace::myManagerClass manager;
    std::vector<std::string> modList;
    size_t n;

    modList = manager.getModules();
    for (n = 0; n < modList.size(); n++) {
        SECTION(modList[n].c_str()) {
            REQUIRE(/*insert testing code here*/);
        }
    }
}

(This is not a complete working example, but you get the idea.)

Here is my dilemma. I would like to test the modules independently, such that if one module fails, it will continue testing the other modules instead of aborting the test. However, the way CATCH works, it will abort the entire Test Case if a single REQUIRE fails. For this reason, I would like to create a separate Test Case for each module, not just a separate Section. I tried putting my for loop outside the TEST_CASE macro, but this code fails to compile (as I expected):

#include "catch.hpp"
#include <vector>
#include <string>
#include "myHeader.h"

myNamespace::myManagerClass manager;
std::vector<std::string> modList;
size_t n;

modList = manager.getModules();
for (n = 0; n < modList.size(); n++) {
    TEST_CASE("Module testing", "[module]") {
        SECTION(modList[n].c_str()) {
            REQUIRE(/*insert testing code here*/);
        }
    }
}

It might be possible to do this by writing my own main(), but I can't see how to do it exactly. (Would I put my TEST_CASE code directly into the main()? What if I want to keep my TEST_CASE code in a different file? Also, would it affect my other, more standard Test Cases?)

I can also use CHECK macros instead of REQUIRE macros to avoid aborting the Test Case when a module fails, but then I get the opposite problem: It tries to continue the test on a module that should have failed out early on. If I could just put each module in its own Test Case, that should give me the ideal behavior.

Is there a simple way to dynamically create Test Cases in CATCH? If so, can you give me an example of how to do it? I read through the CATCH documentation and searched online, but I couldn't find any indication of how to do this.

Answer 1

There is a way to achieve what you're looking for, but I'd observe that you're going about this the wrong way:-

Unit tests are to meant to test each unit, i.e. you write a component and a test to verify the correct behaviour of that component. If you later decide to change one component in some way, you update the corresponding test.

If you aggregate all of your tests for all of your components into the same file, it becomes much harder to isolate the unit that behaves differently.

If you want to factor out the testing of a component because it's essentially the same across all your components, you could do one of the following:

1. Extract the common tests to a separate header file

You can #define the type name of the component you wish to test and then include a header file with all the tests in it:

// CommonTests.t.h
#include "catch.hpp"
TEST_CASE("Object Can be instantiated", "[ctor]")
{
   REQUIRE_NOTHROW(COMPONENT component);
}

// SimpleComponent.t.cpp
#define COMPONENT SimpleComponent
#include "CommonTests.t.h"

This is straightforward to do, but has a downside - when you run the test runner, you'll have duplicate tests (by name) so you can only run all tests, or by tag.

You can solve this by stringizing the component name and pre/appending it to the test case name.

** 2. Call common tests by parameterising the component **

Put your common tests into a separate file and call the common test methods directly:

// CommonTests.t.h
void RunCommonTests(ComponentInterface& itf);

// CommonTests.t.cpp
void RunCommonTests(ComponentInterface& itf)
{
  REQUIRE(itf.answerToLifeUniverseAndEverything() == 42);
}

// SimpleComponent.t.cpp
#include "SimpleComponent.h"
#include "CommonTest.t.h"
#include "catch.hpp"

TEST_CASE("SimpleComponent is default-constructible", "[ctor]")
{
   REQUIRE_NOTHROW(SimpleComponent sc);
}

TEST_CASE("SimpleComponent behaves like common components", "[common]")
{
  SimpleComponent sc;
  RunCommonTests(sc);
}