Running only changed or failed tests with CMake/CTest?

Question

I work on a large code base that has close to 400 test executables, with run times varying between 0.001 second and 1800 seconds. When some bit of code changes CMake will rebuild intelligently only the targets that have changed, many times taking shorter than the actual test run will take.

The only way I know around this is to manually filter on tests you know you want to run. My intuition says that I would want to re-run any test suite that does not have a successful run stored - either because it failed, or because it was recompiled.

Is this possible? If so, how?

Answer 1

ctest command accepts several parameters, which affects on set of tests to run. E.g. ,"-R" - filter tests by name, "-L" - filter tests by label. Probably, using dashboard-related options, you may also choose tests to run.

As for generating values for these options according to changed executables, you may write program or script, which checks modification time of executables and/or parses last log file for find failed tests.

Another way for run only changed executables is to wrap tests into additional script. This script will run executable only if some condition is saticfied.

For Linux wrapper script could be implemented as follows:

test_wrapper.sh:

# test_wrapper.sh <test_name> <executable> <params..>
# Run executable, given as second argument, with parameters, given as futher arguments.
#
# If environment variable `LAST_LOG_FILE` is set,
# checks that this file is older than the executable.
#
# If environment variable LAST_LOG_FAILED_FILE is set,
# check that testname is listed in this file.
#
# Test executable is run only if one of these checks succeed, or if none of checks is performed.

check_succeed=
check_performed=
if [ -n $LAST_LOG_FILE ]; then
    check_performed=1
    executable=$2
    if [ ! ( -e "$LAST_LOG_FILE" ) ]; then
        check_succeed=1 # Log file is absent
    elif [ "$LAST_LOG_FILE" -ot "$executable" ]; then
        check_succeed=1 # Log file is older than executable
    fi
fi

if [ -n "$LAST_LOG_FAILED_FILE" ]; then
    check_performed=1
    testname=$1
    if [ ! ( -e "$LAST_LOG_FAILED_FILE" ) ]; then
        # No failed tests at all
    elif grep ":${testname}\$" "$LAST_LOG_FAILED_FILE" > /dev/null; then
        check_succeed=1 # Test has been failed previously
    fi
fi

if [ -n "$check_performed" -a -z "$check_succeed" ]; then
    echo "Needn't to run test."
    exit 0
fi

shift 1 # remove `testname` argument
eval "$*"

CMake macro for add wrapped test:

CMakeLists.txt:

# Similar to add_test(), but test is executed with our wrapper.
function(add_wrapped_test name command)
    if(name STREQUAL "NAME")
        # Complex add_test() command flow: NAME <name> COMMAND <command> ...
        set(other_params ${ARGN})
        list(REMOVE_AT other_params 0) # COMMAND keyword
        # Actual `command` argument
        list(GET other_params 0 real_command)
        list(REMOVE_AT other_params 0)
        # If `real_command` is a target, need to translate it to path to executable.
        if(TARGET real_command)
            # Generator expression is perfectly OK here.
            set(real_command "$<TARGET_FILE:${real_command}")
        endif()
        # `command` is actually value of 'NAME' parameter
        add_test("NAME" ${command} "COMMAND" /bin/sh <...>/test_wrapper.sh
            ${command} ${real_command} ${other_params}
        )
    else() # Simple add_test() command flow
        add_test(${name} /bin/sh <...>/test_wrapper.sh
            ${name} ${command} ${ARGN}
        )
    endif()
endfunction(add_wrapped_test)

When you want to run only those tests, which executables have been changed since last run or which has been failed last time, use

LAST_LOG_FILE=<build-dir>/Testing/Temporary/LastTest.log \
LAST_FAILED_LOG_FILE=<build-dir>/Testing/Temporary/LastTestsFailed.log \
ctest

All other tests will be automatically passed.