How do I add valgrind tests to my cmake "test" target

Question

I run unit tests by building with ninja and then running ninja test from my build tree:

cmake -G Ninja /source/tree ninja ninja test 

However, to run valgrind I need to run it manually:

valgrind rel/path/to/test 

I'd like valgrind to run automatically when I run ninja test. According to the cmake documentation "setting up [valgrind tests] is extremely easy", but when I run

ctest -D NightlyMemoryCheck 

I just get this error:

Cannot find file: /home/arman/tinman/deb/DartConfiguration.tcl    Site:     Build name: (empty) WARNING: No nightly start time found please set in CTestConfig.cmake or DartConfig.cmake Problem initializing the dashboard. 

I get a similar error when I follow the instructions from this SO question:

How do I make ctest run a program with valgrind without dart?

I don't know what dart is, but according the website it's some kind of online testing doodad.

Clearly extremely easy is not easy enough for me. Does anyone know of a solution that is so supremely easy that you'd have to be some kind of IT warlock to make it not work?

Answer 1

Here is a self contained example that shows how to add valgrind tests to a CMake project. The example consists of a single C++ source file main.cpp:

#include <iostream>  int main() {     double* leak = new double[10];     std::cout << "Hello!" << std::endl; } 

The code contains an intentional leak which should be picked up by valgrind. We also need a CMakeLists.txt file which requires CMake >= 2.8:

cmake_minimum_required(VERSION 2.8)  project (ValgrindExample)  include (CTest) add_executable(example main.cpp) add_test(example_test example) 

Here it is important to include the CTest module with include instead of just enabling tests with enable_testing(). The CTest module takes care of setting up the machinery for being able to run memory checks with tests (e.g., it locates the valgrind executable).

Now we can open a shell session in the project folder and create a Ninja build tree:

$ mkdir build; cd build $ cmake -G Ninja -DCMAKE_BUILD_TYPE=Debug .. 

We can build and run tests without valgrind in the regular way:

$ ninja [2/2] Linking CXX executable example $ ninja test [0/1] Running tests... ... 100% tests passed, 0 tests failed out of 1  Total Test time (real) =   0.01 sec 

To run tests with valgrind we have to use CMake's ctest executable with the test action memcheck:

$ ctest -T memcheck ... 1/1 MemCheck #1: example_test .....................   Passed    0.77 sec  100% tests passed, 0 tests failed out of 1  Total Test time (real) =   0.77 sec -- Processing memory checking output:  Memory checking results: Memory Leak - 2 

ctest prints a summary of the memory checking results. The detailed output of valgrind is located in a temporary directory in the build tree:

$ cat ./Testing/Temporary/MemoryChecker.*.log ==4565== 80 bytes in 1 blocks are definitely lost in loss record 37 of 64 ==4565==    at 0x10000B681: malloc (in /usr/local/Cellar/valgrind/3.12.0/lib/valgrind/vgpreload_memcheck-amd64-darwin.so) ==4565==    by 0x1000507DD: operator new(unsigned long) (in /usr/lib/libc++.1.dylib) ==4565==    by 0x100000F93: main (main.cpp:5) ... 

It is not possible to automatically run valgrind when you run ninja test because CMake's built-in test target cannot be modified and always runs tests in the regular way. We can however add a custom CMake target which invokes ctest with the -T memcheck option and then prints the detailed valgrind report:

add_custom_target(test_memcheck     COMMAND ${CMAKE_CTEST_COMMAND}          --force-new-ctest-process --test-action memcheck     COMMAND cat "${CMAKE_BINARY_DIR}/Testing/Temporary/MemoryChecker.*.log"     WORKING_DIRECTORY "${CMAKE_BINARY_DIR}") 

--test-action is the verbose version of the switch -T.

We then can invoke valgrind testing from Ninja with

$ ninja test_memcheck 

and get the results as if we ran valgrind manually.