Expected build-failure tests in CMake

Question

Sometimes it's good to check that certain things fail to build, e.g.:

// Next line should fail to compile: can't convert const iterator to iterator. my_new_container_type::iterator it = my_new_container_type::const_iterator(); 

Is it possible to incorporate these types of things into CMake/CTest? I'm looking for something like this in CMakeLists.txt:

add_build_failure_executable(     test_iterator_conversion_build_failure     iterator_conversion_build_failure.cpp) add_build_failure_test(     test_iterator_conversion_build_failure     test_iterator_conversion_build_failure) 

(Of course, these specific CMake directives don't exist, to the best of my knowledge.)

Answer 1

You can do this more or less as you described. You can add a target which will fail to compile, then add a test which invokes cmake --build to try to build the target. All that remains is to set the test property WILL_FAIL to true.

So, say you have your tests in a file named "will_fail.cpp" which contains:

#if defined TEST1 non-compiling code for test 1 #elif defined TEST2 non-compiling code for test 2 #endif 

Then you can have something like the following in your CMakeLists.txt:

cmake_minimum_required(VERSION 3.0) project(Example)  include(CTest)  # Add a couple of failing-to-compile targets add_executable(will_fail will_fail.cpp) add_executable(will_fail_again will_fail.cpp) # Avoid building these targets normally set_target_properties(will_fail will_fail_again PROPERTIES                       EXCLUDE_FROM_ALL TRUE                       EXCLUDE_FROM_DEFAULT_BUILD TRUE) # Provide a PP definition to target the appropriate part of # "will_fail.cpp", or provide separate files per test. target_compile_definitions(will_fail PRIVATE TEST1) target_compile_definitions(will_fail_again PRIVATE TEST2)  # Add the tests.  These invoke "cmake --build ..." which is a # cross-platform way of building the given target. add_test(NAME Test1          COMMAND ${CMAKE_COMMAND} --build . --target will_fail --config $<CONFIGURATION>          WORKING_DIRECTORY ${CMAKE_BINARY_DIR}) add_test(NAME Test2          COMMAND ${CMAKE_COMMAND} --build . --target will_fail_again --config $<CONFIGURATION>          WORKING_DIRECTORY ${CMAKE_BINARY_DIR}) # Expect these tests to fail (i.e. cmake --build should return # a non-zero value) set_tests_properties(Test1 Test2 PROPERTIES WILL_FAIL TRUE) 

You can obviously wrap all of this into a function or macro if you have a lot of these to write.

Answer 2

@Fraser's answer is a good approach, in particular the WILL_FAIL property is good advice. There is an alternative to making the failing target part of the main project though. The use case in the question is pretty much what the ctest --build-and-test mode is meant for. Rather than making the expected-to-fail target part of the main build, you can put it in its own separate mini project which is then built as part of a test. An example of how this might look in the main project goes something like this:

add_test(NAME iter_conversion     COMMAND ${CMAKE_CTEST_COMMAND}             --build-and-test                 ${CMAKE_CURRENT_LIST_DIR}/test_iter                 ${CMAKE_CURRENT_BINARY_DIR}/test_iter             --build-generator ${CMAKE_GENERATOR}             --test-command ${CMAKE_CTEST_COMMAND} ) set_tests_properties(iter_conversion PROPERTIES WILL_FAIL TRUE) 

This has the advantage that it will be part of the project's test results and will therefore be more likely to get executed regularly as part of normal testing processes. In the above example, the test_iter directory is essentially it's own separate project. If you need to pass information to it from the main build, you can do that by adding --build-options to define cache variables to pass to it's CMake run. Check the latest docs for recently corrected/clarified help on this area.