Unit testing for compiler errors

Question

How do you test for wanted raised compiler errors in unit testing?

Consider the code:

class ErrorTest
{
    OtherClass& read_write() {
        return other;
    }

    const OtherClass& read_only() const {
        return other;
    }

    private:
        OtherClass other;
};

How can I test for read_only() assignment? It's really important and should be firmly checked to properly generate compiler errors:

ErrorTest test;
OtherClass other = test.read_only();
test.read_write() = other.modify();
test.read_only() = other.modify(); /* This should error */

Answer 1

I guess the main question now is, are you testing that your code or the compiler at this point?

Testing the compiler isn't necessarily a bad thing... I've had compiler upgrades mask errors in the past, so it would be nice to ensure that you are getting the same set of safety checks that you expect.

However, you'll have to do a lot of legwork. Your unit test will have to spawn the compiler, capture it's output, and parse it for the correct error statement on the correct line. It's not trivial, and arguably not worth it.

A slightly easier approach might be to keep a directory of bad code, and have a script compile each file one at a time. Have an '#ifdef MAKEFAIL' flag in there that turns on the exact condition that should fail. Ensure the compiler returns 0 when you don't set that flag, and non-zero when you do. That assumes that the compiler returns non-zero on failure... I don't know if MSVC follows that rule.

A third option I'll throw out there, to address portability, is autoconf. It can be a pain to set up, but part of its purpose was to ensure that you have a sane development environment before compiling. You could add a test like this in it, and it let handle finding the compiler and trying it.

Answer 2

This seems a little like the automatic detection that happens when you "./configure" for a build from source on a *nix machine. The autoconf scripts build little programs and attempt to compile them to determine what's available and supported by your compiler.

It probably isn't practical to reuse any of that, but you might want the same model. Each test would have its own file or set of files, and a separate project file/make target/etc.. Then your test script would attempt to make each test case and check that the expected error occurred, either via grep or by comparing the output to a baseline output stored with the test cases.