What mutation-testing frameworks exist for C/C++?

Question

Mutation testing has been out there for a while now, and it seems there are at least one or two commercial mutation testing frameworks for C/C++. Have you used them? What are your experiences? Are there any open source alternatives?

Answer 1

A brief search resulted in:

• PlexTest: http://www.itregister.com.au/products/plextest_detail.htm
• Insure++: http://www.parasoft.com/jsp/products/insure.jsp;jsessionid=baacpvbaDywLID?itemId=63
• MILU (may be only for C): http://www.dcs.kcl.ac.uk/pg/jiayue/milu/

With that said, you need to realize that mutation testing isn't particularly useful (at least from some stuff I've previously read). It's an interesting tool when faced with hard (metaphorically speaking) asserts and for making sure that data requirements are heeded to (when dealing with if and only if situations).

In my opinion, there are much more established ways of analyzing the robustness of code.

Answer 2

Notice that Parasoft's tool only generate equivalent mutations. That echoes the problem described on Wikipedia article about Mutation Testing - it is hard to distinguish between equivalent and non-equivalent mutations so they decided to stick with equivalent.

I tried another interesting tool that can automatically discover invariants in instrumented C and C++ code - it is called "Daikon". Essentially it is doing same thing as tool that generates equivalent mutations, but instead of identifying problematic code it gives you a set of invariants such as "A == B + 1". I think invariants are more useful because when you look at discovered invariant it gives you assurance that your code is correct if invariant make sense, and then you can convert invariants into asserts and that gives you more confidence when you change code.