Is mutation testing useful in practice?

Question

Do you have any examples of real life applications of mutation testing? Does it work better than simple test coverage tools? Or is it useless?

What are the advantages/disadvantages of mutation testing in the real world?

Answer 1

The usefulness of unit tests is no longer discussed. They are essential in conception of a quality application. But, how can we assess their relevance? A code coverage indicator up to 100% doesn’t mean the code is 100% tested. This is just a view of executed code during unit tests execution. Mutation testing will allow you to have more confidence in your tests.

This is a two step process:

1. Generate mutants.
2. Check that the mutations are found by the tests.

I wrote a entire article about this process, including some concrete cases.

Answer 2

I looked at mutation test some time ago as a method for checking the efficacy of my automated regression testing scripts. Basically, a number of these scripts had missing checkpoints, so while they were exercising the application being tested correctly, they weren't verifying the results against the baseline data. I found that a far simpler method than changing the code was to write another application to introduce modifications to a copy of the baseline, and re-run the tests against the modified baseline. In this scenario, any test that passed was either faulty or incomplete.

This is not genuine mutation testing, but a method that uses a similar paradigm to test the efficacy of test scripts. It is simple enough to implement, and IMO does a good job.