Making unit tests fail quickly for mutation testing

Question

One problem encountered with mutation testing is that it's slow, because by default you do a full test run (either a test file, or a suite of test files) for each mutation generated.

One way to make mutation testing faster would be to stop the test run for a given mutant once a single failure has been encountered (but only during mutation testing). Even better would be for the mutation tester to remember what was the first test to kill the last mutant, and give that first to the next mutant. Is there anything in ruby that does either of these things, or is my best bet to start monkey patching?

(Yes, I know unit tests ought to be fast. And showing all of the failed tests is useful outside of mutation testing, as it helps you not merely identify that something's wrong, but pinpoint where it's going wrong)

Edit: I'm currently using heckle with test/unit. If it's not possible for test/unit to remember which tests fail between runnings, maybe heckle or something running heckle could remember it.

Answer 1

Your best best is to check out the heckle source from github, patch it, and submit that patch to the developers. You ought to be able to write a custom test runner for heckle.

Monkey patching is never the answer for something like this. In fact, monkey patching is almost never the answer for anything.

Answer 2

My mutant tool uses the rspec2 --fail-fast option to stop immediately once a failing example was encountered. Together with the --rspec-dm2 strategy that only executes affected unit tests we get very fast mutation coverage testing. See this asciicast for a (speed) demonstration.