How do i test/refactor my tests?

Question

I have a test suit for my app. As the test suit grew organically, the tests have a lot of repeated code which can be refactored.

However I would like to ensure that the test suite doesn't change with the refactor. How can test that my tests are invariant with the refactor.

(I am using Python+UnitTest), but I guess the answer to this can be language agnostic.

Answer 1

The real test for the tests is the production code.

An effective way to check that a test code refactor hasn't broken your tests would be to do Mutation Testing, in which a copy of the code under test is mutated to introduce errors in order to verify that your tests catch the errors. This is a tactic used by some test coverage tools.

I haven't used it (and I'm not really a python coder), but this seems to be supported by the Python Mutant Tester, so that might be worth looking at.

Answer 2

Coverage.py is your friend.

Move over all the tests you want to refactor into "system tests" (or some such tag). Refactor the tests you want (you would be doing unit tests here right?) and monitor the coverage:

• After running your new unit tests but before running the system tests
• After running both the new unit tests and the system tests.

In an ideal case, the coverage would be same or higher but you can thrash your old system tests.

FWIW, py.test provides mechanism for easily tagging tests and running only the specific tests and is compatible with unittest2 tests.