What is the difference between Property Based Testing and Mutation testing?

Question

My context for this question is in Python.

Hypothesis Testing Library (i.e. Property Based Testing): https://hypothesis.readthedocs.io/en/latest/

Mutation Testing Library: https://github.com/sixty-north/cosmic-ray

Answer 1

These are very different beasts but both would improve the value and quality of your tests. Both tools contribute to and make the "My code coverage is N%" statement more meaningful.

---

Hypothesis would help you to generate all sorts of test inputs in the defined scope for a function under test.

Usually, when you need to test a function, you provide multiple example values trying to cover all the use cases and edge cases driven by the code coverage reports - this is so called "Example based testing". Hypothesis on the other hand implements a property-based testing generating a whole bunch of different inputs and input combinations helping to catch different common errors like division by zero, None, 0, off-by-one errors etc and helping to find hidden bugs.

Mutation testing is all about changing your code under test on the fly while executing your tests against a modified version of your code.

This really helps to see if your tests are actually testing what are they supposed to be testing, to understand the value of your tests. Mutation testing would really shine if you already have a rich test code base and a good code coverage.

---

What helped me to get ahold of these concepts were these Python Podcasts:

• Property-based Testing with Hypothesis
• Validating Python tests with mutation testing
• Hypothesis with David MacIver

Answer 2

I'm the author or mutmut, the (imo) best mutation tester for python. @alecxe has a very good answer but I would like to expand on it. Read his answer before mine for basic context.

There are some big other differences, like PBT requires mental work to specify the rules for each function under test while MT requires you to justify all behavior in the code which requires much less cognitive effort.

MT is effectively white box and PBT black box.

Another difference is that MT is the exploration of a (fairly small) finite space, while PBT is an exploration of an infinite space (practically speaking). A practical consequence is that you can trivially know when you are done with MT, while you can have a PBT run going for years and you can't know if it has searched the relevant parts of the space. Better rules for PBT radically cuts the run time for this reason.

Mutation testing also forces minimal code. This is a surprising effect, but it's something I have experienced again and again. This is a nice little bonus for MT.

You can also use MT as a simple checklist to get to 100% mutation coverage, you don't need to start with 100% coverage, not at all. But with PBT you can start way below 100% coverage, in essence at 0% before you start.

I hope this clarifies the situation a bit more.