Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Why should I really use Property-based testing if I already practice Example-based testing?

One caveat that some developers argue about TDD with Example-based tests is the possible lack of every valid input handling.

Let's take a simple example where those developers could argue:

Make a program that add two numbers.

Bob, a developer starts writing the first test:
Given 1 + 3, then result is 4.
Implementation: def add(x: Int, y: Int) = 4
Nice, it passes.

Now second test:
Given 2 + 2, then result is 4. Implementation still the same: def add(x: Int, y: Int) = 4

So other developers would come and tell him:

"Now you see Bob that Example-based testing is not enough with your poor 2 examples !
You didn't test every output and looking at your actual implementation, it will fail for other inputs whose result is distinct from 4!
Now, listen to us and start writing some property-based tests to cover all valid inputs.
So let's start with commutativity test, associativity etc.. that are specific to the addition: properties of the addition !"

But but but.... Bob's TDD practice is really bad!
Indeed, he wanted to apply triangulation but he wrote a test that already succeeded, since the implementation hadn't to be altered!

To lead to Triangulation, one should write a test that fails. And triangulation is one of the master key of the practice of TDD.
It allows the main step: refactoring since you should manage two paths that lead to 2 distinct results!

=> As long as the tests get specific, the code gets more generic thanks to refactoring.

So my question is:
If we practice strict TDD with good practice of triangulation, what is the real benefit of using Property-based testing, asserting invariants that are in 99% of cases already cover by a good TDD?
Indeed, supposing that developers have a good IQ and build implementation that makes sense ;)

My example is taken from those slides.

like image 524
Mik378 Avatar asked Jun 22 '15 18:06

Mik378


People also ask

How is property based testing PBT different from normal unit testing?

Generally speaking, property-based tests require only a few lines of code (like unit tests), but unlike unit tests they test a different set of inputs each time. Because of this, you end up covering more domain space with roughly the same amount of test code.

What is example based testing?

Traditional, or example-based, testing specifies the behavior of your software by writing examples of it—each test sets up a single concrete scenario and asserts how the software should behave in that scenario.

What are property based tests?

What is property-based testing? Property-based tests are designed to test the aspects of a property that should always be true. They allow for a range of inputs to be programmed and tested within a single test, rather than having to write a different test for every value that you want to test.

What is property based testing in Python?

Property based testing is considered as generative testing, we don't supply specific examples with inputs and expected outputs. Rather we define certain properties and generate randomized inputs to ensure the properties are correct.


2 Answers

property based testing is very helpful when edge cases are hard to find or there is so many of them that programmer can easily miss one. i used it for example when i was implementing hirschberg's algorithm. there is no obvious way to divide algorithm into smaller, trivial, easily TDD-testable pieces. and it's hard to hand-craft input that cover all possible algorithm paths. the best way is to generate large number of different input to cover all the paths. and when automatic checking find the bug, add that specific input to your regression tests

like image 166
piotrek Avatar answered Oct 18 '22 22:10

piotrek


If you practice TDD, you know it is a way of thinking about and doing design rather than than being about testing.

TDD originated as incremental, mostly, state based unit testing. Then Interaction-style (ie., mockist-style, or London-style) TDD changed how we think about code design code.

Property-based TDD has the opportunity to change the way we design code as well. Instead of generating examples to push our design forward, we use properties. This results in fewer tests that are more maintainable with better test coverage.

On the other hand, it is harder and requires more thinking than example based TDD.

like image 39
zhon Avatar answered Oct 18 '22 21:10

zhon