Is Pex (Test generation) really useful tool?

Question

Yes, it is possible to generate tests on boundary values for functions like "Sum" or "Divide". Pex is a good tool here.

But more often we create tests on business behaviour. Let's consider example from classic Beck's tdd book:

[Test] public void ShouldRoundOnCreation() {   Money money = new Money(20.678);   Assert.AreEqual(20.68,money.Amount);   Assert.AreEqual(2068,money.Cents); } 

Can this test be generated? No :) 95 % of tests in my projects check business logic, and can not be generated.

Pex (Especially in pair with Moles) can give 100% code coverage, but a high code coverage rate of a test suite does never indicate, that code is well tested - It only gives false confidence that everything is tested. And this is very dangerous.

So, the question is - Is Pex really useful tool?

Answer 1

I think you are mistaken in the way Pex should be used: we highly recommend users to write assertions in their parameterized unit tests.

If you write assertions, Pex will try systematically to invalidate them - that's where the power of Pex comes in: the more you write assertions, the more it tries to find bugs for you.

If you use Pex to get a high code coverage test suite without writing assertions, you only get what you asked for: code coverage and guaranteed runtime exceptions. Pex 'only' tries to cover branches (1 assertion = 1 branch), if there are no branches to cover (no assertion), it won't generate interresting test cases.

In general, writing assertions in parameterized unit tests are harder to write because... well they are more general. Let's start with the rounding behavior: there is certainly a bound on which you allow rounding to occur, this should translate naturally into a parameterized unit test:

[PexMethod] public void RoundInBounds(double value)  {     var money = new Money(value);     // distance between value and amount should be at most 0.01/2     Assert.AreEqual(value, money.Amount, 0.005); } 

There are also a number of patterns that can be used to write those. For example, your 'Money' class is probably idempotent: if you feed the value of 'Amount' back in a Money instance, you get Amount back. This translate elegantly into a parameterized unit test:

[PexMethod] public void RoundIsIdempotent(double value)  {      var first = new Money(value).Amount;      var second = new Money(first).Amount;      Assert.AreEqual(first, second, 0.0001); } 

Note also, that parameterized unit tests definitely belong in the TDD world. Just write the parameterized unit test first, Pex will find the failing bug, fix the bug, Pex finds the next failing bug, etc...

Does this make Pex a useful tool? You be the judge.