When implementing property-based testing, when should I use an input generator over a precondition expression?

Question

When implementing property-based testing, when should I use an input generator over a precondition expression?

Are there performance considerations when selecting a particular option?

Internally, does one method inevitably use the other?

I would think that a precondition expression would take longer to execute in comparison to an input generator. Has anyone tested this?

Why would we need both?

Answer 1

When you use a precondition expression (such as FsCheck's ==> operator), you're essentially throwing away data. Even if this only happens in one out of a hundred cases, you'd still be throwing away 1 input set for a normal property (because the default number of executions is 100, in FsCheck).

Throwing away one out of 100 is probably not a big deal.

Sometimes, however, you'd be throwing away a lot more data. If, for example, you want only positive numbers, you could write a precondition like x > 0, but since FsCheck generates negative numbers as well, you'd be throwing away 50 % of all values, after they have been generated. That's likely to make your tests run slower (but as always, when it comes to performance considerations: measure).

FsCheck comes with built-in generators for positive numbers for that very reason, but sometimes, you need more fine-grained control of the range of possible input values, as in this example.

If doing the FizzBuzz kata, for example, you may write your test for the FizzBuzz case like this:

[<Property(MaxFail = 2000)>]
let ``FizzBuzz.transform returns FizzBuzz`` (number : int) =
    number % 15 = 0 ==> lazy
    let actual = FizzBuzz.transform number
    let expected = "FizzBuzz"
    expected = actual

Notice the use of the MaxFail property. The reason you need it is because that precondition throws away 14 out of 15 generated candidates. By default, FsCheck will attempt 1000 candidates before it gives up, but if you throw away 14 out 15 candidates, on average you'll have only 67 values that match the precondition. Since FsCheck's default goal is to execute a property 100 times, it gives up.

As the MaxFail property implies, you can tweak the defaults. With 2000 candidates, you should expect 133 precondition matches on average.

It doesn't feel particularly efficient, though, so you can, alternatively use a custom generator:

[<Property(QuietOnSuccess = true)>]
let ``FizzBuzz.transform returns FizzBuzz`` () =
    let fiveAndThrees =
        Arb.generate<int> |> Gen.map ((*) (3 * 5)) |> Arb.fromGen
    Prop.forAll fiveAndThrees <| fun number ->

        let actual = FizzBuzz.transform number

        let expected = "FizzBuzz"
        expected = actual

This uses an ad-hoc in-line Arbitrary. This is more efficient because no data is thrown away.

My inclination is to use preconditions if it'd only throw away the occasional unmatching input. In most cases, I prefer custom generators.