How can I re-try a property-based-test if the randomly-generated inputs are not useful?

Question

I am a n00b to unit testing. I have installed FsCheck.Nunit and NUnitTestAdapter from Nuget, and I'm trying to do property-based-testing, largely inspired by the inestimable Scott Wlaschin.

I am using the [<Property>] attribute, and I would like the ability to "skip" inputs that don't meet the test's requirements:

[<Property(MaxTest=10)>]
let ``Calling unzipTo with an invalid destination will yield a failure.`` badDest =
    if Directory.Exists(badDest)
    then // somehow skip to the next randomized input
    else // do the actual test

What's the simplest way to do this?

I would prefer an answer for FsCheck/NUnit if it exists, but I would also consider any other framework whose tests can be run in Visual Studio. (I thought I saw some framework where there was a simple function to do exactly this, but I can't figure out what it was.)

I have preferred FsCheck.NUnit so far because it can generate random inputs for F# types (discriminated unions, etc) without additional work.

Answer 1

You should be able to do something like this:

open FsCheck
open FsCheck.Xunit

[<Property(MaxTest=10)>]
let ``Calling unzipTo with an invalid destination will yield a failure.`` badDest =
    (not Directory.Exists(badDest)) ==> lazy
    // do the actual test

The first line is a boolean condition, and ==> is a custom operator defined by the FsCheck module. It'll only force evaluation of the lazy expression if the condition on the right-hand side evaluates to true.

Do consider, however, refactoring this test so that it doesn't depend on the file system. The file system is persistent, so that automatically creates a Persistent Fixture, which is a lot of trouble to manage; not impossible to deal with, but better avoided.

This example uses FsCheck.Xunit, but IIRC FsCheck.Nunit works the same way. You should seriously consider using FsCheck.Xunit instead of FsCheck.Nunit, though. The extensibility model of NUnit 2 is extraordinarily poor, which means that most Glue Libraries that attempt to extend NUnit come with lots of problems. This isn't a problem with FsCheck.Nunit, but with NUnit itself, but it'll manifest itself in lots of trouble for you.

Answer 2

FsCheck.Prop.discard() seems to do what I want -- when I run the test with logging, I can see that some attempts were discarded, but 10 runs were completed without being discarded.

The ==> operator works for running the tests with FsCheck.Quick or similar. However, that requires the lazy part to be in the format of 'Testable, where the tests I'm currently writing are just <inputs>->unit.