How to generate null strings for FsCheck tests

Question

Using FsCheck, the F# version of the Haskell QuickCheck test library, to generate tests from C#, I found that the random string generator does not generate the null string.

using FsCheck.Fluent;
Spec.ForAny<string>(s => s != null).QuickCheck(); // always pass

Furthermore, there seems not to handle null strings by design, but I have not managed to pin it down from the documentation. For example, just picking between two strings, one of them null, won't work:

var strings = Any.ValueIn<string>(null, "non-null string");
Spec.For(strings, s => true).QuickCheck(); // throws null ref exception

And strings seem to be a special case, because it handles custom-made objects such as

class Thing {}

when mixed with null values:

var objects = Any.ValueIn(null, new Thing());
Spec.For(objects, s => true).QuickCheck(); // pass

Answer 1

I tried to dig a bit into this and it appears that you have discovered a bug in FsCheck.

It appears that the problem is in file Arbitrary.fs and is really only string-related. I had to replace this, where they call ToCharArray on the string

    static member String() = 
        { new Arbitrary<string>() with
            override x.Generator = Gen.map (fun chars -> new String(List.toArray chars)) generate
            override x.Shrinker s = s.ToCharArray() |> Array.toList |> shrink |> Seq.map (fun chars -> new String(List.toArray chars))
        }

with this

    static member String() = 
        { new Arbitrary<string>() with
            override x.Generator = Gen.map (fun chars -> new String(List.toArray chars)) generate
            override x.Shrinker s = 
                match s with
                    | null  -> seq {yield null;}
                    | _ -> s.ToCharArray() |> Array.toList |> shrink |> Seq.map (fun chars -> new String(List.toArray chars))
        }

You may want to raise this with fscheck developers here and also check if my fix works well - there is probably a better way to implement it, but it would be simpler for someone, who already knows the code.

Answer 2

For FsCheck 1.x, I found a solution which involves modifying the default random string generator:

public class MyArbitraries
{
    public static Arbitrary<string> String()
    {
        var nulls = Any.Value<string>(null);
        var nonnulls = Arb.Default.String().Generator;
        return Any.GeneratorIn(nulls, nonnulls).ToArbitrary;
    }
}

and then initialise it with:

DefaultArbitraries.Add<MyArbitraries>();

Then the test in the question fails as intended:

Spec.ForAny<string>(s => s != null).QuickCheck() // now fails, which is good

This will generate around 50% nulls and 50% random strings, the weights can be adjusted:

Spec.ForAny<string>(s => true)
    .Classify(s => s==null, "null")
    .Classify(s => s!=null, "not null")
    .QuickCheck(); // displays percentages

However, effectively overriding the default string generator might not be a good idea if the decision of not including the null value by default was intentional and not a bug in the library. And, if it were a bug, it would distort the distributions when fixed.