Scalacheck won't properly report the failing case

Question

I've wrote the following spec

"An IP4 address" should "belong to just one class" in {
    val addrs = for {
        a <- Gen.choose(0, 255)
        b <- Gen.choose(0, 255)
        c <- Gen.choose(0, 255)
        d <- Gen.choose(0, 255)
    } yield s"$a.$b.$c.$d"

    forAll (addrs) { ip4s =>
        var c: Int = 0
        if (IP4_ClassA.unapply(ip4s).isDefined) c = c + 1
        if (IP4_ClassB.unapply(ip4s).isDefined) c = c + 1
        if (IP4_ClassC.unapply(ip4s).isDefined) c = c + 1
        if (IP4_ClassD.unapply(ip4s).isDefined) c = c + 1
        if (IP4_ClassE.unapply(ip4s).isDefined) c = c + 1
        c should be (1)
    }
}

That is very clear in its scope.

The test passes successfully but when I force it to fail (for example by commenting out one of the if statements) then ScalaCheck correctly reports the error but the message doesn't mention correctly the actual value used to evaluate the proposition. More specifically I get:

[info] An IP4 address
[info] - should belong to just one class *** FAILED ***
[info]   TestFailedException was thrown during property evaluation.
[info]     Message: 0 was not equal to 1
[info]     Location: (NetSpec.scala:105)
[info]     Occurred when passed generated values (
[info]       arg0 = "" // 4 shrinks
[info]     )

where you can see arg0 = "" // 4 shrinks doesn't show the value.

I've tried to add even a simple println statement to review the cases but the output appears to be trimmed. I get something like this

192.168.0.1
189.168.
189.
1

SOLUTION

import org.scalacheck.Prop.forAllNoShrink
import org.scalatest.prop.Checkers.check

"An IP4 address" should "belong to just one class" in {
  val addrs = for {
    a <- Gen.choose(0, 255)
    b <- Gen.choose(0, 255)
    c <- Gen.choose(0, 255)
    d <- Gen.choose(0, 255)
  } yield s"$a.$b.$c.$d"
  check {
    forAllNoShrink(addrs) { ip4s =>
      var c: Int = 0
      if (IP4.ClassA.unapply(ip4s).isDefined) c = c + 1
      if (IP4.ClassB.unapply(ip4s).isDefined) c = c + 1
      if (IP4.ClassC.unapply(ip4s).isDefined) c = c + 1
      if (IP4.ClassD.unapply(ip4s).isDefined) c = c + 1
      if (IP4.ClassE.unapply(ip4s).isDefined) c = c + 1
      c == (1)
    }
  }
}

Answer 1

This is caused by ScalaCheck's test case simplification feature. ScalaCheck just sees that your generator produces a string value. Whenever it finds a value that makes your property false, it tries to simplify that value. In your case, it simplifies it four times until it ends up with an empty string, that still makes your property false.

So this is expected, although confusing, behavior. But you can improve the situation in three different ways.

You can select another data structure to represent your IP addresses. This will make ScalaCheck able to simplify your test cases in a more intelligent way. For example, use the following generator:

val addrs = Gen.listOfN(4, Gen.choose(0,255))

Now ScalaCheck knows that your generator only produces lists of length 4, and that it only contains numbers between 0 and 255. The test case simplification process will take this into account and not create any values that couldn't have been produced by the generator from start. You can do the conversion to string inside your property instead.

A second method is to add a filter directly to your generator, which tells ScalaCheck how an IP address string should look like. This filter is used during test case simplification. Define a function that checks for valid strings and attach it to your existing generator this way:

def validIP(ip: String): Boolean = ...

val validAddrs = addrs.suchThat(validIP)

forAll(validAddrs) { ... }

The third method is to simply disable the test case simplification feature altogether by using forAllNoShrink instead of forAll:

Prop.forAllNoShrink(addrs) { ... }

I should also mention that the two first methods require ScalaCheck version >= 1.11.0 to function properly.

UPDATE:

The listOfN list length is actually not respected by the shrinker any more, due to https://github.com/rickynils/scalacheck/issues/89. Hopefully this can be fixed in a future version of ScalaCheck.