How design a Specs2 database test, with interdependent tests?

Question

Is there some preferred way to design a Specs2 test, with lots of tests that depend on the results of previous tests?

Below, you'll find my current test suite. I don't like the vars inbetween the test fragments. They're "needed" though, since some tests generate ID numbers that subsequent tests reuses.

1. Should I perhaps store the ID numbers in a Specs2 Context instead, or create a separate Object that holds all mutable state? And place only test fragments in the specification object? Or is there some even better approach?

2. If a test fails, I'd like to cancel the remaining test at the same depth. Can I make the test fragments depend upon each other? (I know I can cancel remaining matchers in a single test fragment (by using mutable tests, or via orSkip), but what about cancelling whole fragments?)

.

object DatabaseSpec extends Specification {
  sequential

  "The Data Access Object" should {

    var someId = "" // These var:s feels error prone, is there a better way?

    "save an object" >> {
      someId = database.save(something)
      someId must_!= ""

      // I'd like to cancel the remaining tests, below, at this "depth",
      // if this test fragmen fails. Can I do that?
      // (That is, cancel "load one object", "list all objects", etc, below.)
    }

    "load one object" >> {
      anObject = database.load(someId)
      anObject.id must_== someId
    }

    "list all objects" >> {
      objs = database.listAll()
      objs.find(_.id == someId) must beSome
    }

    var anotherId = ""
    ...more tests that create another object, and
    ...use both `someId` and `anotherId`...

    var aThirdId = ""
    ...tests that use `someId`, `anotherId` and `aThirdId...
  }


  "The Data Access Object can also" >> {
    ...more tests...
  }

}

Answer 1

There are 2 parts to your question: using vars for storing intermediary state, and stopping examples when one is failing.

1 - Using vars

There are some alternatives to using vars when using a mutable specification.

You can use lazy vals representing the steps of your process:

object DatabaseSpec extends mutable.Specification { 
  sequential

  "The Data Access Object" should {

    lazy val id1    = database.save(Entity(1))
    lazy val loaded = database.load(id1)
    lazy val list   = database.list

    "save an object"   >> { id1 === 1 }
    "load one object"  >> { loaded.id === id1 }
    "list all objects" >> { list === Seq(Entity(id1)) }
  }

  object database {
    def save(e: Entity) = e.id
    def load(id: Int) = Entity(id)
    def list = Seq(Entity(1))
  }
  case class Entity(id: Int)
}

Since those values are lazy, they will only be called when the examples are executed.

If you're ready to change the structure of your current specification you can also use the latest 1.12.3-SNAPSHOT and group all those small expectations into one example:

"The Data Access Object provides a save/load/list api to the database" >> {

  lazy val id1    = database.save(Entity(1))
  lazy val loaded = database.load(id1)
  lazy val list   = database.list

  "an object can be saved"  ==> { id1 === 1 }
  "an object can be loaded" ==> { loaded.id === id1 }
  "the list of all objects can be retrieved" ==> {
    list === Seq(Entity(id1))
  }
}

If any of those expectations fail then the rest will not be executed and you will get a failure message like:

x The Data Access Object provides a save/load/list api to the database
  an object can not be saved because '1' is not equal to '2' (DatabaseSpec.scala:16)

Another possibility, which would require 2 small improvements, would be to use the Given/When/Then way of writing specifications but using "thrown" expectations inside Given and When steps. As you can see in the User Guide, the Given/When/Then steps extract data from strings and pass typed information to the next Given/When/Then:

import org.specs2._
import specification._
import matcher.ThrownExpectations

class DatabaseSpec extends Specification with ThrownExpectations { def is = 
  "The Data Access Object should"^
    "save an object"             ^ save^
    "load one object"            ^ load^
    "list all objects"           ^ list^
  end

  val save: Given[Int] = groupAs(".*") and { (s: String) =>
    database.save(Entity(1)) === 1
    1
  }

  val load: When[Int, Int] =  groupAs(".*") and { (id: Int) => (s: String) =>
    val e = database.load(id)
    e.id === 1
    e.id
  }

  val list: Then[Int] =  groupAs(".*") then { (id: Int) => (s: String) =>
    val es = database.list
    es must have size(1)
    es.head.id === id
  }
}

The improvements, which I'm going to do, are:

• catch failure exceptions to report them as failures and not errors
• remove the necessity to use groupAs(".*") and when there's nothing to extract from the string description.

In that case it should be enough to write:

val save: Given[Int] = groupAs(".*") and { (s: String) =>
  database.save(Entity(1)) === 1
  1
}

Another possibility would be to allow to directly write:

val save: Given[Int] = groupAs(".*") and { (s: String) =>
  database.save(Entity(1)) === 1
}

where a Given[T] object can be created from a String => MatchResult[T] because the MatchResult[T] object already contains a value of type T, that would become a "Given".

2 - Stop the execution after a failing example

Using the implicit WhenFail Around context is certainly the best way to do what you want (unless you go with the expectations descriptions as shown above the G/W/T example).

Note on step(stepOnFail = true)

The step(stepOnFail = true) works by interrupting the following examples if one example in the previous block of concurrent examples failed. However, when you're using sequential, that previous block is limited to just one example. Hence what you're seeing. Actually I think that this is a bug and that all the remaining examples should not be executed, whether you're using sequential or not. So stay tuned for a fix coming up this week-end.

Answer 2

(Concerning question 1: I don't know if there's some better alternative to the vars inside the examples. Perhaps my examples are simply too long, and perhaps I should split my Spec:s into many smaller specs.)

Concerning question 2, I found in this email by etorreborre that stopping subsequent tests can be done like so:

"ex1" >> ok
"ex2" >> ok
"ex3" >> ko
 step(stopOnFail=true)

"ex4" >> ok

(Ex4 will be skipped if ex1, ex2 or ex3 fails. (This doesn't work as expected in Specs2 < 1.12.3 if you're using a sequential spec, however.))

---

Here's another way: According to this Specs2 Googl groups email by etorreborre one can have subsequent tests stop on failure, like so: ("example2" would be skipped, but "example3" and "4" would run)

class TestSpec extends SuperSpecification {

    sequential

    "system1" >> {
      implicit val stop = WhenFail()
      "example1" >> ko
      "example2" >> ok
    }
    "system2" >> {
      implicit val stop = WhenFail()
      "example3" >> ok
      "example4" >> ok
    }
}

case class WhenFail() extends Around {
  private var mustStop = false

  def around[R <% Result](r: =>R) = {
    if (mustStop)          Skipped("one example failed")
    else if (!r.isSuccess) { mustStop = true; r }
    else                   r
  }
}

---

In this email by etorreborre there's a method to cancel subsequent specifications if an example fails, if you've include a list of specifications:

sequential ^ stopOnFail ^
"These are the selenium specifications"         ^
  include(childSpec1, childSpec2, childSpec3)

And you'd need to edit test options in build.sbt so the child specs aren't executed again indepentendly after they've been included. From the email:

 testOptions := Seq(Tests.Filter(s =>
  Seq("Spec", "Selenium").exists(s.endsWith(_)) &&
    ! s.endsWith("ChildSpec")))