How to group tests using specs2?

Question

I'm used to JUnit, in JUnit it is possible to group several tests (usually related to a class) just by defining these tests in a single file (class) and annotating them with @Test. Then, to run several of these tests, a TestSuite is created with @Suite.SuiteClasses and so on.

In specs2 it is possible to group several tests at two different levels extending some Specification. For example:

  "Whatever" should {
    "do its job when possible" in {
      whatever(new Thing).work must beSome
    }
    "return none when not possible" in {
      whatever(null).work must beNone
    }
  }

We can group several Specification of this type in a single file, and each one of them can pack several checks, each check would be like a @Test, each group like a file in JUnit and then each Specification as a Suite in JUnit, except a Suite is splitted into several classes and a Specification is in a single class (i.e. file), which tends to produce huge files.

So the question is two fold:

• Where should I put the stuff as a matter of organization and readability: a Specification and the things that each class should do, i.e. the checks that it should pass.
• If the whole group of tests is split into several files, how can I create a Suite that groups them if possible in a hierarchical way, e.g. as Suites for ScalaTest.

BTW: I'm using Specs2 because I think it is the standard (was by default with the archetype, a (very reduced) small (and anecdotal) sample corroborates this [1, 2]), but I am considering to use ScalaTest. Judging by numbers (specs2, scalatest) that may be the best option to follow standards and customs of the Scala community. I mention this because an answer like "it's not possible, use ScalaTest" would be acceptable for these reasons.

Answer 1

In specs2 there is no concept of a hierarchical suite. A Specification is just a list of examples. Even when you group them with xxx should yyy, this just affects the way the examples are displayed in the console, with more or less indentation.

On the other hand there are ways to organize specifications with specs2:

• references
• tags

References

You can create a hierarchy of specifications by creating a top-level specification referencing other ones:

// code for specs2 3.x
class ParentSpec extends Specification { def is = s2"""
  These are all the important specifications about our domain
  ${"child1" ~ ChildSpec1}   
  ${"child2" ~ ChildSpec2}   
  """
}

The child specifications can reference other specifications and so on. What's different from JUnit (and possibly from ScalaTest) is that your reference graph doesn't need to be a tree. When you execute a Specification with the all argument

sbt> test-only ParentSpec -- all

then dependencies are followed from the ParentSpec so that the low-level dependencies are executed before the high-level ones. And any cycles are broken so that you won't be executing things infinitely (or get a StackOverflowError).

Tags

Tags are a very convenient way to classify things because a given "thing" doesn't need to belong to only one category. This was, at the time, one of the great improvements brought by TestNG. In specs2 you can tag single examples or whole specifications and then declare which examples you want to run based on the inclusion/exclusion of some tags. For example

class Spec1 extends mutable.Specification { section("functional")
   "simple test" >> ok

   tag("io")
   "a bit of IO" >> ok
}

class Spec2 extends mutable.Specification { section("functional")
   "another simple test" >> ok

   tag("io")
   "another bit of IO" >> ok
}

Then you can execute only the specifications with the functional tag but not with the examples having the io tag

sbt> test-only -- include functional exclude io

Organization

Using references and tags you can probably imagine several ways to slice and dice your test source:

• you can use references to create a main "taxonomy" of specifications
• you can use tags to create "cross-cutting" concerns like io, slow, database, scalacheck...

Note that you can also mix all of the above and have tags on your references, specifications with both examples and references, and so on.

The criteria for choosing a given structure are:

• the navigation around concepts in the code base
• the execution speed of different suites
• the necessity to re-run only some aspects of your tests after a change
• the infrastructure constraints (not everything can run in any environment)