Group ScalaTest tests and run in order

Question

I would like to partition my tests suites and run them in the way that tests from one partition are not interleaved with tests from some other partition. Is it a way to do this in SBT, ScalaTest?

Answer 1

There are 2 levels of parallelism available for tests - all test suits that SBT finds can be run in parallel or sequentially. You can control it with this SBT setting:

parallelExecution in Test := false

Default value is true.

Additionally each test suit can execute its test cases in parallel or sequentially. You can control that in your testing framework. For instance in Specs2 you can add sequential in your test class:

class MyTest extends Specification {

  sequential

  // these test cases will be run sequentially based on the setting in the class
  "test 1" should {
    "do something" in {
      ...
    }

    "do something else" in {
      ...
    }
  }
} 

I don't know if there is similar setting for ScalaTest, seems like it didn't exist before.

Sometimes it's also useful to filter which tests should be run sequentially based on their name for example in SBT. You can do it with something like this:

object EventHubBuild extends Build {

  lazy val root = Project(id = "root",
                          base = file("."))
                          .aggregate(common, rest, backup)

  /**
* All unit tests should run in parallel by default. This filter selects such tests
* and afterwards parallel execution settings are applied.
* Thus don't include word 'Integration' in unit test suite name.
*
* SBT command: test
*/
  def parFilter(name: String): Boolean = !(name contains "Integration")

 /**
* Integration tests should run sequentially because they take lots of resources,
* or shared use of resources can cause conflicts.
*
* SBT command: serial:test
**/
  def serialFilter(name: String): Boolean = (name contains "Integration")

  // config for serial execution of integration tests
  lazy val Serial = config("serial") extend(Test)

  lazy val rest = Project(id = "rest",
                          base = file("rest"))
                          .configs(Serial)
                          .settings(inConfig(Serial)(Defaults.testTasks) : _*)
                          .settings(
                            testOptions in Test := Seq(Tests.Filter(parFilter)),
                            testOptions in Serial := Seq(Tests.Filter(serialFilter))
                          )
                          .settings(parallelExecution in Serial := false : _*)
                          .settings(restSettings : _*) dependsOn(common % "test->test;compile->compile", backup)


  //// the rest of the build file ...
}

Copied from https://github.com/pgxcentre/eventhub/blob/master/project/Build.scala. This introduces a task test:serial in SBT.

More details from official docs: http://www.scala-sbt.org/0.13/docs/Parallel-Execution.html

Answer 2

Here is the easiest way that I've found to do it:

Create a new Scala class:

class TestController extends Suites (
  new TestSuiteToRunFirst,
  new TestSuiteToRunSecond,
  new AnotherTestSuite
)

The test suites (classes) will be run in the order in which you add them in the list above. From now on, when you add a new test class, you will add it in the TestController class declaration above.

One other thing that you need to do: Inside your individual test class definition, annotate each class with @DoNotDiscover. This will prevent them from being auto-discovered by the test runner, then run again a second time by TestController.

@DoNotDiscover
class TestSuiteToRunFirst extends FlatSpec {
...
}