how to set number of forked JVMs for integration tests?

Question

The following sets up all my integration tests to run in separate forked JVMs, which expands on How to fork the jvm for each test in sbt

fork in IntegrationTest := true,
testForkedParallel in IntegrationTest := true,
testGrouping in IntegrationTest <<= (
  definedTests in IntegrationTest,
  baseDirectory in IntegrationTest,
  javaOptions in IntegrationTest,
  outputStrategy in IntegrationTest,
  envVars in IntegrationTest,
  javaHome in IntegrationTest,
  connectInput in IntegrationTest
).map { (tests, base, options, strategy, env, javaHomeDir, connectIn) =>
  val opts = ForkOptions(
    bootJars = Nil,
    javaHome = javaHomeDir,
    connectInput = connectIn,
    outputStrategy = strategy,
    runJVMOptions = options,
    workingDirectory = Some(base),
    envVars = env
  )
  tests.map { test =>
    Tests.Group(test.name, Seq(test), Tests.SubProcess(opts))
  }
}

but it only runs one at a time. This is not unsurprising, because according to http://www.scala-sbt.org/0.13/docs/Testing.html Tags.ForkedTestGroup is 1 by default.

However, my attempts to customise the Tags.ForkedTestGroup - to enable more than one JVM at a time - are being ignored. This is what I've tried (each independently):

concurrentRestrictions in Global -= Tags.limit(Tags.ForkedTestGroup, 1)

concurrentRestrictions in Global += Tags.limit(Tags.ForkedTestGroup, 4)

concurrentRestrictions in Global := Seq(Tags.limitAll(4))

So how does one get more than one forked JVM to run tests?

Answer 1

Using the following snippet from your post in build.sbt

concurrentRestrictions in Global := Seq(Tags.limitAll(4))

seems to work almost as expected.

In an interactive sbt shell you can check the value of the global::concurrentRestrictions setting by entering this:

inspect global:concurrentRestrictions

On my machine without setting the limit explicitly this returns:

[info] Setting: scala.collection.Seq[sbt.Tags$Rule] = List(Limit all to 8, Limit forked-test-group to 1)
[info] Description:
[info]  Rules describing restrictions on concurrent task execution.
[info] Provided by:
[info]  */*:concurrentRestrictions
[info] Defined at:
[info]  (sbt.Defaults) Defaults.scala:145
[info] Delegates:
[info]  global:concurrentRestrictions
[info]  *:concurrentRestrictions
[info]  {.}/global:concurrentRestrictions
[info]  {.}/*:concurrentRestrictions
[info]  */global:concurrentRestrictions
[info]  */*:concurrentRestrictions
[info] Related:
[info]  */*:concurrentRestrictions

And after using the snippet you posted I see this output (I've only shown the top bit here - the rest is the same as before)

> inspect global:concurrentRestrictions
[info] Setting: scala.collection.Seq[sbt.Tags$Rule] = List(Limit all to 4)
...

So that certainly seems to be working - indeed running some simple tests with the configuration you have above indicates that a separate JVM is being used for each test class and each is run in parallel by default. Without that configuration a single JVM is used for all tests. I could get this to run tests sequentially by setting:

parallelExecution in Test := false

With this setting, the tests are run sequentially but each test class still uses it's own JVM if your previous configuration is included.

If I set:

concurrentRestrictions in Global := Seq(Tags.limitAll(2))

and then perform a test run with three test classes, the tests from two classes are run concurrently and then tests from the third class are run after one of the first two finishes. This seems to indicate that this setting is indeed controlling the number of concurrent forked tests although confusingly it also indicates that the default which matters in the initial defaults for concurrent forked JVMs is not the limit of 1 for "forked-test-group" but the limit of 8 for "all".