How SBT test task manages class path and how to correctly start a Java process from SBT test

Question

In one of my Scala tests, using ProcessBuilder, I fire up 3 Apache Spark streaming applications in separate JVMs. (Two or more Spark streaming applications can not co-exist in the same JVM.) One Spark application processes data and ingests into Apache Kafka, which the other ones read. Moreover the test involves writing into a NoSQL database.

While using ProcessBuilder, the Spark application's class path is set using:

val classPath = System.getProperty("java.class.path")

Running the test in IntelliJ works as expected, but on a CI system, the test is invoked by SBT's test task. The java.class.path in the latter case, will be solely the sbt.jar, so the child JVM exits with NoClassFoundException, again, as expected. :-)

I'm looking for a way to "span" JVMs from SBT tests using the same class path that the tests are actually using. For example if the test is invoked in project core, the class path of project core should be supplied to the child JVM, where the Spark application starts. Unfortunately I have got no idea how to retrieve the correct class path in SBT tasks - which then could be supplied to child JVMs.

Answer 1

Tests.Setup can be used to access the classpath within SBT:

testOptions in Test += Tests.Setup { classLoader =>
   // give Spark classpath via classLoader
}

For example, on my machine Tests.Setup(classLoader => println(classLoader)) gives

> test
ClasspathFilter(
  parent = URLClassLoader with NativeCopyLoader with RawResources(
  urls = List(/home/mario/sandbox/sbt/so-classpath/target/scala-2.12/test-classes, /home/mario/sandbox/sbt/so-classpath/target/scala-2.12/classes, /home/mario/.ivy2/cache/org.scala-lang/scala-library/jars/scala-library-2.12.4.jar, /home/mario/.ivy2/cache/org.scalatest/scalatest_2.12/bundles/scalatest_2.12-3.0.5.jar, /home/mario/.ivy2/cache/org.scalactic/scalactic_2.12/bundles/scalactic_2.12-3.0.5.jar, /home/mario/.ivy2/cache/org.scala-lang/scala-reflect/jars/scala-reflect-2.12.4.jar, /home/mario/.ivy2/cache/org.scala-lang.modules/scala-xml_2.12/bundles/scala-xml_2.12-1.0.6.jar),
  parent = DualLoader(a = java.net.URLClassLoader@3fcb37f1, b = java.net.URLClassLoader@271053e1),
  resourceMap = Set(app.class.path, boot.class.path),
  nativeTemp = /tmp/sbt_741bc913/sbt_c770779a
)
  root = sun.misc.Launcher$AppClassLoader@33909752
  cp = Set(/home/mario/.ivy2/cache/jline/jline/jars/jline-2.14.5.jar, /home/mario/.ivy2/cache/org.scala-lang/scala-reflect/jars/scala-reflect-2.12.4.jar, /home/mario/.ivy2/cache/org.scala-lang/scala-compiler/jars/scala-compiler-2.12.4.jar, /home/mario/.ivy2/cache/org.scala-lang.modules/scala-xml_2.12/bundles/scala-xml_2.12-1.0.6.jar, /home/mario/sandbox/sbt/so-classpath/target/scala-2.12/classes, /home/mario/.ivy2/cache/org.scalatest/scalatest_2.12/bundles/scalatest_2.12-3.0.5.jar, /home/mario/.sbt/boot/scala-2.10.7/org.scala-sbt/sbt/0.13.17/test-interface-1.0.jar, /home/mario/.ivy2/cache/org.scalactic/scalactic_2.12/bundles/scalactic_2.12-3.0.5.jar, /home/mario/sandbox/sbt/so-classpath/target/scala-2.12/test-classes, /home/mario/.ivy2/cache/org.scala-lang/scala-library/jars/scala-library-2.12.4.jar)
)

where we see that

.../target/scala-2.12/test-classes
.../target/scala-2.12/classes

are present.

On the other hand, to retrieve the classpath from within the test itself:

val classLoader = this.getClass.getClassLoader
// give Spark classpath via classLoader

For example, on my machine, println(classLoader) given in the following test

class CubeCalculatorTest extends FunSuite {
  test("CubeCalculator.cube") {
    val classLoader = this.getClass.getClassLoader
    println(classLoader)
    assert(CubeCalculator.cube(3) === 27)
  }
}

prints

URLClassLoader with NativeCopyLoader with RawResources(
  urls = List(/home/mario/sandbox/sbt/so-classpath/target/scala-2.12/test-classes, /home/mario/sandbox/sbt/so-classpath/target/scala-2.12/classes, /home/mario/.ivy2/cache/org.scala-lang/scala-library/jars/scala-library-2.12.4.jar, /home/mario/.ivy2/cache/org.scalatest/scalatest_2.12/bundles/scalatest_2.12-3.0.5.jar, /home/mario/.ivy2/cache/org.scalactic/scalactic_2.12/bundles/scalactic_2.12-3.0.5.jar, /home/mario/.ivy2/cache/org.scala-lang/scala-reflect/jars/scala-reflect-2.12.4.jar, /home/mario/.ivy2/cache/org.scala-lang.modules/scala-xml_2.12/bundles/scala-xml_2.12-1.0.6.jar),
  parent = DualLoader(a = java.net.URLClassLoader@6307eb76, b = java.net.URLClassLoader@271053e1),
  resourceMap = Set(app.class.path, boot.class.path),
  nativeTemp = /tmp/sbt_fa64d1a1/sbt_66bd50e2
)

where again we can see

.../target/scala-2.12/test-classes
.../target/scala-2.12/classes

are present.

To actually pass the classpath to ProcessBuilder within a test:

import java.net.URLClassLoader
import sys.process._

class CubeCalculatorTest extends FunSuite {
  test("CubeCalculator.cube") {
    val classLoader = this.getClass.getClassLoader
    val classpath = classLoader.asInstanceOf[URLClassLoader].getURLs.map(_.getFile).mkString(":")
    s"java -classpath $classpath MyExternalApp".!
    ...
  }
} 

Answer 2

You can get the full classpath, to be used in the ProcessBuilder's JVM, from your sbt tests, with:

Thread
  .currentThread
  .getContextClassLoader
  .getParent
  .asInstanceOf[java.net.URLClassLoader]
  .getURLs
  .map(_.getFile)
  .mkString(System.getProperty("path.separator"))