SBT integration test setup

Question

I would like to add an Integration Test phase to my SBT + Spray app.

Ideally it would be just like Maven, with the following phases:

• compile: The app is built
• test: The unit tests are run
• pre-integration-test: The app is launched in a separate process
• integration-test: The integration tests are run; they issue requests to the app running in the background and verify that the correct results are returned
• post-integration-test: The instance of the app previously launched is shut down

I'm having a lot of trouble getting this to work. Is there a worked example that I can follow?

1) Separate "it" codebase:

I started by adding the code shown in the "Integration Test" section of the SBT docs to a new file at project/Build.scala.

This allowed me to add some integration tests under "src/it/scala" and to run them with "sbt it:test", but I can't see how to add a pre-integration-test hook.

The question "Ensure 're-start' task automatically runs before it:test" seems to address how to set up such a hook, but the answer doesn't work for me (see my comment on there).

Also, adding the above code to my build.scala has stopped the "sbt re-start" task from working at all: it tries to run the app in "it" mode, instead of in "default" mode.

2) Integration tests in "test" codebase:

I am using IntelliJ, and the separate "it" codebase has really confused it. It can't compile any of the code in that dir, as it thinks that all the dependencies are missing.

I tried to paste instead the code from "Additional test configurations with shared sources" from the SBT docs, but I get a compile error:

[error] E:\Work\myproject\project\Build.scala:14: not found: value testOptions
[error]         testOptions in Test := Seq(Tests.Filter(unitFilter)),

Is there a worked example I can follow?

I'm considering giving up on setting this up via SBT and instead adding a test flag to mark tests as "integration" and writing an external script to handle this.

Answer 1

I have now written my own code to do this. Issues that I encountered:

• I found that converting my build.sbt to a project/Build.scala file fixed most of the compile errors (and made compile errors in general much easier to fix, as IntelliJ could help much more easily).

• The nicest way I could find for launching the app in a background process was to use sbt-start-script and to call that script in a new process.

• Killing the background process was very difficult on Windows.

The relevant code from my app is posted below, as I think a few people have had this problem. If anyone writes an sbt plugin to do this "properly", I would love to hear of it.

Relevant code from project/Build.scala:

object MyApp extends Build {
  import Dependencies._

  lazy val project = Project("MyApp", file("."))

    // Functional test setup.
    // See http://www.scala-sbt.org/release/docs/Detailed-Topics/Testing#additional-test-configurations-with-shared-sources
    .configs(FunctionalTest)
    .settings(inConfig(FunctionalTest)(Defaults.testTasks) : _*)
    .settings(
      testOptions in Test := Seq(Tests.Filter(unitTestFilter)),
      testOptions in FunctionalTest := Seq(
        Tests.Filter(functionalTestFilter),
        Tests.Setup(FunctionalTestHelper.launchApp _),
        Tests.Cleanup(FunctionalTestHelper.shutdownApp _)),

      // We ask SBT to run 'startScriptForJar' before the functional tests,
      // since the app is run in the background using that script
      test in FunctionalTest <<= (test in FunctionalTest).dependsOn(startScriptForJar in Compile)
    )
    // (other irrelvant ".settings" calls omitted here...)


  lazy val FunctionalTest = config("functional") extend(Test)

  def functionalTestFilter(name: String): Boolean = name endsWith "FuncSpec"
  def unitTestFilter(name: String): Boolean = !functionalTestFilter(name)
}

This helper code is in project/FunctionTestHelper.scala:

import java.net.URL
import scala.concurrent.{TimeoutException, Future}
import scala.concurrent.ExecutionContext.Implicits.global
import scala.concurrent.duration._
import scala.sys.process._

/**
 * Utility methods to help with the FunctionalTest phase of the build
 */
object FunctionalTestHelper {

  /**
   * The local port on which the test app should be hosted.
   */
  val port = "8070"
  val appUrl = new URL("http://localhost:" + port)

  var processAndExitVal: (Process, Future[Int]) = null

  /**
   * Unfortunately a few things here behave differently on Windows
   */
  val isWindows = System.getProperty("os.name").startsWith("Windows")

  /**
   * Starts the app in a background process and waits for it to boot up
   */
  def launchApp(): Unit = {

    if (canConnectTo(appUrl)) {
      throw new IllegalStateException(
        "There is already a service running at " + appUrl)
    }

    val appJavaOpts =
      s"-Dspray.can.server.port=$port " +
      s"-Dmyapp.integrationTests.itMode=true " +
      s"-Dmyapp.externalServiceRootUrl=http://localhost:$port"
    val javaOptsName = if (isWindows) "JOPTS" else "JAVA_OPTS"
    val startFile = if (isWindows) "start.bat" else "start"

    // Launch the app, wait for it to come online
    val process: Process = Process(
      "./target/" + startFile,
      None,
      javaOptsName -> appJavaOpts)
        .run()
    processAndExitVal = (process, Future(process.exitValue()))

    // We add the port on which we launched the app to the System properties
    // for the current process.
    // The functional tests about to run in this process will notice this
    // when they load their config just before they try to connect to the app.
    System.setProperty("myapp.integrationTests.appPort", port)

    // poll until either the app has exited early or we can connect to the
    // app, or timeout
    waitUntilTrue(20.seconds) {
      if (processAndExitVal._2.isCompleted) {
        throw new IllegalStateException("The functional test target app has exited.")
      }
      canConnectTo(appUrl)
    }
  }

  /**
   * Forcibly terminates the process started in 'launchApp'
   */
  def shutdownApp(): Unit = {
    println("Closing the functional test target app")
    if (isWindows)
      shutdownAppOnWindows()
    else
      processAndExitVal._1.destroy()
  }

  /**
   * Java processes on Windows do not respond properly to
   * "destroy()", perhaps because they do not listen to WM_CLOSE messages
   *
   * Also there is no easy way to obtain their PID:
   * http://stackoverflow.com/questions/4750470/how-to-get-pid-of-process-ive-just-started-within-java-program
   * http://stackoverflow.com/questions/801609/java-processbuilder-process-destroy-not-killing-child-processes-in-winxp
   *
   * http://support.microsoft.com/kb/178893
   * http://stackoverflow.com/questions/14952948/kill-jvm-not-forcibly-from-command-line-in-windows-7
   */
  private def shutdownAppOnWindows(): Unit = {
    // Find the PID of the server process via netstat
    val netstat = "netstat -ano".!!

    val m = s"(?m)^  TCP    127.0.0.1:${port}.* (\\d+)$$".r.findFirstMatchIn(netstat)

    if (m.isEmpty) {
      println("FunctionalTestHelper: Unable to shut down app -- perhaps it did not start?")
    } else {
      val pid = m.get.group(1).toInt
      s"taskkill /f /pid $pid".!
    }
  }

  /**
   * True if a connection could be made to the given URL
   */
  def canConnectTo(url: URL): Boolean = {
    try {
      url.openConnection()
        .getInputStream()
        .close()
      true
    } catch {
      case _:Exception => false
    }
  }

  /**
   * Polls the given action until it returns true, or throws a TimeoutException
   * if it does not do so within 'timeout'
   */
  def waitUntilTrue(timeout: Duration)(action: => Boolean): Unit = {
    val startTimeMillis = System.currentTimeMillis()
    while (!action) {
      if ((System.currentTimeMillis() - startTimeMillis).millis > timeout) {
        throw new TimeoutException()
      }
    }
  }
}