How to set up SBT build to return zero exit code on test failure for Jenkins?

Question

When I am running my Specs2 tests in Jenkins via SBT, then the build is marked as a failure as soon as one test fails. Since Jenkins usually distinguishes between failure to build and test failures, I want to change this.

I know that the build failure in Jenkins is detected by the exit code of the call to SBT, which appears to return 1 as soon as at least one test fails.

What are the options I have assuming I want to avoid changing my build.sbt (or the project in general) just to fix this inconvenience?

Somehow I think it should be possible to put a standard sbt project into a standard Jenkins install and have it work as intended.

Answer 1

tl;dr Use testResultLogger with a custom test result logger that doesn't throw TestsFailedException that in turn sets the non-0 exit code.

Just noticed that I missed that requirement "to avoid changing build.sbt. You can use any other *.sbt file, say exitcodezero.sbt or ~/.sbt/0.13/default.sbt with the custom testResultLogger.

It turns out that since sbt 0.13.5 there's a way to have such behaviour - see Added setting 'testResultLogger' which allows customisation of test reporting where testResultLogger was born.

> help testResultLogger
Logs results after a test task completes.

As it may have been read in the implementation of TestResultLogger.SilentWhenNoTests that's the default value of testResultLogger:

results.overall match {
  case TestResult.Error | TestResult.Failed => throw new TestsFailedException
  case TestResult.Passed                    =>
}

It means that when there's an issue executing tests, TestsFailedException exception is thrown that's in turn caught to report it as follows:

[error] Failed: Total 3, Failed 1, Errors 0, Passed 2
[error] Failed tests:
[error]         HelloWorldSpec
[error] (test:test) sbt.TestsFailedException: Tests unsuccessful

My idea is to disable throwing the exception regardless of the outcome of executing tests. Add the following to build.sbt and have the exit code always 0:

testResultLogger in (Test, test) := new TestResultLogger {
    import sbt.Tests._
    def run(log: Logger, results: Output, taskName: String): Unit = {
        println("Exit code always 0...as you wish")
        // uncomment to have the default behaviour back
        // TestResultLogger.SilentWhenNoTests.run(log, results, taskName)
    }
}

Uncomment TestResultLogger.SilentWhenNoTests.run to have the default behaviour back.

➜  failing-tests-dont-break-build  xsbt test; echo $?
JAVA_HOME=/Library/Java/JavaVirtualMachines/java8/Contents/Home
SBT_OPTS= -Xms512M -Xmx1536M -Xss1M -XX:+CMSClassUnloadingEnabled -Dfile.encoding=UTF-8
[info] Loading global plugins from /Users/jacek/.sbt/0.13/plugins
[info] Set current project to failing-tests-dont-break-build (in build file:/Users/jacek/sandbox/failing-tests-dont-break-build/)
[info] HelloWorldSpec
[info]
[info] The 'Hello world' string should
[info] x contain 11 characters
[error]    'Hello world' doesn't have size 12 but size 11 (HelloWorldSpec.scala:7)
[info]
[info] + start with 'Hello'
[info] + end with 'world'
[info]
[info] Total for specification HelloWorldSpec
[info] Finished in 15 ms
[info] 3 examples, 1 failure, 0 error
Exit code always 0...as you wish
[success] Total time: 1 s, completed Sep 19, 2014 9:58:09 PM
0