sbt: How to write a task that runs testQuick only if test fails?

Question

This question is from Martin Grotzke on Twitter.

He wants to write a task (or a command, I think) that could:

1. First run test and if it fails, follow up with testOnly
2. Aggregate across multiple sub-projects in a build

Effectively emulating Bash:

$ sbt test || sbt testQuick

The motivation is to run a failing test twice to work around flaky tests.

Answer 1

project/TestExtraShotPlugin.scala

If you want the quick answer you can use the following code to achieve running test twice behavior.

import sbt._
import Keys._

object TestExtraShotPlugin extends AutoPlugin {
  override def requires = plugins.JvmPlugin
  override def trigger = allRequirements
  object autoImport {
    val testNoFail = taskKey[Unit]("test but don't fail")
  }
  import autoImport._

  override def buildSettings = {
    addCommandAlias("testExtraShot", ";testNoFail;testQuick")
  }

  override def projectSettings = {
    Test / testNoFail := (Test / test).result.value
  }
}

If you want to know more, please read on.

Error handling

First, we need to stop test from halting the task execution. The error handling of tasks are described in Tasks page of the documentation. But quick gist is that you call .result.value. You can pattern match on that to get the values, but we don't really need it here.

I am using that to define an alternative test task called testNoFail.

Ad-hoc plugin

Next, we want testNoFail to work in a multi-project build. One way of injecting settings to all subproject is defining a triggered AutoPlugin in project/*.scala.

  override def requires = plugins.JvmPlugin
  override def trigger = allRequirements

Command composition

If you want to tell sbt to do something and then do something else, probably a natural way is to use commands. This is analogous to humans typing things into to the shell consecutively.

Commands can be composed using semicolon ;.

  override def buildSettings = {
    addCommandAlias("testExtraShot", ";testNoFail;testQuick")
  }

This is possible because we can safely run testQuick task regardless of the previous state of the test.

See also Sequencing How to series for other methods of sequencing things.

How to use this

Run:

> testExtraShot

inside the sbt shell. This will run testNoFail and then testQuick.

Appendix: Tackling the conditional continuation

Note that I've circumvented Martin's original specification of "in case of failed tests" by running testQuick regardless. It's possible to implement this, but it's a bit more advanced.

Monadic continuation

We can define a task that first runs normal test, then depending on the returned result value, change the continuation of the next task. In sbt, this type of monadic continuation can be achieved using a dynamic task.

We normally try to avoid this composition, since it prevents the task engine from parallelizing concurrent tasks, but it's handy when we need if-then-do-something.

project/TestExtraShotPlugin.scala alternative version

The following implements an ad-hoc plugin that defines a dynamic task:

import sbt._
import Keys._

object TestExtraShotPlugin extends AutoPlugin {
  override def requires = plugins.JvmPlugin
  override def trigger = allRequirements
  object autoImport {
    val testExtraShot = taskKey[Unit]("test then testQuick only on failure")
  }
  import autoImport._

  override def projectSettings = {
    testExtraShot := (Def.taskDyn {
      val t = (Test / test).result.value
      if (t.toEither.isLeft) (Test / testQuick).toTask("")
      else
        Def.task {
          val s = streams.value
          s.log.info("ok!")
        }
    }).value
  }
}