Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

ScalaTest: Running a test 50 times

Is there a simple way to implement failure tolerance in ScalaTest? I'm looking to run the same test 50 times and give it a tolerable error margin, e.g. 10%.

In the above case, the test would only pass if 45 out of 50 tests were successful.

like image 494
flavian Avatar asked Dec 07 '22 02:12

flavian


2 Answers

The best way to do that is to override withFixture and rerun failed tests using whatever algorithm makes sense in your particular case. For inspiration, I'd suggest you look at the Retries trait in ScalaTest itself. The Scaladoc is here:

http://doc.scalatest.org/2.1.0/index.html#org.scalatest.Retries

The actual source code for Retries is here:

https://github.com/scalatest/scalatest/blob/master/src/main/scala/org/scalatest/Retries.scala

like image 114
Bill Venners Avatar answered Dec 08 '22 14:12

Bill Venners


Here is an addition to Bill Venners proposed solution. I needed to implement a few retries for flickering/unstable tests.


val retries = 4

override def withFixture(test: NoArgTest) = {
  if (isRetryable(test)) withFixture(test, retries) else super.withFixture(test)
}

def withFixture(test: NoArgTest, count: Int): Outcome = {
  val outcome = super.withFixture(test)
  outcome match {
    case Failed(_) | Canceled(_) => if (count == 1) super.withFixture(test) else withFixture(test, count - 1)
    case other => other
  }
}

Extended test class for retries (with Retries) and each test with taggedAs Retryable. Such tests when needed will be retried up to 4 times.

like image 22
Igor K Avatar answered Dec 08 '22 16:12

Igor K