How can I use @FlakyTest annotation now?

Question

I try to run flaky test with espresso framework (and with Junit4) on Android Studio.

I want to set how many times it should to repeat. Before I can use

@FlakyTest(tolerance=5)

// (5 is number for repeat, for example)

But this annotation was deprecated in API level 24. - (link on android.developers.com)

Now is availible new @FlakyTest annotation - without tolerance variable. (link on android.developers.com)

I need to set how many times test can be repeated, but don't know how to do it. Any idea?

Answer 1

This annotation has been deprecated because the entire testing framework was replaced by a new one. Thus, the annotation has been also deprecated in favor of a new one.

Unfortunately, comparing to the old annotation this one can not be used to re-run a failed test. That makes it less useful for a practical purpose.

However, you can still use it for something useful. As the documentation says when running tests you can filter out those that are flaky. To do so you need to adjust a build script:

android {
    defaultConfig {
        testInstrumentationRunnerArgument "notAnnotation", "android.support.test.filters.FlakyTest"
    }
}

More about options can be found here.

Answer 2

I found this online https://gist.github.com/abyx/897229#gistcomment-2851489

You create your own test rule

class RetryTestRule(val retryCount: Int = 3) : TestRule {

    private val TAG = RetryTestRule::class.java.simpleName

    override fun apply(base: Statement, description: Description): Statement {
        return statement(base, description)
    }

    private fun statement(base: Statement, description: Description): Statement {
        return object : Statement() {

            override fun evaluate() {
                Log.e(TAG, "Evaluating ${description.methodName}")

                var caughtThrowable: Throwable? = null

                for (i in 0 until retryCount) {
                    try {
                        base.evaluate()
                        return
                    } catch (t: Throwable) {
                        caughtThrowable = t
                        Log.e(TAG, description.methodName + ": run " + (i + 1) + " failed")
                    }
                }

                Log.e(TAG, description.methodName + ": giving up after " + retryCount + " failures")
                if (caughtThrowable != null)
                    throw caughtThrowable
            }
        }
    }
}

and then add it to your test, like this

@Rule
@JvmField
val mRetryTestRule = RetryTestRule()