Espresso test passes individually, fails when run in the suite

Question

I have the following Espresso test. It always passes if I run it by itself, but always fails when I run all the tests in the class together.

What's also a bit strange is that it used to work even as part of the suite. I'm not sure why now it stopped working. It must be something I've done but I don't know what.

@Test
public void itemHasImage_ShowsImage() {
    closeSoftKeyboard();
    if (mItem.getImageUrl() != null) {
        onView(withId(R.id.edit_item_image)).perform(scrollTo())
            .check(matches(isDisplayed()));
    }
}

The error I'm getting is:

Error performing 'scroll to'...
...
Caused by: java.lang.RuntimeException: Action will not be performed
because the target view does not match one or more of the following
constraints:(view has effective visibility=VISIBLE and is descendant
of a: (is assignable from class: class android.widget.ScrollView...

The view is visible and a descendant of a scroll view, as evidenced by it passing when run on it's own.

When it gets to this test (in the suite) it just doesn't scroll. But when I run it by itself it scrolls just fine.

In another stack overflow question I asked recently Espresso not starting Activity the same for second iteration in parameterised test, I found out that onDestroy from the previous test was getting called after onResume in the current test, causing it to set a value to null and fail the test. Again in that situation, the problem was that the test passed by itself but not in the suite. I now have a similar problem but no obvious way to fix it. (Edit: the workaround for the other question can no longer be applied).

Any ideas anyone? Could it be reading the wrong Activity somehow? Like maybe it's looking at the one from the previous test. That sounds ridiculous but after that last problem I had it seems possible.

Edit: Ok it turns out that when running this test as part of the suite, the image is in fact not visible causing the test to fail. I found this using the debugger and scrolling the view manually. But why?

---

I think it's a bug and have logged an issue here:

https://code.google.com/p/android/issues/detail?id=235247

Answer 1

It can able to solve using Orchestrator android testing utility library, It will executing each test case independently, so there is no chance of breaking in test suite

• When using AndroidJUnitRunner version 1.0 or higher, you have access to a tool called Android Test Orchestrator, which allows you to run each of your app's tests within its own invocation of Instrumentation.

Enabling using build.gradle

    android {
  defaultConfig {
   ...
   testInstrumentationRunner "android.support.test.runner.AndroidJUnitRunner"
 }

  testOptions {
    execution 'ANDROID_TEST_ORCHESTRATOR'
  }
}

dependencies {
  androidTestImplementation 'com.android.support.test:runner:1.0.1'
  androidTestUtil 'com.android.support.test:orchestrator:1.0.1'
}

For more info:https://developer.android.com/training/testing/junit-runner.html#using-android-test-orchestrator

Answer 2

I had a similar issue but the cause was different, @Rakshith-Kumar suggestion of using Orchestrator might work, also putting each test method in a TestSuite could also be a solution, since the flakiness could be addressed by running each test individually.

Also want to mention that if you're testing component that's using RxJava schedulers (io scheduler for example) to run some tasks in a background thread, the flakiness could happen because Espresso will not be able to synchronize with the background threads.

A solution can be injecting a the ObservableTransformer to do the subscribeOn and observeOn operations. And for Espresso tests substitute the ObservableTransformer instance with another one that uses Schedulers.from(AsyncTask.THREAD_POOL_EXECUTOR) for the io scheduler. Something like that:

public <T> ObservableTransformer<T, T> ioTransformer() {
        return observable -> observable.subscribeOn(AsyncTask.THREAD_POOL_EXECUTOR).observeOn(AndroidSchedulers.mainThread());
    }

This way Espresso can synchronize with background threads and avoid flakiness. More info: https://blog.danlew.net/2015/03/02/dont-break-the-chain/ https://developer.android.com/training/testing/espresso/#sync