Firebase Test Lab Coverage with Orchestrator Permission Denied

Question

We are currently attempting to use Google's Firebase Test Labs for testing our android app on devices. To test things, I've created a small app that doesn't do anything except get tested, and it's worked well so far, except that the recommended way (taken directly from the documentation page) to achieve code coverage, does not seem to work properly with the orchestrator.

With the following gcloud command:

gcloud firebase test android run --results-bucket=<hidden> --use-orchestrator --type instrumentation --device model=Pixel2,version=29 --device model=Pixel2,version=28 --app app/build/outputs/apk/debug/app-debug.apk  --test app/build/outputs/apk/androidTest/debug/app-debug-androidTest.apk --environment-variables clearPackageData=true,coverage=true,coverageFilePath=/sdcard/  --directories-to-pull=/sdcard/

the tests run successfully, but for every single test execution, the creation of the coverage execution data fails. The logcat entry looks like:

04-08 09:50:37.874: E/CoverageListener(8114): Failed to generate Emma/JaCoCo coverage. 
04-08 09:50:37.874: E/CoverageListener(8114): java.lang.reflect.InvocationTargetException
04-08 09:50:37.874: E/CoverageListener(8114):   at java.lang.reflect.Method.invoke(Native Method)
04-08 09:50:37.874: E/CoverageListener(8114):   at androidx.test.internal.runner.listener.CoverageListener.generateCoverageReport(CoverageListener.java:101)
04-08 09:50:37.874: E/CoverageListener(8114):   at androidx.test.internal.runner.listener.CoverageListener.instrumentationRunFinished(CoverageListener.java:70)
04-08 09:50:37.874: E/CoverageListener(8114):   at androidx.test.internal.runner.TestExecutor.reportRunEnded(TestExecutor.java:92)
04-08 09:50:37.874: E/CoverageListener(8114):   at androidx.test.internal.runner.TestExecutor.execute(TestExecutor.java:65)
04-08 09:50:37.874: E/CoverageListener(8114):   at androidx.test.runner.AndroidJUnitRunner.onStart(AndroidJUnitRunner.java:392)
04-08 09:50:37.874: E/CoverageListener(8114):   at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:2145)
04-08 09:50:37.874: E/CoverageListener(8114): Caused by: java.io.FileNotFoundException: /sdcard/com.example.citest.AnotherInstrumentedTest#testSdkVersion.ec (Permission denied)
04-08 09:50:37.874: E/CoverageListener(8114):   at java.io.FileOutputStream.open0(Native Method)
04-08 09:50:37.874: E/CoverageListener(8114):   at java.io.FileOutputStream.open(FileOutputStream.java:308)
04-08 09:50:37.874: E/CoverageListener(8114):   at java.io.FileOutputStream.<init>(FileOutputStream.java:238)
04-08 09:50:37.874: E/CoverageListener(8114):   at com.vladium.emma.rt.RT.dumpCoverageData(RT.java:50)
04-08 09:50:37.874: E/CoverageListener(8114):   ... 7 more

This indicated to me that the app does not have permission to write to external storage, but it still does not work after adding

<uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE"/>
<uses-permission android:name="android.permission.READ_EXTERNAL_STORAGE"/>

to both the app's manifest, as well as the additional manifest in the androidTest folder.

When changing the command to no longer use the orchestrator, and instead use the single coverage file as indicated in the documentation:

gcloud firebase test android run --results-bucket=<hidden> --no-use-orchestrator --type instrumentation --device model=Pixel2,version=29 --device model=Pixel2,version=28 --app app/build/outputs/apk/debug/app-debug.apk  --test app/build/outputs/apk/androidTest/debug/app-debug-androidTest.apk --environment-variables coverage=true,coverageFile=/sdcard/coverage.ec  --directories-to-pull=/sdcard/

The coverage file for the device using version 28 is successfully created, but for the device using version 29, it still fails with

04-08 05:07:43.341: E/CoverageListener(9398): Failed to generate Emma/JaCoCo coverage. 
04-08 05:07:43.341: E/CoverageListener(9398): java.lang.reflect.InvocationTargetException
04-08 05:07:43.341: E/CoverageListener(9398):   at java.lang.reflect.Method.invoke(Native Method)
04-08 05:07:43.341: E/CoverageListener(9398):   at androidx.test.internal.runner.listener.CoverageListener.generateCoverageReport(CoverageListener.java:101)
04-08 05:07:43.341: E/CoverageListener(9398):   at androidx.test.internal.runner.listener.CoverageListener.instrumentationRunFinished(CoverageListener.java:70)
04-08 05:07:43.341: E/CoverageListener(9398):   at androidx.test.internal.runner.TestExecutor.reportRunEnded(TestExecutor.java:92)
04-08 05:07:43.341: E/CoverageListener(9398):   at androidx.test.internal.runner.TestExecutor.execute(TestExecutor.java:65)
04-08 05:07:43.341: E/CoverageListener(9398):   at androidx.test.runner.AndroidJUnitRunner.onStart(AndroidJUnitRunner.java:392)
04-08 05:07:43.341: E/CoverageListener(9398):   at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:2189)
04-08 05:07:43.341: E/CoverageListener(9398): Caused by: java.io.FileNotFoundException: /sdcard/coverage.ec: open failed: EACCES (Permission denied)
04-08 05:07:43.341: E/CoverageListener(9398):   at libcore.io.IoBridge.open(IoBridge.java:496)
04-08 05:07:43.341: E/CoverageListener(9398):   at java.io.FileOutputStream.<init>(FileOutputStream.java:235)
04-08 05:07:43.341: E/CoverageListener(9398):   at com.vladium.emma.rt.RT.dumpCoverageData(RT.java:50)
04-08 05:07:43.341: E/CoverageListener(9398):   ... 7 more
04-08 05:07:43.341: E/CoverageListener(9398): Caused by: android.system.ErrnoException: open failed: EACCES (Permission denied)
04-08 05:07:43.341: E/CoverageListener(9398):   at libcore.io.Linux.open(Native Method)
04-08 05:07:43.341: E/CoverageListener(9398):   at libcore.io.ForwardingOs.open(ForwardingOs.java:167)
04-08 05:07:43.341: E/CoverageListener(9398):   at libcore.io.BlockGuardOs.open(BlockGuardOs.java:252)
04-08 05:07:43.341: E/CoverageListener(9398):   at libcore.io.ForwardingOs.open(ForwardingOs.java:167)
04-08 05:07:43.341: E/CoverageListener(9398):   at android.app.ActivityThread$AndroidOs.open(ActivityThread.java:7255)
04-08 05:07:43.341: E/CoverageListener(9398):   at libcore.io.IoBridge.open(IoBridge.java:482)
04-08 05:07:43.341: E/CoverageListener(9398):   ... 9 more

In all cases, simply running the android device tests on a local emulator via gradle cAT, produces valid coverage files, and the problems only occur when running the tests via the Firebase Test labs.

Answer 1

The coverage file for the device using version 28 is successfully created, but for the device using version 29, it still fails with

So, I was running into this same issue, and it was this comment that tipped me off to the solution.

Basically in Android 10+ they have changed the way that apps interact with external devices to a "scoped" approach. More details here: https://developer.android.com/training/data-storage

As far as I can tell whichever library is trying to write the coverage.ec to the sdcard, is not doing it correctly as per this "scoped" approach.

So by adding this to your AndroidManifest.xml:

<application
  android:requestLegacyExternalStorage="true">
</application>

It will tell Android to use the old non-"scoped" method of writing to the SD card, and then I found this to work correctly.

I think this is only temporary though, and this might not work with Android 11. But some library needs to update to change how it writes to the /sdcard.

Answer 2

I've answered the same question here:

https://stackoverflow.com/a/65034774/4142087

tl;dr

Add

@get:Rule
val runtimePermissionRule: GrantPermissionRule = GrantPermissionRule.grant(
    Manifest.permission.WRITE_EXTERNAL_STORAGE,     
    Manifest.permission.READ_EXTERNAL_STORAGE
)