How to get code coverage reports from google Firebase for Android Espresso tests

Question

Based on this documentation - https://developer.android.com/studio/test/command-line.html#AMOptionsSyntax it is possible to get code coverage results back from Firebase lab. Some folks in #test-lab at firebase-community.slack.com are able to get it working but after few attempts I am still hitting a wall.

I am able to get a combined code coverage report of jacaco and emma by following this guide so there is nothing wrong with my local setup but problematic when trying to give arguments to gcloud cmd line to ask for coverage numbers and it talks about emma coverage.

Essentially, when I run this command locally

gcloud beta test android run \
  --type instrumentation \
  --app app/build/outputs/apk/*-debug-unaligned.apk \
  --test app/build/outputs/apk/*-debug-androidTest-unaligned.apk \
  --device-ids Nexus6\
  --os-version-ids 22 \
  --locales en \
  --orientations portrait \
  --environment-variables coverage=true,coverageFile="/sdcard/coverage.ec" \
--directories-to-pull=/sdcard

I am expecting the coverage report to be generated but I get this in the instruments.results file

INSTRUMENTATION_STATUS: numtests=1
INSTRUMENTATION_STATUS: stream=
com.godaddy.gdm.telephony.uitests.DialerTabTest:
INSTRUMENTATION_STATUS: id=AndroidJUnitRunner
INSTRUMENTATION_STATUS: test=dialerTabNumberFormattingTest
INSTRUMENTATION_STATUS: current=1
INSTRUMENTATION_STATUS_CODE: 1
INSTRUMENTATION_STATUS: numtests=1
INSTRUMENTATION_STATUS: stream=.
INSTRUMENTATION_STATUS: id=AndroidJUnitRunner
INSTRUMENTATION_STATUS: test=dialerTabNumberFormattingTest
INSTRUMENTATION_STATUS: current=1
INSTRUMENTATION_STATUS_CODE: 0
INSTRUMENTATION_RESULT: stream=

Time: 6.022

OK (1 test)


Error: **Failed to generate emma coverage.**
INSTRUMENTATION_CODE: -1

And logcat says this:

11-18 21:38:39.400: I/TestRunner(5246): run finished: 1 tests, 0 failed, 0 ignored
11-18 21:38:39.400: I/TestRunner(5246): [ 11-18 21:38:39.400  5246: 5263 E/         ]
11-18 21:38:39.400: I/TestRunner(5246): Failed to generate emma coverage.
11-18 21:38:39.400: I/TestRunner(5246): java.lang.reflect.InvocationTargetException
11-18 21:38:39.400: I/TestRunner(5246):   at java.lang.reflect.Method.invoke(Native Method)
11-18 21:38:39.400: I/TestRunner(5246):   at java.lang.reflect.Method.invoke(Method.java:372)
11-18 21:38:39.400: I/TestRunner(5246):   at android.support.test.internal.runner.listener.CoverageListener.generateCoverageReport(CoverageListener.java:80)
11-18 21:38:39.400: I/TestRunner(5246):   at android.support.test.internal.runner.listener.CoverageListener.instrumentationRunFinished(CoverageListener.java:68)
11-18 21:38:39.400: I/TestRunner(5246):   at android.support.test.internal.runner.TestExecutor.reportRunEnded(TestExecutor.java:94)
11-18 21:38:39.400: I/TestRunner(5246):   at android.support.test.internal.runner.TestExecutor.execute(TestExecutor.java:69)
11-18 21:38:39.400: I/TestRunner(5246):   at android.support.test.runner.AndroidJUnitRunner.onStart(AndroidJUnitRunner.java:262)
11-18 21:38:39.400: I/TestRunner(5246):   at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:1853)
11-18 21:38:39.400: I/TestRunner(5246): Caused by: java.io.FileNotFoundException: /sdcard/coverage.ec: open failed: EACCES (Permission denied)
11-18 21:38:39.400: I/TestRunner(5246):   at libcore.io.IoBridge.open(IoBridge.java:456)
11-18 21:38:39.400: I/TestRunner(5246):   at java.io.FileOutputStream.<init>(FileOutputStream.java:87)
11-18 21:38:39.400: I/TestRunner(5246):   at com.vladium.emma.rt.RT.dumpCoverageData(RT.java:50)
11-18 21:38:39.400: I/TestRunner(5246):   ... 8 more
11-18 21:38:39.400: I/TestRunner(5246): Caused by: android.system.ErrnoException: open failed: EACCES (Permission denied)
11-18 21:38:39.400: I/TestRunner(5246):   at libcore.io.Posix.open(Native Method)
11-18 21:38:39.400: I/TestRunner(5246):   at libcore.io.BlockGuardOs.open(BlockGuardOs.java:186)
11-18 21:38:39.400: I/TestRunner(5246):   at libcore.io.IoBridge.open(IoBridge.java:442)
11-18 21:38:39.400: I/TestRunner(5246):   ... 10 more

I can give more information if needed.

Answer 1

In case somebody meets this on Android 10, on Firebase Test Lab and above answer doesn't work, try adding grant rule (along with permission in Manifest and testCoverageEnabled true).

@get:Rule
val runtimePermissionRule: GrantPermissionRule = GrantPermissionRule.grant(
    Manifest.permission.WRITE_EXTERNAL_STORAGE,     
    Manifest.permission.READ_EXTERNAL_STORAGE
)

Answer 2

It turned out all that needed was to enable the proper permission for Writing on SD card.

<uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE"/>

in AndroidManifest.xml like in this SO answer

Also, make sure testCoverageEnabled true is enabled in app/build.gradle for debug

debug {
            testCoverageEnabled true
}

from this medium post