Jacoco Coverage Report issues

Question

I am trying to define the location, where jacoco will create the coverage file for instrumentation tests running on real devices.

From the --debug run of the gradle task I see this log:

[DEBUG] [org.gradle.api.Task] DeviceConnector 'Nexus 5X - 6.0.1': installing /home/martin/workspace/lib/my-lib/build/outputs/apk/my-lib-debug-androidTest-unaligned.apk
[INFO] [org.gradle.api.Task] Starting 1 tests on Nexus 5X - 6.0.1
[INFO] [org.gradle.api.Task]  de.my.lib.utils.UtilsTest testMyTest[Nexus 5X - 6.0.1] [32mSUCCESS [0m
[DEBUG] [org.gradle.api.Task] DeviceConnector 'Nexus 5X - 6.0.1': fetching coverage data from /data/data/de.my.lib.test/coverage.ec
[DEBUG] [org.gradle.api.Task] DeviceConnector 'Nexus 5X - 6.0.1': uninstalling de.my.lib.test 13:46:14.538
[DEBUG] [org.gradle.api.internal.tasks.execution.ExecuteAtMostOnceTaskExecuter] Finished executing task ':my-lib:connectedDebugAndroidTest'

I tried 3 ways to define the location:

Using the <instrumentation> tag in the manifest file didn't change anything.

<?xml version="1.0" encoding="utf-8"?>
<manifest
    package="de.my.lib.test"
    xmlns:android="http://schemas.android.com/apk/res/android">

    <instrumentation
        android:name="android.support.test.runner.AndroidJUnitRunner"
        xmlns:tools="http://schemas.android.com/tools"
        android:targetPackage="de.my.lib.test"
        tools:replace="android:targetPackage">
        <meta-data
            android:name="coverage"
            android:value="true" />
        <meta-data
            android:name="coverageFile"
            android:value="/sdcard/coverage.ec" />
    </instrumentation>
</manifest>

I tried it with gradle but the output was the same:

defaultConfig {
    // unimportant stuff
    testApplicationId "de.my.lib.test"

    testInstrumentationRunner "android.support.test.runner.AndroidJUnitRunner"
    testInstrumentationRunnerArgument('coverageFile', '/sdcard/coverage.ec')
}

And finally I tried it with adb command:

adb shell am instrument -w -e coverage true -e coverageFile /sdcard/coverage.ec de.my.lib.test/android.support.test.runner.AndroidJUnitRunner

But there I get 2 errors:

de.my.lib.utils.UtilsTest:. Could not find class: org.jacoco.agent.rt.internal_773e439.CoverageTransformer . Time: 0,072

OK (1 test)

Error: Failed to generate emma coverage.

I am completely lost here. Any ideas?

Background Why I need it to have it stored in another place: There is a bug with adb shell run-as command on some devices and Android version so I have devices in my test device farm which return 0% coverage because the file can't be pulled. So I need the file to be stored in a publicly available location.

Answer 1

However, the coverage report is not generated yet. To enable this option, we need to add a property to our debug build variant. Using the Android plugin DSL, we can enable the coverage through the testCoverageEnabled property:

android {
    ...
    buildTypes {
        debug {
            testCoverageEnabled true
        }
        ...
    }
}

To enable the coverage report for local tests when using version 2.2.+ of Android Gradle plugin, you need to enable it in your app’s build.gradle:

android {
    //...

    testOptions {
        unitTests.all {
            jacoco {
                includeNoLocationClasses = true
            }
        }
    }
}

The instrumentation performed by Jacoco produces execution files that contain the necessary data to create the report (HTML, XML, etc.). The problem here, is that Espresso generates .ec file, while the unit tests execution generates .exec file… we have different formats!

As we are not able to configure the coverage task of Espresso, we must ensure that it is executed first. Next, we need to run unit tests, and then create the coverage data with both files (ec and exec).

To enable this, we need to edit our task once more and add the coverage.ec file as a parameter in executionData property:

apply plugin: 'jacoco'

task jacocoTestReport(type: JacocoReport, dependsOn: ['testDebugUnitTest', 'createDebugCoverageReport']) {

    reports {
        xml.enabled = true
        html.enabled = true
    }

    def fileFilter = ['**/R.class', '**/R$*.class', '**/BuildConfig.*', '**/Manifest*.*', '**/*Test*.*', 'android/**/*.*']
    def debugTree = fileTree(dir: "${buildDir}/intermediates/classes/debug", excludes: fileFilter)
    def mainSrc = "${project.projectDir}/src/main/java"

    sourceDirectories = files([mainSrc])
    classDirectories = files([debugTree])
    executionData = fileTree(dir: "$buildDir", includes: [
            "jacoco/testDebugUnitTest.exec", 
            "outputs/code-coverage/connected/*coverage.ec"
    ])
}

As Android Gradle plugin 2.2.+ now generates a coverage file for each execution, using the device / emulator in the file name, now we need to pass every file to execution data, as the file name is now dynamic. In addition, I added the createDebugCoverageReport task as a dependency of our custom task, so we don’t need to run it manually :)

Finally, when we execute both tasks, running the Espresso tests first, we will get the unified coverage report!

gradle clean jacocoTestReport

For detailed explanation see here.

Edit:

So, the command adb shell am instrument does not generates the report, for this you have to handle it using gradle solution. If you really want to test it your self, then you've to use a custom runner.

References:

1) How to Generate Android Testing Report in HTML Automatically

2) How to retrieve test results when using "adb shell am instrument"

3) https://github.com/jsankey/android-junit-report

---

Test from the command line

---

Updated to Command line method follow these steps.

Running the instrumented application

Once we have the EmmaInstrumentation class (is in ref link) in place we need a few more adjustments to be able to get the coverage report of the running application.

Firstly, we need to add the new Activity to the manifest. Secondly, we should allow our application to write to the sdcard if this is where we decided to generate the coverage report. To do it you should grant the android.permission.WRITE_EXTERNAL_STORAGE permission. Then, it's time to build and install the instrumented apk:

$ ant clean
$ ant instrument
$ ant installi

Everything is ready to start the instrumented application

$ adb shell am instrument -e coverage true \
    -w com.example.i2at.tc/\
        com.example.instrumentation.EmmaInstrumentation

Here is Source code.

Answer 2

the missing CoverageTransformer class hints for a missing dependency to the JaCoCo agent runtime. one simple solution approach might be, to provide it with what it demands (to be available on device):

testImplementation "org.jacoco:org.jacoco.agent:0.8.3"

in case you get a FileNotFoundException, when the ClassNotFoundException was fixed, make sure to have the permission to write to SD card, which appears to be another possible pitfall:

<uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE"/>

as I've just seen, there is also an org.jacoco.report package, which could be optionally added:

testImplementation "org.jacoco:org.jacoco.report:0.8.3"

package org.jacoco.agent is in every case the one which contains class CoverageTransformer.

Answer 3

However, the coverage report is not generated yet. To enable this option, we need to add a property to our debug build variant. Using the Android plugin DSL, we can enable the coverage through the testCoverageEnabled property:

android {
    ...
    buildTypes {
        debug {
            testCoverageEnabled true
        }
        ...
    }
}

To enable the coverage report for local tests when using version 2.2.+ of Android Gradle plugin, you need to enable it in your app’s build.gradle:

android {
    //...

    testOptions {
        unitTests.all {
            jacoco {
                includeNoLocationClasses = true
            }
        }
    }
}

The instrumentation performed by Jacoco produces execution files that contain the necessary data to create the report (HTML, XML, etc.). The problem here, is that Espresso generates .ec file, while the unit tests execution generates .exec file… we have different formats!

As we are not able to configure the coverage task of Espresso, we must ensure that it is executed first. Next, we need to run unit tests, and then create the coverage data with both files (ec and exec).

To enable this, we need to edit our task once more and add the coverage.ec file as a parameter in executionData property:

apply plugin: 'jacoco'

task jacocoTestReport(type: JacocoReport, dependsOn: ['testDebugUnitTest', 'createDebugCoverageReport']) {

    reports {
        xml.enabled = true
        html.enabled = true
    }

    def fileFilter = ['**/R.class', '**/R$*.class', '**/BuildConfig.*', '**/Manifest*.*', '**/*Test*.*', 'android/**/*.*']
    def debugTree = fileTree(dir: "${buildDir}/intermediates/classes/debug", excludes: fileFilter)
    def mainSrc = "${project.projectDir}/src/main/java"

    sourceDirectories = files([mainSrc])
    classDirectories = files([debugTree])
    executionData = fileTree(dir: "$buildDir", includes: [
            "jacoco/testDebugUnitTest.exec", 
            "outputs/code-coverage/connected/*coverage.ec"
    ])
}

As Android Gradle plugin 2.2.+ now generates a coverage file for each execution, using the device / emulator in the file name, now we need to pass every file to execution data, as the file name is now dynamic. In addition, I added the createDebugCoverageReport task as a dependency of our custom task, so we don’t need to run it manually :)

Finally, when we execute both tasks, running the Espresso tests first, we will get the unified coverage report!

gradle clean jacocoTestReport

For detailed explanation see here.

Edit:

So, the command adb shell am instrument does not generates the report, for this you have to handle it using gradle solution. If you really want to test it your self, then you've to use a custom runner.

References:

1) How to Generate Android Testing Report in HTML Automatically

2) How to retrieve test results when using "adb shell am instrument"

3) https://github.com/jsankey/android-junit-report

---

Test from the command line

---

Updated to Command line method follow these steps.

Running the instrumented application

Once we have the EmmaInstrumentation class (is in ref link) in place we need a few more adjustments to be able to get the coverage report of the running application.

Firstly, we need to add the new Activity to the manifest. Secondly, we should allow our application to write to the sdcard if this is where we decided to generate the coverage report. To do it you should grant the android.permission.WRITE_EXTERNAL_STORAGE permission. Then, it's time to build and install the instrumented apk:

$ ant clean
$ ant instrument
$ ant installi

Everything is ready to start the instrumented application

$ adb shell am instrument -e coverage true \
    -w com.example.i2at.tc/\
        com.example.instrumentation.EmmaInstrumentation

Here is Source code.

Answer 4

For us it was an AOSP app. We were generating the test and module apks with disabling Jack compiler which was causing this further issue. Because for us Jack compiler was needed to generate coverage.em. But the error was not quite relevant. So had no clue to go any further.

So the fix was to run the jack compiler server and get the apks generated using it.