Jacoco code coverage dropped with migration to Java 11

Question

I have several Gradle projects that were built using Java 8, and after converting them recently to use Java 11, Jacoco code coverage reports have been reporting much lower percentages than before. On one project, immediately after the transition, my coverage dropped from 81% to 16%.

I tried updating the Jacoco plugin to 0.8.3 (which has official JDK 11 support), Gradle to 5.4, and TestNG to 6.14.3 (not sure if this has any effect; thought it couldn't hurt to be on the latest version). Even after these changes, the project I mentioned above has 16% coverage. I manually checked a few of the classes it reported 0% coverage on and found that they actually did have test coverage on them.

For example, I added this method to one of my classes:

public String helloWorld(){
        return "hello";
    }

I then used it in a test:

@Test(groups = IntegrationTest.INTEGRATION_GROUP)
    public void testHelloWorld() {
        String helloWorld = authManager.helloWorld();
        assertEquals(helloWorld, "hello");
    }

And the coverage reported as 0:

If it's helpful, here are my Jacoco Gradle settings. I'm using a custom plugin to configure them.

  class ManagedJacocoPlugin implements ManagedPlugin {
  @Override
  void apply(PluginManager pluginManager) {
    pluginManager.apply(JacocoPlugin.class)
  }

  @Override
  void configure(Project project, GradlePluginConfig pluginConfig) {
    def jacoco = project.extensions.getByName("jacoco")
    jacoco.toolVersion = "0.8.3"

    def jacocoTestReport = project.tasks.getByName('jacocoTestReport')
    jacocoTestReport.reports {
      xml.enabled false
      csv.enabled false
    }

    project.tasks.withType(Test).each { t ->
      t.jacoco {
        destinationFile = project.file("$project.buildDir/jacoco/test.exec")
      }
    }

    jacocoTestReport.dependsOn "integrationTest"
  }
}

As far as I'm able to gather, Java 11 should be fully supported for Jacoco coverage given the versions of the tools I'm using. What am I missing here?

Answer 1

Here is what page https://stackoverflow.com/help/mcve says about How to create a Minimal, Complete, and Verifiable example:

Make sure it's complete

Copy the code from your question into a new file or project, then run it. If it doesn't run for you, then it won't run for anyone else.

However who knows what is ManagedPlugin in your example?

But ok, let's try to follow the above advice and use what we have, pretending that we have time on guessing and that we'll be lucky to guess correctly.

Everything except ManagedPlugin after addition of many missing pieces becomes a following build.gradle

apply plugin: 'java'
apply plugin: 'jacoco'

repositories {
    mavenCentral()
}

dependencies {
    testCompile group: 'org.testng', name: 'testng', version: '6.14.3'
}

test {
    useTestNG() {
        includeGroups('unit')
    }
}

task integrationTest(type: Test, dependsOn: ['test']) {
    useTestNG() {
        includeGroups('integration')
    }
}


def jacoco = project.extensions.getByName("jacoco")
jacoco.toolVersion = "0.8.3"

def jacocoTestReport = project.tasks.getByName('jacocoTestReport')
jacocoTestReport.reports {
    xml.enabled false
    csv.enabled false
}

project.tasks.withType(Test).each { t ->
    t.jacoco {
        destinationFile = project.file("$project.buildDir/jacoco/test.exec")
    }
}

jacocoTestReport.dependsOn "integrationTest"

method helloWorld goes into src/main/Example.java

class Example {
    public String helloWorld() {
        return "hello";
    }
}

method testHelloWorld goes into src/test/ExampleTest.java

import org.testng.annotations.Test;
import static org.testng.Assert.*;

class ExampleTest {
    Example authManager = new Example();

    @Test(groups = "integration")
    public void testHelloWorld() {
        String helloWorld = authManager.helloWorld();
        assertEquals(helloWorld, "hello");
    }
}

Execution of gradle clean jacocoTestReport using Gralde 5.4 and JDK 11.0.1 produces following report

Thus we can conclude that provided example is definitely not complete.

Let's try to guess againg and add into src/main/java/Example.java

    public void anotherMethod() {
    }

and into src/test/java/ExampleTest.java

    @Test(groups = "unit")
    public void test() {
       new Example().anotherMethod();
    }

Now execution of gradle clean jacocoTestReport produces following report

Seems that now we can reproduce your problem.

Why anotherMethod is not covered? Let's follow another great advice from https://stackoverflow.com/help/mcve :

Divide and conquer. When you have a small amount of code, but the source of the problem is entirely unclear, start removing code a bit at a time until the problem disappears – then add the last part back.

which also works not only for code, but for changes in versions - let's try to reverse your change of Gradle version back from 5.4 to 4.10.3 and with it execution of gradle clean jacocoTestReport produces

Thus we can conclude that something in Gradle was changed. Let's check its changelog - https://docs.gradle.org/5.0/release-notes.html contains a very interesting statement:

JaCoCo plugin now works with the build cache and parallel test execution

... the tasks running with code coverage are configured to delete the execution data just before they starts executing ...

Task integrationTest removes data collected by task test. Let's try to not use same file:

//project.tasks.withType(Test).each { t ->
//    t.jacoco {
//        destinationFile = project.file("$project.buildDir/jacoco/test.exec")
//    }
//}

jacocoTestReport.executionData(test)
jacocoTestReport.executionData(integrationTest)

Now execution of gradle clean jacocoTestReport even with Gradle 5.4 produces