coverage per test with gradle using jacoco offline instrumentation - java.lang.IllegalStateException: JaCoCo agent not started

Question

I am trying to do a coverage per test using gradle+jacoco.

Idea is to have an implementation of org.gradle.api.tasks.testing.TestListener, and reset session-id at regular events (test start and test end) and dump coverage.

(Entire project to demo the problem is at https://github.com/jayanmn/gradle-coverage-per-test-debug/tree/main )

Instrumentation is done via jacoco-offline instrumentation using ant tasks inside gradle. (offline as the testing environment includes some tests - somewhat integration type ).

buildscript {
    repositories {
        jcenter()
        mavenCentral()
    }
    dependencies {
        classpath group: 'org.jacoco', name: 'org.jacoco.agent', version: '0.8.5', classifier: 'runtime'
        classpath group: 'org.jacoco', name: 'org.jacoco.core', version: '0.8.5'
    }
}

The dependency section is like below

    dependencies {
    //offline instrument via ant task
    jacocoAnt group: 'org.jacoco', name: 'org.jacoco.ant', version: '0.8.5', classifier: 'nodeps'

    //the jacoco runtime private for RT.getAgent()
    jacocoRuntime group: 'org.jacoco', name: 'org.jacoco.agent', version: '0.8.5', classifier: 'runtime'

    testImplementation 'junit:junit:4.12'
}

Running of test (sampletest)

task sampletest(type: Test) {
    ignoreFailures = true
    //jars used every step is same.
    println classpath.asPath

    forkEvery 1
    maxParallelForks 1

    doFirst {
        println "org.jacoco.agent.rt.RT loaded from " + RT.class.getProtectionDomain().getCodeSource()

        systemProperty 'jacoco-agent.destfile', buildDir.path + '/jacoco/firsttests.exec'
        classpath = files(instrument.outputDir) + classpath + configurations.jacocoRuntime

    }
    beforeTest { TestDescriptor descriptor ->
        println "org.jacoco.agent.rt.RT loaded from " + RT.class.getProtectionDomain().getCodeSource()
    }

    afterTest { TestDescriptor descriptor ->
        //get agent will fail with Agent not started.
        systemProperty 'jacoco-agent.destfile', buildDir.path + '/jacoco/aftertests.exec'

        //agent not started error
        def agent = RT.getAgent()
        agent.sessionId = descriptor.name
        agent.dump(true)

    }
}

For some reason

> java.lang.IllegalStateException: JaCoCo agent not started.

* Exception is:
org.gradle.api.tasks.TaskExecutionException: Execution failed for task ':sampletest'.
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.lambda$executeIfValid$1(ExecuteActionsTaskExecuter.java:207)
        at org.gradle.internal.Try$Failure.ifSuccessfulOrElse(Try.java:263)
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeIfValid(ExecuteActionsTaskExecuter.java:205)
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:186)
        at org.gradle.api.internal.tasks.execution.CleanupStaleOutputsExecuter.execute(CleanupStaleOutputsExecuter.java:114)
        at org.gradle.api.internal.tasks.execution.FinalizePropertiesTaskExecuter.execute(FinalizePropertiesTaskExecuter.java:46)
        at org.gradle.api.internal.tasks.execution.ResolveTaskExecutionModeExecuter.execute(ResolveTaskExecutionModeExecuter.java:62)
        at org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.java:57)
        at org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:56)
        at org.gradle.api.internal.tasks.execution.CatchExceptionTaskExecuter.execute(CatchExceptionTaskExecuter.java:36)
        at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.executeTask(EventFiringTaskExecuter.java:77)
        at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:55)
        at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:52)
        at org.gradle.internal.operations.DefaultBuildOperationExecutor$CallableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:409)
        at org.gradle.internal.operations.DefaultBuildOperationExecutor$CallableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:399)
        at org.gradle.internal.operations.DefaultBuildOperationExecutor$1.execute(DefaultBuildOperationExecutor.java:157)
        at org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:242)
        at org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:150)
        at org.gradle.internal.operations.DefaultBuildOperationExecutor.call(DefaultBuildOperationExecutor.java:94)
        at org.gradle.internal.operations.DelegatingBuildOperationExecutor.call(DelegatingBuildOperationExecutor.java:36)
        at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter.execute(EventFiringTaskExecuter.java:52)
        at org.gradle.execution.plan.LocalTaskNodeExecutor.execute(LocalTaskNodeExecutor.java:41)
        at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:356)
        at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:343)
        at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:336)
        at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:322)
        at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.lambda$run$0(DefaultPlanExecutor.java:127)
        at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.execute(DefaultPlanExecutor.java:191)
        at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.executeNextNode(DefaultPlanExecutor.java:182)
        at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.run(DefaultPlanExecutor.java:124)
        at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
        at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48)
        at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:56)
Caused by: java.lang.IllegalStateException: JaCoCo agent not started.
        at org.jacoco.agent.rt.internal_43f5073.Agent.getInstance(Agent.java:77)
        at org.jacoco.agent.rt.RT.getAgent(RT.java:33)
        at org.jacoco.agent.rt.RT$getAgent.call(Unknown Source)
        at build_q0s35v9r0an7s8a7ayxelgpr$_run_closure6$_closure13.doCall(C:\repos\number-utils\build.gradle:91)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at org.gradle.listener.ClosureBackedMethodInvocationDispatch.dispatch(ClosureBackedMethodInvocationDispatch.java:41)
        at org.gradle.listener.ClosureBackedMethodInvocationDispatch.dispatch(ClosureBackedMethodInvocationDispatch.java:25)
        at org.gradle.internal.event.AbstractBroadcastDispatch.dispatch(AbstractBroadcastDispatch.java:42)
        at org.gradle.internal.event.BroadcastDispatch$SingletonDispatch.dispatch(BroadcastDispatch.java:245)
        at org.gradle.internal.event.BroadcastDispatch$SingletonDispatch.dispatch(BroadcastDispatch.java:157)
        at org.gradle.internal.event.AbstractBroadcastDispatch.dispatch(AbstractBroadcastDispatch.java:58)
        at org.gradle.internal.event.BroadcastDispatch$CompositeDispatch.dispatch(BroadcastDispatch.java:346)
        at org.gradle.internal.event.BroadcastDispatch$CompositeDispatch.dispatch(BroadcastDispatch.java:249)
        at org.gradle.internal.event.ListenerBroadcast.dispatch(ListenerBroadcast.java:141)
        at org.gradle.internal.event.ListenerBroadcast.dispatch(ListenerBroadcast.java:37)
        at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
        at com.sun.proxy.$Proxy69.afterTest(Unknown Source)
        at org.gradle.api.internal.tasks.testing.results.TestListenerAdapter.completed(TestListenerAdapter.java:50)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
        at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
        at org.gradle.internal.event.AbstractBroadcastDispatch.dispatch(AbstractBroadcastDispatch.java:42)
        at org.gradle.internal.event.BroadcastDispatch$SingletonDispatch.dispatch(BroadcastDispatch.java:245)
        at org.gradle.internal.event.BroadcastDispatch$SingletonDispatch.dispatch(BroadcastDispatch.java:157)
        at org.gradle.internal.event.AbstractBroadcastDispatch.dispatch(AbstractBroadcastDispatch.java:58)
        at org.gradle.internal.event.BroadcastDispatch$CompositeDispatch.dispatch(BroadcastDispatch.java:346)
        at org.gradle.internal.event.BroadcastDispatch$CompositeDispatch.dispatch(BroadcastDispatch.java:249)
        at org.gradle.internal.event.ListenerBroadcast.dispatch(ListenerBroadcast.java:141)
        at org.gradle.internal.event.ListenerBroadcast.dispatch(ListenerBroadcast.java:37)
        at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
        at com.sun.proxy.$Proxy71.completed(Unknown Source)
        at org.gradle.api.internal.tasks.testing.results.StateTrackingTestResultProcessor.completed(StateTrackingTestResultProcessor.java:96)
        at org.gradle.api.internal.tasks.testing.results.AttachParentTestResultProcessor.completed(AttachParentTestResultProcessor.java:56)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
        

Problem is likely to be with classloaders of "buildscript" and test block? What is the correct way to add jacoco-agent-runtime ?

Answer 1

afterTest org.gradle.api.tasks.testing.TestListener is executed by Gradle in the VM for Gradle, but not in the VM for tests with JaCoCo.

For the proof:

for src/test/java/ExampleTest.java

public class ExampleTest {
  @org.junit.Test
  public void test() {
    System.out.println(java.lang.management.ManagementFactory.getRuntimeMXBean().getName());
  }
}

and build.gradle

plugins {
  id "java"
}

repositories {
    mavenCentral()
}

dependencies {
    testImplementation "junit:junit:4.12"
}

test {
    testLogging.showStandardStreams = true
    afterTest {
        System.println(java.lang.management.ManagementFactory.getRuntimeMXBean().getName())
    }
}

execution of command gradle build will produce something like

> Task :test

ExampleTest > test STANDARD_OUT
    [email protected]
[email protected]

two numbers 33542 and 33018 are IDs of Java processes, and as you can see they are different.

---

org.jacoco.agent.rt.RT can be used to communicate only with JaCoCo running in the same VM.

To communicate with JaCoCo in another VM you can use agent output modes tcpclient and tcpserver, or via JMX using agent option jmx=true - see https://www.jacoco.org/jacoco/trunk/doc/agent.html And the same options available for the case of offline instrumentation - see https://www.eclemma.org/jacoco/trunk/doc/offline.html

---

Idea is to have an implementation of org.gradle.api.tasks.testing.TestListener, and reset session-id at regular events (test start and test end) and dump coverage.

JUnit 4 has org.junit.runner.notification.RunListener which is executed inside VM for tests by JUnit, an example of its implementation similar to your idea can be found at https://github.com/SonarSource/sonar-java/blob/5.14.0.18788/sonar-jacoco-listeners/src/main/java/org/sonar/java/jacoco/JUnitListener.java

Unfortunately according to https://github.com/gradle/gradle/issues/1330 seems that as of today there is no easy way in Gradle to configure JUnit 4 to use org.junit.runner.notification.RunListener, however seems possible to configure JUnit 5 to use its org.junit.platform.launcher.TestExecutionListener.