Merging test coverage and test results from multiple Jenkins jobs

Question

Background:

We have a rather large REST API written in Java that we're testing with combination of unit and functional tests. There are many variations that are required when testing it, particularly at the functional level. While the unit tests live in-tree, the functional tests are in a separate code repository.

We are currently using Jacoco for test coverage and TestNG for running our unittests, though I believe answers to my question should be applicable to other tool combinations.

We have several different jobs in Jenkins that are triggered by a check-in to the primary project. These include jobs that run tools like Coverity as well as several different functional test jobs. These jobs are triggered by the initial commit, which is not considered to be "green", until all of the downstream jobs complete successfully.

The Problem:

How do we take coverage reports (like the Jacoco binaries and the TestNG xml files) and combine them to show total code coverage over all of our tests? Specifically, we know how to combine them if they are present in the same job/directory, but these files are spread across multiple Jenkins jobs which may be running at different times.

In my experience, the most commonly accepted way of handling this is to use the Promoted Builds Plugin to trigger all jobs, then pull their artifacts when it completes down to the triggering job. I don't feel like this scales very well, however, when you have more than one or two jobs that you're attempting to roll-up. This is especially true when you may have more than one variation on the master project (old releases, etc).

I understand it is possible to fingerprint files in Jenkins such that we know that -.jar is the same version used in Jobs A, B, and C. Does a plugin exist that can retrieve all files matching a pattern based on the existence of a different fingerprinted file?

One alternative solution (which would probably be run from an ant/groovy script), is to push test data to a directory somewhere that is tied to a git commit hash, and retrieve all such data in a roll-up job based on the git commit hash of the base project.

Are there any simple ways to do this? Has anyone figured out any better other ways to solve this problem?

Thanks,

Michael

Answer 1

Faced similar issue, tweaked jacoco maven plugin config to merge jacoco results. Basically merged jacoco-unit.exec and jacoco-it.exec into one binary and published that merged result on Jenkins via pipeline step.

pom.xml:

            <plugin>
                    <inherited>false</inherited>
                    <groupId>org.jacoco</groupId>
                    <artifactId>jacoco-maven-plugin</artifactId>
                    <version>${jacoco.agent.version}</version>
                    <executions>

                        <execution>
                            <id>merge-results</id>
                            <phase>post-integration-test</phase>
                            <goals>
                                <goal>merge</goal>
                            </goals>
                            <configuration>
                                <fileSets>
                                    <fileSet>
                                        <directory>${project.parent.build.directory}</directory>
                                        <includes>
                                            <include>jacoco-*.exec</include>
                                        </includes>
                                    </fileSet>
                                </fileSets>
                                <destFile>${project.parent.build.directory}/jacoco.exec</destFile>
                            </configuration>
                        </execution>
                    </executions>
                </plugin>

Jenkinsfile:

            echo 'Publish Jacoco trend'
            step($class: 'JacocoPublisher',
                    execPattern: '**/jacoco.exec',
                    classPattern: '**/classes',
                    sourcePattern: '**/src/main/java',
            )

However you still have to fetch jacoco binaries from several Jenkins builds by another build step or specify their locations explicitly.