Jenkins Not Outputting Junit Report Info From File

Question

Problem

A junit-formatted report is not being picked up by Jenkins, causing the reports not to be listed in the project's status screen.

Details

The junit-formatted report data is generated by a testing framework called Karma-runner (formerly Testacular). The file being ignored is created in /target/surefire-reports -- the same location as where surefire-generated reports are created. The report data looks practically the same as that generated by the maven surefire plugin except that its parent element is <testsuites> instead of <testsuite> -- <testsuite> is what the surefire generated reports have as the parent element of a report file. Here's a snippet from the karma-generated junit-formatted report, named TEST-karma.resultsTest.xml:

Junit-formatted Karma-generated report file, TEST-karma.resultsTest.xml

<?xml version="1.0"?>
<testsuites>
  <testsuite name="PhantomJS 1.9 (Mac)" package="karma.tests" timestamp="2013-04-10T13:32:26" id="0" hostname="jgmbp.local" tests="16" errors="0" failures="0" time="0.069">
    <properties>
      <property name="browser.fullName" value="Mozilla/5.0 (Macintosh; Intel Mac OS X) AppleWebKit/534.34 (KHTML, like Gecko) PhantomJS/1.9.0 Safari/534.34"/>
    </properties>
    <testcase name="successfully cancels a new member" time="0.006" classname="karma.tests PhantomJS 1.9 (Mac).Household Controller"/>
    <testcase name="should parse an existing re-eval create date, setting the data in scope" time="0.003" classname="karma.tests PhantomJS 1.9 (Mac).Re-Eval Controller"/>
    <system-out><![CDATA[
]]></system-out>
    <system-err/>
  </testsuite>
</testsuites>

The Karma tests are run during the test phase of my maven build. I've tried creating projects that only produce this one junit-report file as well as running the build so that all junit tests and karma tests generate report files. Jenkins will always pick up the surefire tests, but never the karma tests.

Thanks for any input!

Answer 1

You can publish your Karma test results in Jenkins even when building a Maven project, if you fool Jenkins with a little hack: run the Surefire plugin in also in the Javascript module you run the Karma tests in.

The surefire plugin will not find any JUnit tests in your Javascript module, but it will notify Jenkins that Surefire test results are available.

Below is a sample pom.xml, and snippets from Gruntfile.js, which runs Karma tests and a JSHint code analysis, and causes Jenkins to publish the test results and the code analysis results.

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>

<groupId>com.xxx.yyyy</groupId>
<artifactId>zzzzz</artifactId>
<version>1.0-SNAPSHOT</version>
<packaging>pom</packaging>
<name>Zzzzz</name>

<build>
    <plugins>
        <plugin>
            <artifactId>maven-antrun-plugin</artifactId>
            <version>1.7</version>
            <executions>
                <execution>
                    <phase>generate-resources</phase>
                    <configuration>
                        <target>
                            <exec
                                dir="${basedir}"
                                executable="npm"
                                failonerror="true">
                                <arg value="install"/>
                            </exec>
                            <exec
                                dir="${basedir}"
                                executable="bower"
                                failonerror="true">
                                <arg value="install"/>
                            </exec>
                            <exec
                                    dir="${basedir}"
                                    executable="grunt"
                                    failonerror="true">
                                <arg value="--no-color"/>
                                <arg value="build"/>
                            </exec>
                        </target>
                    </configuration>
                    <goals>
                        <goal>run</goal>
                    </goals>
                </execution>

                <execution>
                    <id>jshint</id>
                    <phase>test</phase>
                    <configuration>
                        <target>
                            <exec
                                dir="${basedir}"
                                executable="grunt"
                                failonerror="false">
                                <arg value="--no-color"/>
                                <arg value="karma:run"/>
                            </exec>
                            <exec
                                dir="${basedir}"
                                executable="grunt"
                                failonerror="false">
                                <arg value="--no-color"/>
                                <arg value="jshint:jenkins"/>
                            </exec>
                        </target>
                    </configuration>
                    <goals>
                        <goal>run</goal>
                    </goals>
                </execution>

            </executions>
        </plugin>
        <plugin>
            <artifactId>maven-surefire-plugin</artifactId>
            <version>2.12.4</version>
            <executions>
                <!-- This is a hack to get Jenkins to publish Karma test results when running a Maven project: we run 0 surefire tests, so Jenkins publishes the report of the Karma tests. -->
                <execution>
                    <id>dummySureFire</id>
                    <phase>test</phase>
                    <goals>
                        <goal>test</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>
        <plugin>
            <artifactId>maven-checkstyle-plugin</artifactId>
            <version>2.12</version>
            <executions>
                <!-- This is a hack to get Jenkins to publish JSHint results when running a Maven project: we run checkstyle, so Jenkins publishes the report of the JSHint run. -->
                <execution>
                    <id>dummyCheckStyle</id>
                    <phase>test</phase>
                    <goals>
                        <goal>checkstyle</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>
    </plugins>
</build>
<reporting>
    <plugins>
        <plugin>
            <artifactId>maven-surefire-report-plugin</artifactId>
            <version>2.12.4</version>
        </plugin>
    </plugins>
</reporting>
</project>

In Gruntfile.js we have the following. Outputfile name must begin with TEST- in order for it to get published:

karma: {
        run: { // produces reports for Jenkins
            configFile: 'karma.conf.js',
            singleRun: true,
            reporters : ['junit', 'coverage'],
            junitReporter : {
                outputFile: 'target/surefire-reports/TEST-results.xml'
            },
        ...

And for JSHint in Gruntfile.js:

jshint: {
        all: [
            'Gruntfile.js',
            '<%= yeoman.app %>/scripts/{,*/}*.js'
        ],
        options: {
            jshintrc: '.jshintrc',
            reporter: require('jshint-stylish')
        },
        jenkins: {
            options: {
                reporter: 'checkstyle',
                reporterOutput: "target/checkstyle-result.xml"
            },
            src: [
                'Gruntfile.js',
                '<%= yeoman.app %>/scripts/{,*/}*.js'
            ]
        }
    }

In your Jenkins job configuration you just need to check the "Publish Checkstyle analysis results" box in "Build Settings" section in order to have Jenkins publish the JSHint results. No additional Jenkins job configuration is necessary for publishing the test results.

Answer 2

Finally had the time to figure it out, thanks to amey and Wouter's input.

The key is that I'm building a maven project, and that won't work with displaying the karma tests. The only way to display karma results is to add a post build report for Junit tests. For some reason, maven-based jobs do not give you the option to add a junit report.

The solution is to create a free form job instead of a maven job. You can configure the free form job to build the maven project just fine. As long as you add the publish junit task per amey's post above, all is well.

Answer 3

Although this question is quite old:

There is a solution that works also with a Maven project in Jenkins. Problem is that the Jenkins recorder recording the tests runs directly after the surefire:test goal. So in order for your tests to get picked up, whichever plugin you use to run karma must run BEFORE the test phase, for example in the process-test-classes phase (because you cannot put something in the same phase before an existing plugin).