Intermittent NoClassDefFoundError when running a maven/surefire build in jenkins

Question

We are building a large multi module Maven project on Jenkins, including running a large number of unit test.

Once every few builds the build fails on NoClassDefFoundError on RunListener - which is located in the unit jar. As you can see from the log below - JUnit is included in the classpath.

The error seems to appear completely at random.

Log

Waiting for Jenkins to finish collecting data
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.17:test (default-test) on project taboola-svc: Execution default-test of goal org.apache.maven.plugins:maven-surefire-plugin:2.17:test failed: A required class was missing while executing org.apache.maven.plugins:maven-surefire-plugin:2.17:test: org/junit/runner/notification/RunListener
[ERROR] -----------------------------------------------------
[ERROR] realm =    plugin>org.apache.maven.plugins:maven-surefire-plugin:2.17
[ERROR] strategy = org.codehaus.plexus.classworlds.strategy.SelfFirstStrategy
[ERROR] urls[0] = file:/home/builder/.m2/repository/org/apache/maven/plugins/maven-surefire-plugin/2.17/maven-surefire-plugin-2.17.jar
[ERROR] urls[1] = file:/home/builder/.m2/repository/org/apache/maven/surefire/surefire-junit47/2.17/surefire-junit47-2.17.jar
[ERROR] urls[2] = file:/home/builder/.m2/repository/org/apache/maven/surefire/common-junit48/2.17/common-junit48-2.17.jar
[ERROR] urls[3] = file:/home/builder/.m2/repository/org/apache/maven/surefire/common-junit4/2.17/common-junit4-2.17.jar
[ERROR] urls[4] = file:/home/builder/.m2/repository/org/apache/maven/surefire/common-junit3/2.17/common-junit3-2.17.jar
[ERROR] urls[5] = file:/home/builder/.m2/repository/org/apache/maven/surefire/surefire-grouper/2.17/surefire-grouper-2.17.jar
[ERROR] urls[6] = file:/home/builder/.m2/repository/org/apache/maven/shared/maven-shared-utils/0.4/maven-shared-utils-0.4.jar
[ERROR] urls[7] = file:/home/builder/.m2/repository/org/apache/maven/surefire/common-java5/2.17/common-java5-2.17.jar
[ERROR] urls[8] = file:/home/builder/.m2/repository/junit/junit/4.11/junit-4.11.jar
[ERROR] urls[9] = file:/home/builder/.m2/repository/org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3.jar
[ERROR] urls[10] = file:/home/builder/.m2/repository/org/apache/maven/surefire/maven-surefire-common/2.17/maven-surefire-common-2.17.jar
[ERROR] urls[11] = file:/home/builder/.m2/repository/org/apache/maven/surefire/surefire-booter/2.17/surefire-booter-2.17.jar
[ERROR] urls[12] = file:/home/builder/.m2/repository/org/codehaus/plexus/plexus-utils/1.5.1/plexus-utils-1.5.1.jar
[ERROR] urls[13] = file:/home/builder/.m2/repository/org/apache/maven/reporting/maven-reporting-api/2.0.9/maven-reporting-api-2.0.9.jar
[ERROR] urls[14] = file:/home/builder/.m2/repository/org/apache/commons/commons-lang3/3.1/commons-lang3-3.1.jar
[ERROR] urls[15] = file:/home/builder/.m2/repository/org/apache/maven/surefire/surefire-api/2.17/surefire-api-2.17.jar
[ERROR] urls[16] = file:/home/builder/.m2/repository/org/apache/maven/plugin-tools/maven-plugin-annotations/3.2/maven-plugin-annotations-3.2.jar
[ERROR] Number of foreign imports: 1
[ERROR] import: Entry[import  from realm ClassRealm[maven.api, parent: null]]
[ERROR] 
[ERROR] -----------------------------------------------------: org.junit.runner.notification.RunListener
[ERROR] -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.17:test (default-test) on project taboola-svc: Execution default-test of goal org.apache.maven.plugins:maven-surefire-plugin:2.17:test failed: A required class was missing while executing org.apache.maven.plugins:maven-surefire-plugin:2.17:test: org/junit/runner/notification/RunListener
-----------------------------------------------------
realm =    plugin>org.apache.maven.plugins:maven-surefire-plugin:2.17
strategy = org.codehaus.plexus.classworlds.strategy.SelfFirstStrategy
urls[0] = file:/home/builder/.m2/repository/org/apache/maven/plugins/maven-surefire-plugin/2.17/maven-surefire-plugin-2.17.jar
urls[1] = file:/home/builder/.m2/repository/org/apache/maven/surefire/surefire-junit47/2.17/surefire-junit47-2.17.jar
urls[2] = file:/home/builder/.m2/repository/org/apache/maven/surefire/common-junit48/2.17/common-junit48-2.17.jar
urls[3] = file:/home/builder/.m2/repository/org/apache/maven/surefire/common-junit4/2.17/common-junit4-2.17.jar
urls[4] = file:/home/builder/.m2/repository/org/apache/maven/surefire/common-junit3/2.17/common-junit3-2.17.jar
urls[5] = file:/home/builder/.m2/repository/org/apache/maven/surefire/surefire-grouper/2.17/surefire-grouper-2.17.jar
urls[6] = file:/home/builder/.m2/repository/org/apache/maven/shared/maven-shared-utils/0.4/maven-shared-utils-0.4.jar
urls[7] = file:/home/builder/.m2/repository/org/apache/maven/surefire/common-java5/2.17/common-java5-2.17.jar
urls[8] = file:/home/builder/.m2/repository/junit/junit/4.11/junit-4.11.jar
urls[9] = file:/home/builder/.m2/repository/org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3.jar
urls[10] = file:/home/builder/.m2/repository/org/apache/maven/surefire/maven-surefire-common/2.17/maven-surefire-common-2.17.jar
urls[11] = file:/home/builder/.m2/repository/org/apache/maven/surefire/surefire-booter/2.17/surefire-booter-2.17.jar
urls[12] = file:/home/builder/.m2/repository/org/codehaus/plexus/plexus-utils/1.5.1/plexus-utils-1.5.1.jar
urls[13] = file:/home/builder/.m2/repository/org/apache/maven/reporting/maven-reporting-api/2.0.9/maven-reporting-api-2.0.9.jar
urls[14] = file:/home/builder/.m2/repository/org/apache/commons/commons-lang3/3.1/commons-lang3-3.1.jar
urls[15] = file:/home/builder/.m2/repository/org/apache/maven/surefire/surefire-api/2.17/surefire-api-2.17.jar
urls[16] = file:/home/builder/.m2/repository/org/apache/maven/plugin-tools/maven-plugin-annotations/3.2/maven-plugin-annotations-3.2.jar
Number of foreign imports: 1
import: Entry[import  from realm ClassRealm[maven.api, parent: null]]

-----------------------------------------------------

    at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:225)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84)
    at org.apache.maven.lifecycle.internal.LifecycleThreadedBuilder$1.call(LifecycleThreadedBuilder.java:167)
    at org.apache.maven.lifecycle.internal.LifecycleThreadedBuilder$1.call(LifecycleThreadedBuilder.java:163)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
    at java.util.concurrent.FutureTask.run(FutureTask.java:138)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
    at java.util.concurrent.FutureTask.run(FutureTask.java:138)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
    at java.lang.Thread.run(Thread.java:662)
Caused by: org.apache.maven.plugin.PluginExecutionException: Execution default-test of goal org.apache.maven.plugins:maven-surefire-plugin:2.17:test failed: A required class was missing while executing org.apache.maven.plugins:maven-surefire-plugin:2.17:test: org/junit/runner/notification/RunListener
-----------------------------------------------------
realm =    plugin>org.apache.maven.plugins:maven-surefire-plugin:2.17
strategy = org.codehaus.plexus.classworlds.strategy.SelfFirstStrategy
urls[0] = file:/home/builder/.m2/repository/org/apache/maven/plugins/maven-surefire-plugin/2.17/maven-surefire-plugin-2.17.jar
urls[1] = file:/home/builder/.m2/repository/org/apache/maven/surefire/surefire-junit47/2.17/surefire-junit47-2.17.jar
urls[2] = file:/home/builder/.m2/repository/org/apache/maven/surefire/common-junit48/2.17/common-junit48-2.17.jar
urls[3] = file:/home/builder/.m2/repository/org/apache/maven/surefire/common-junit4/2.17/common-junit4-2.17.jar
urls[4] = file:/home/builder/.m2/repository/org/apache/maven/surefire/common-junit3/2.17/common-junit3-2.17.jar
urls[5] = file:/home/builder/.m2/repository/org/apache/maven/surefire/surefire-grouper/2.17/surefire-grouper-2.17.jar
urls[6] = file:/home/builder/.m2/repository/org/apache/maven/shared/maven-shared-utils/0.4/maven-shared-utils-0.4.jar
urls[7] = file:/home/builder/.m2/repository/org/apache/maven/surefire/common-java5/2.17/common-java5-2.17.jar
urls[8] = file:/home/builder/.m2/repository/junit/junit/4.11/junit-4.11.jar
urls[9] = file:/home/builder/.m2/repository/org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3.jar
urls[10] = file:/home/builder/.m2/repository/org/apache/maven/surefire/maven-surefire-common/2.17/maven-surefire-common-2.17.jar
urls[11] = file:/home/builder/.m2/repository/org/apache/maven/surefire/surefire-booter/2.17/surefire-booter-2.17.jar
urls[12] = file:/home/builder/.m2/repository/org/codehaus/plexus/plexus-utils/1.5.1/plexus-utils-1.5.1.jar
urls[13] = file:/home/builder/.m2/repository/org/apache/maven/reporting/maven-reporting-api/2.0.9/maven-reporting-api-2.0.9.jar
urls[14] = file:/home/builder/.m2/repository/org/apache/commons/commons-lang3/3.1/commons-lang3-3.1.jar
urls[15] = file:/home/builder/.m2/repository/org/apache/maven/surefire/surefire-api/2.17/surefire-api-2.17.jar
urls[16] = file:/home/builder/.m2/repository/org/apache/maven/plugin-tools/maven-plugin-annotations/3.2/maven-plugin-annotations-3.2.jar
Number of foreign imports: 1
import: Entry[import  from realm ClassRealm[maven.api, parent: null]]

-----------------------------------------------------

    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:127)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:209)
    ... 13 more
Caused by: org.apache.maven.plugin.PluginContainerException: A required class was missing while executing org.apache.maven.plugins:maven-surefire-plugin:2.17:test: org/junit/runner/notification/RunListener
-----------------------------------------------------
realm =    plugin>org.apache.maven.plugins:maven-surefire-plugin:2.17
strategy = org.codehaus.plexus.classworlds.strategy.SelfFirstStrategy
urls[0] = file:/home/builder/.m2/repository/org/apache/maven/plugins/maven-surefire-plugin/2.17/maven-surefire-plugin-2.17.jar
urls[1] = file:/home/builder/.m2/repository/org/apache/maven/surefire/surefire-junit47/2.17/surefire-junit47-2.17.jar
urls[2] = file:/home/builder/.m2/repository/org/apache/maven/surefire/common-junit48/2.17/common-junit48-2.17.jar
urls[3] = file:/home/builder/.m2/repository/org/apache/maven/surefire/common-junit4/2.17/common-junit4-2.17.jar
urls[4] = file:/home/builder/.m2/repository/org/apache/maven/surefire/common-junit3/2.17/common-junit3-2.17.jar
urls[5] = file:/home/builder/.m2/repository/org/apache/maven/surefire/surefire-grouper/2.17/surefire-grouper-2.17.jar
urls[6] = file:/home/builder/.m2/repository/org/apache/maven/shared/maven-shared-utils/0.4/maven-shared-utils-0.4.jar
urls[7] = file:/home/builder/.m2/repository/org/apache/maven/surefire/common-java5/2.17/common-java5-2.17.jar
urls[8] = file:/home/builder/.m2/repository/junit/junit/4.11/junit-4.11.jar
urls[9] = file:/home/builder/.m2/repository/org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3.jar
urls[10] = file:/home/builder/.m2/repository/org/apache/maven/surefire/maven-surefire-common/2.17/maven-surefire-common-2.17.jar
urls[11] = file:/home/builder/.m2/repository/org/apache/maven/surefire/surefire-booter/2.17/surefire-booter-2.17.jar
urls[12] = file:/home/builder/.m2/repository/org/codehaus/plexus/plexus-utils/1.5.1/plexus-utils-1.5.1.jar
urls[13] = file:/home/builder/.m2/repository/org/apache/maven/reporting/maven-reporting-api/2.0.9/maven-reporting-api-2.0.9.jar
urls[14] = file:/home/builder/.m2/repository/org/apache/commons/commons-lang3/3.1/commons-lang3-3.1.jar
urls[15] = file:/home/builder/.m2/repository/org/apache/maven/surefire/surefire-api/2.17/surefire-api-2.17.jar
urls[16] = file:/home/builder/.m2/repository/org/apache/maven/plugin-tools/maven-plugin-annotations/3.2/maven-plugin-annotations-3.2.jar
Number of foreign imports: 1
import: Entry[import  from realm ClassRealm[maven.api, parent: null]]

-----------------------------------------------------

    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:125)
    ... 14 more
Caused by: java.lang.NoClassDefFoundError: org/junit/runner/notification/RunListener
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
    at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
    at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
    at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
    at org.apache.maven.surefire.booter.IsolatedClassLoader.loadClass(IsolatedClassLoader.java:97)
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
    at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
    at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
    at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
    at org.apache.maven.surefire.booter.IsolatedClassLoader.loadClass(IsolatedClassLoader.java:97)
    at java.lang.Class.getDeclaredConstructors0(Native Method)
    at java.lang.Class.privateGetDeclaredConstructors(Class.java:2398)
    at java.lang.Class.getConstructor0(Class.java:2708)
    at java.lang.Class.getConstructor(Class.java:1659)
    at org.apache.maven.surefire.util.ReflectionUtils.getConstructor(ReflectionUtils.java:76)
    at org.apache.maven.surefire.util.ReflectionUtils.instantiateOneArg(ReflectionUtils.java:129)
    at org.apache.maven.surefire.booter.SurefireReflector.instantiateProvider(SurefireReflector.java:235)
    at org.apache.maven.surefire.booter.ProviderFactory.createProvider(ProviderFactory.java:113)
    at org.apache.maven.plugin.surefire.booterclient.ForkStarter.getSuitesIterator(ForkStarter.java:512)
    at org.apache.maven.plugin.surefire.booterclient.ForkStarter.runSuitesForkPerTestSet(ForkStarter.java:277)
    at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:169)
    at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:967)
    at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:831)
    at org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:729)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:101)
    ... 14 more
Caused by: java.lang.ClassNotFoundException: org.junit.runner.notification.RunListener
    at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
    at org.apache.maven.surefire.booter.IsolatedClassLoader.loadClass(IsolatedClassLoader.java:97)
    ... 53 more
[ERROR] 
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginContainerException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :taboola-svc

---

More Info

• The build is running on a slave
• Jenkins ver. 1.565.2
• Tried with both maven 3.0.4, and 3.1.1
• Tried with surefire 2.16, 17 & 18
• Tried with junit 4.10 & 4.11

---

I have verified the SHA1 of JUnit jar in the Maven repository and made sure the jar file is not corrupted.

---

Command line

(defined in a jenkins job)

clean install -P FastTests -DIsBuildSystem=1 -T 4 -X -e

---

Relevant parts from the parent pom.xml

The project itself is a multi module pom project, with the parent pom defining the surefire definitions as attached.

....

        <!-- Generic tests plugin for maven -->
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-surefire-plugin</artifactId>
            <version>${testing.surefire.version}</version>
            <configuration>

                <systemPropertyVariables>
                    <startPortForDebug>${startPortForDebug}</startPortForDebug>
                    <surefireForkNumber>fork_${surefire.forkNumber}</surefireForkNumber>
                    <geoip.db.dir>${basedir}/../framework/target/</geoip.db.dir>
                    <java.awt.headless>true</java.awt.headless>
                    <redirectTestOutputToFile>true</redirectTestOutputToFile>
                </systemPropertyVariables>



                <!-- Use new JVM for each test -->
                <parallel>classes</parallel>

                <threadCount>1</threadCount>
                <forkCount>${testing.forkCount}</forkCount>
                <reuseForks>${testing.reuseFork}</reuseForks>
                <useSystemClassLoader>${testing.useSystemClassLoader}</useSystemClassLoader>
                <useManifestOnlyJar>${testing.useManifestOnlyJar}</useManifestOnlyJar>

                <!-- Display extra information on exception prints -->
                <useFile>true</useFile>
                <redirectTestOutputToFile>true</redirectTestOutputToFile>
                <runOrder>alphabetical</runOrder>

                <!-- Display extra information on exception prints -->
                <useFile>false</useFile>

                <!-- -->
                <forkedProcessTimeoutInSeconds>300</forkedProcessTimeoutInSeconds>


                <argLine>-XX:PermSize=128m -XX:MaxPermSize=512m -DIsBuildSystem=${is.build.system}</argLine>

            </configuration>
            <dependencies>
                <dependency>
                    <groupId>org.apache.maven.surefire</groupId>
                    <artifactId>surefire-junit47</artifactId>
                    <version>${testing.surefire.version}</version>
                </dependency>
                <dependency>
                    <groupId>junit</groupId>
                    <artifactId>junit</artifactId>
                    <version>${junit.version}</version>
                </dependency>
            </dependencies>

        </plugin>

....
        <profile>
            <id>FastTests</id>
            <properties>
                <testing.forkCount>0.5C</testing.forkCount>
                <testing.reuseFork>false</testing.reuseFork>
                <testing.useSystemClassLoader>true</testing.useSystemClassLoader>
                <testing.useManifestOnlyJar>false</testing.useManifestOnlyJar>
            </properties>
        </profile>

Answer 1

Try running the maven build without the -T 4.

The parallel builds feature in Maven 3 is still in the experimental stages, and it appears that there are known issues with thread safety in the plexus-utils library prior to version 2.0.5 - you're using version 1.5.1.

Edit:

Your problem is this dependency:

        <dependency>
            <groupId>org.apache.maven.surefire</groupId>
            <artifactId>surefire-junit47</artifactId>
            <version>${testing.surefire.version}</version>
        </dependency>

surefire-junit47 depends on maven-surefire-common, which in turn depends on maven-artifact, which depends on plexus-utils-1.5.1. As stated above, this has known thread safety issues, which is likely what is causing your problems. I'm not sure what parts of the surefire-junit47 library you're using, but if switching to surefire-junit4 is an option, then that should solve your problem.

After looking at the docs for the surefire plugin a little more, it looks like you need the surefire-junit47 provider to run parallel tests. I'd try overriding plexus-utils to be version 2.0.5 and see if that works; otherwise, you can override the maven-artifact version to 3.0.3 or higher and see if that works.