Why would changing directory order of jar files in WEB-INF/lib cause a NoClassDefFoundError in Tomcat 8?

Question

We have a web app that we run in Tomcat 8, and recently we've observed that the artifacts (.war files) built by some developers on our team throw a NoClassDefFoundError, while the same code built by others functions as expected.

From logs/localhost.2018-05-11.log:

org.jboss.resteasy.spi.UnhandledException: java.lang.NoClassDefFoundError: Could not initialize class org.geotools.referencing.datum.DefaultEllipsoid
    ...
Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.geotools.referencing.datum.DefaultEllipsoid
    at org.geotools.referencing.GeodeticCalculator.<init>(GeodeticCalculator.java:277)
    ...

This is sometimes, but not always, accompanied(preceded by) by:

org.jboss.resteasy.spi.UnhandledException: java.lang.IncompatibleClassChangeError: Implementing class
    ...

Examining the war files, the contents of working and broken artifacts appear to be identical, with one notable exception, the "directory ordering" of the jar files in WEB-INF/lib is different.

Performing the following procedure on the exploded war file and restarting Tomcat seems to eliminate the exception:

$ # jars in "bad" order
$ ls -U WEB-INF/lib
x.jar
b.jar
y.jar
a.jar
c.jar
z.jar
$ cp -p WEB-INF/lib/* /tmp/lib/
$ rm -r WEB-INF/lib
$ mv /tmp/lib WEB-INF/lib
$ # jars in "good" order (appears to be alphabetical after a 'cp' on my system)
$ ls -U WEB-INF/lib
a.jar
b.jar
c.jar
x.jar
y.jar
z.jar

The "good" wars don't have the jars in alphabetical order, but there appear to be a number of "good" orders a number of "bad" orders.

I initially thought we might have multiple versions of the DefaultEllipsoid class in different jars, causing a race condition between the correct version and another version, but this does not seem to be the case.

I enabled verbose classloader debugging in tomcat, and in both cases, logs/catalina.out shows this class being loaded from the correct jar:

[Loaded org.geotools.referencing.datum.DefaultEllipsoid from file: /opt/tomcat/temp/1-webapp/WEB-INF/lib/gt-referencing-11.4.jar]

Any idea of what might be going on here?

Details:

• CentOS 7
• Apache Tomcat/8.0.43
• Java 1.8.0_144
• Apache Maven 3.3.9

Answer 1

The line:

Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.geotools.referencing.datum.DefaultEllipsoid

means that the class DefaultEllipsoid is found but to be valid, there is some other class that need to be loaded but this fail. Another class is invalid.

It is this class that perhaps is duplicated with two very differents version, or one version is used for compilation, another version with different method signature at runtime.

Also, from tomcat8, the applicative jar files in WEB-INF/lib are NOT loaded according to alphabetical order anymore. I think there was a document with this on tomcat site, but for now I don't find it anymore but I found a regression bug (won't fix) on tomcat bugzilla bug 57129

This classloader stuff means that if you change some stuff on the WEB-INF/lib and restart Tomcat, then there is a bit of random class loading that make your application load one way or the other if there is duplicate jar version.

To sum up: Check the DefaultEllipsoid import and check if there is duplicate on those class. Your build also need to be cleaned to use the same version as runtime (I hope you use tools like maven to do the build)