What could cause a sudden ClassNotFoundException in a long running process?

Question

We have a very small Web service (less than 1K lines of code) which is run by Jetty. The service worked always fine even during our stress testing phase. However, after 13 days of uptime we experienced a ClassNotFoundException in two nodes the same day.

The strange thing is that the class that wasn't found was already there (it is part of the startup routine and it was used constantly servicing previous requests). In fact, simply restarting the process solved the issue. Both nodes are in separate machines and are independent of each other. They don't depend on external resources, except on one JMS connection.

I couldn't find relevant information while Googling this, as most of the reported issues are related to missing classes in the class path while starting up the Java process, which is not our case. We suspect there could be a memory leak that in a way corrupted the JVM memory, however this can't explain why the same problem happened in two nodes around the same time. We've been running intensive stress testing during the last five days attaching a JVM monitor and a memory leak analyzer, and everything seems fine. For this tests we decreased the process memory from 2GB to 512MB.

Details:

• Using Java HotSpot(TM) 64-Bit Server VM (build 16.3-b01, mixed mode)
• Using jetty-runner-8.1.0.RC5.jar
• Original cmd line: java -Xmx2048M -jar jetty-runner-8.1.0.RC5.jar --port 5000 webapp.war
• Intel Xeon E5-2680 8 cores (x2) + 16GB RAM
• Red Hat Enterprise Linux 6
• Some frameworks in use: JBoss Resteasy, Spring IoC, Guava.

Can you please contribute with ideas regarding what could make the JVM to suddenly "forget" about the existence of a previously loaded class, not being able to load it again?

Caused by: java.lang.ClassNotFoundException: com.a.b.c.SomeClass
    at java.net.URLClassLoader$1.run(URLClassLoader.java:202) ~[na:1.6.0_37]
    at java.security.AccessController.doPrivileged(Native Method) ~[na:1.6.0_37]
    at java.net.URLClassLoader.findClass(URLClassLoader.java:190) ~[na:1.6.0_37]
    at java.lang.ClassLoader.loadClass(ClassLoader.java:306) ~[na:1.6.0_37]
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) ~[na:1.6.0_37]
    at java.lang.ClassLoader.loadClass(ClassLoader.java:247) ~[na:1.6.0_37]
    at org.eclipse.jetty.webapp.WebAppClassLoader.loadClass(WebAppClassLoader.java:424) ~[na:na]
    at org.eclipse.jetty.webapp.WebAppClassLoader.loadClass(WebAppClassLoader.java:377) ~[na:na]
    at java.lang.Class.forName0(Native Method) ~[na:1.6.0_37]
    at java.lang.Class.forName(Class.java:247) ~[na:1.6.0_37]
    at sun.reflect.generics.factory.CoreReflectionFactory.makeNamedType(CoreReflectionFactory.java:95) ~[na:1.6.0_37]
    at sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:107) ~[na:1.6.0_37]
    at sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:31) ~[na:1.6.0_37]
    at sun.reflect.annotation.AnnotationParser.parseSig(AnnotationParser.java:370) ~[na:1.6.0_37]
    at sun.reflect.annotation.AnnotationParser.parseClassValue(AnnotationParser.java:351) ~[na:1.6.0_37]
    at sun.reflect.annotation.AnnotationParser.parseMemberValue(AnnotationParser.java:280) ~[na:1.6.0_37]
    at sun.reflect.annotation.AnnotationParser.parseAnnotation(AnnotationParser.java:222) ~[na:1.6.0_37]
    at sun.reflect.annotation.AnnotationParser.parseAnnotations2(AnnotationParser.java:69) ~[na:1.6.0_37]
    at sun.reflect.annotation.AnnotationParser.parseAnnotations(AnnotationParser.java:52) ~[na:1.6.0_37]
    at java.lang.reflect.Field.declaredAnnotations(Field.java:1014) ~[na:1.6.0_37]
    at java.lang.reflect.Field.getDeclaredAnnotations(Field.java:1007) ~[na:1.6.0_37]

Edit:

Someone mentioned me that while using NFS mounts under Win, it could happen that the JVM decides to unload a class, and then re-loads it when it's needed. If in the middle of this process the NFS connection was broken, the file handle will be invalid and the re-loading will fail with a similar stacktrace. In our case, we are using Linux and all the involved files are in the same mount, which is a local hard disk. Just for the sake of more testing, I've CD'd into the Jetty temporary directory and manually deleted one well known for one specific service class. If the JVM unloads it and then attempts to re-load it from the classes directory, it will fail. While this does not explain the original problem, it might put more information on the table...

Answer 1

This is what is happening:

1. When the service is started using the cmd detailed above, Jetty creates a sub-dir under the "/tmp", which holds the application classes and resources loaded by the JVM.
2. After a period of inactivity (in our specific scenario, between 13 and 20 days) that directory disappears. As a result, the JVM can't load the file. We still don't know precisely if the JVM unloaded the class before this error, or why it tried to re-read the *.class file. It would be interesting to look into the source code and learn about this, but that's not in our short-term ToDo list.
3. Simply restarting Jetty caused the missing directories to be recreated, and the service is up again.

A good hint we had is that some people reported a similar issue while loading resources in JARs over NFS on Windows (if network connectivity is lost for a brief moment, NFS handles become invalid and the JVM fails with a similar error). This is not our case (/tmp is local storage), but quite similar.

Thanks everyone for their help.