Classes missing if application runs for a long time

Question

I have a funny problem - if my application runs for a long time (> 20h), then sometimes I get NoClassDefFound error - seems like JVM decided that the class is not going to be used anyway and GCd it.

To be a bit more specific, here's an example case:

object ErrorHandler extends PartialFunction[Throwable,Unit] {
  def isDefinedAt(t: Throwable) = true
  def apply(e: Throwable) =e match {
    // ... handle errors
  }
}

// somewhere else in the code...
try {
  // ... long running code, can take more than 20 hours to complete
} catch (ErrorHandler)

And I get the following exception:

Exception in thread "main" java.lang.NoClassDefFoundError: org/rogach/avalanche/ErrorHandler$

If that try/catch block runs for smaller amounts of time, everything works as expected.

If anyone is interested, here is the codebase in question: Avalanche

I need to note that I saw this and similar problems only on Cent OS 5 machines, using JRE 6u26 and Scala 2.9.1 / 2.9.2.

What could be the cause of this problem?

Answer 1

If you ran out of memory trying to initialize a class, do you suppose you'd see the OutOfMemory or the NoClassDef?

  //from initialize_impl
  if (NULL == message) {
    // Out of memory: can't create detailed error message
    THROW_MSG(vmSymbols::java_lang_NoClassDefFoundError(), className);

It's possible your code threw OOM, then couldn't load your exception handler object.

There are, of course, other possible transient conditions: network was down and the class is on a networked drive; or you cleaned your classes directory during the test. Another possibility is that you built your app on a case-insensitive file system and are testing on a case-sensitive file system with an anomalously named class file. For example, if you change object "handler" to "Handler" without deleting *.class, you'll still see "Handler.class". (But I suspect the error message detail would include the name conflict; unless you're OOM, of course.)

I haven't had a chance to try to break AbstractFileClassLoader; my previous speculation follows:

It's worth explaining that Avalanche is a build tool, and you run an instance of scalac to compile the build.scala to an in-memory classfile that is loaded with scalac's AbstractFileClassLoader, where "AbstractFile" is the abstraction. Since whatever build.scala mucks with the tool configuration, obviously it's essential that AFCL respect classloader delegation. That seems true, but I notice that it does not delegate first to parent on getResourceAsStream (as a quick test confirmed). The suspicious thing is that findClass uses classBytes, which on failure calls super, which is the nice ScalaClassLoader, but which uses getResourceAsStream to load Foo.class. So calling findClass could return a class from the parent CL (which is wrong), though that may be moot if the parent is known to have failed already. Because it's the middle of my night, I can't draw a conclusion, but I would want to nail that down if my build tool relied on this behavior.

I don't know what's in your build.scala (or av.scala) that runs for a day, but perhaps you've got Avalanche (re-)loaded by a misbehaving child classloader, then when it throws, the CL can't findClass your ErrorHandler.