Could not load org.apache.hadoop.util.ShutdownHookManager when shutdown tomcat server

Question

I have a simple web application run on tomcat. This web application read and write file to HDFS.

The issue I am facing is each time when I stop the server by using ./bin/shutdown.sh. I am getting could not load hadoop shutdownHookManager exception.

I am sure the hadoop-common (contains ShutDownManager) is in tomcat classpath.

Can anyone help me out?

Exception I got:

Oct 14, 2013 5:57:54 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [] created a ThreadLocal with key of type [com.ebay.kernel.calwrapper.CalTransactionHelper$1] (value [com.ebay.kernel.calwrapper.CalTransactionHelper$1@3d5a1c5f]) and a value of type [com.ebay.kernel.calwrapper.CalTransactionHelper.Stack] (value [[]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Oct 14, 2013 5:57:54 PM org.apache.coyote.AbstractProtocol stop 
INFO: Stopping ProtocolHandler ["http-bio-8080"]
Oct 14, 2013 5:57:54 PM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["ajp-bio-8009"]
Oct 14, 2013 5:57:54 PM org.apache.coyote.AbstractProtocol destroy
INFO: Destroying ProtocolHandler ["http-bio-8080"]
Oct 14, 2013 5:57:54 PM org.apache.coyote.AbstractProtocol destroy
INFO: Destroying ProtocolHandler ["ajp-bio-8009"]
Oct 14, 2013 5:57:54 PM org.apache.catalina.loader.WebappClassLoader loadClass
INFO: Illegal access: this web application instance has been stopped already.  Could not load org.apache.hadoop.util.ShutdownHookManager$2.  The eventual following stack trace is caused by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no functional impact.
java.lang.IllegalStateException
  at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1600)
  at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1559)
  at org.apache.hadoop.util.ShutdownHookManager.getShutdownHooksInOrder(ShutdownHookManager.java:124)
  at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:52)
Exception in thread "Thread-9" java.lang.NoClassDefFoundError: org/apache/hadoop/util/ShutdownHookManager$2
  at org.apache.hadoop.util.ShutdownHookManager.getShutdownHooksInOrder(ShutdownHookManager.java:124)
  at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:52)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.util.ShutdownHookManager$2
  at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1714)
  at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1559)
  ... 2 more

Answer 1

I have the same problem in a maven Mojo I'm building. I think that the problem is caused by the context class loader that loads hadoop classes in your web application. These definitions are not visible by the WebAppClassLoader class loader. When JVM shutdown occurs, shutdown hooks are being executed and that calls the ShutdownHookManager but it was loaded in now destroyed context class loader, and there you have the problem.

Putting hadoop jars in the lib directory of your tomcat will probably help. Or setting some *_CLASSPATH java option when you're starting tomcat. That way the WebAppClassLoader will have definitions of your classes.

Otherwise, you can just execute hadoop's shutdown hooks manually before you destroy your context and ignore the above exception.