Java Tomcat, underlying Netty threads not stopped

Question

I'm trying to use a log4j appender to send logs to GrayLog2 (log4j2-gelf). So I add my dependency to my pom.xml configure the log4j2.xml to configure my appender. Build the whole thing with maven and deploy it to a local tomcat, and everything works fine. The problem happen when I try to shutdown the tomcat. Tomcat does not stop if I don't kill -9 PID the tomcat and the catalina.out ends up with this : Complete stack trace

What got my attention is the

java.lang.NoClassDefFoundError: io/netty/util/concurrent/DefaultPromise$1

First I thought that I may have another dependency importing netty that would cause this exception. But I've looking for some time and was not able to find any other netty import. Then I saw the :

Caused by: java.lang.ClassNotFoundException: Illegal access: this web application instance has been stopped already. Could not load [io.netty.buffer.PoolArena$1]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.

And this is where my question is coming. My understanding is that the gelf-appender is using Netty as a way to communicate asynchronously with the GrayLog server. But it's a server itself. So would it be possible that the Tomcat is stopping the parent servlet before killing the underlying Netty ? Which would lead to this problem ? I've trying to find some documentation on the way that Tomcat shutsdown, but so far I did not find anything, more than that I could perfectly be wrong concerning this diagnostic...

Answer 1

It looks like you have a ClassLoader problem

Caused by: java.lang.ClassNotFoundException: Illegal access: this web application instance has been stopped already. Could not load [io.netty.buffer.PoolArena$1]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access. at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForClassLoading(WebappClassLoaderBase.java:1342)

WebappClassLoader stops serving incoming request to load, when it receives shutdown signal. You need Netty jar be accessible beyond WebappClassLoader. Put it to $CATALINA_HOME/lib (see Tomcat classloader howto, section Common). To find the right jar, run mvn dependency:tree for your webapp (if you use maven)