Apache Avro ThreadLocal objects linger around on Tomcat un-deploy

Question

We are using Apache Avro as a JSON interface between our Python application and some third party Java libraries that we run within a Tomcat service. We decided to simply extend the org.apache.avro.ipc.ResponderServlet class to implement our own servlet. The servlet is really simple, in that it instantiates the ResponderServlet super-class in the constructor, and overrides the init() and destroy() methods to do some house-keeping for the third party libraries that we run in the servlet.

When Tomcat un-deploys our webapp however, we see a number of SEVERE errors warning of ThreadLocal-related memory leaks.

SEVERE: The web application [/hotwire] created a ThreadLocal with key of type [org.apache.avro.Schema$3] (value [org.apache.avro.Schema$3@4464784f]) and a value of type [java.lang.Boolean] (value [true]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Jan 24, 2013 2:19:36 AM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/hotwire] created a ThreadLocal with key of type [org.apache.avro.generic.GenericDatumReader$1] (value [org.apache.avro.generic.GenericDatumReader$1@2016ad9d]) and a value of type [org.apache.avro.util.WeakIdentityHashMap] (value [org.apache.avro.util.WeakIdentityHashMap@30e02ee0]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

We are probably doing something naive somewhere because we were unable to find any help anywhere on the web for this scenario. Nevertheless, we're hoping someone here can tell us where we're going wrong.

Here's a glimpse into our servlet.

public class HotWire extends ResponderServlet{
    public HotWire() throws IOException
    {
              super(new SpecificResponder(Engine.class, new EngineImpl()));
    }

    @Override
        public void init() {
        try {
         super.init();
                 try {
                      init_engine();                
                      } catch (EngineInitException e) {
                      e.printStackTrace();
                      }
            } catch (ServletException e) {
                 e.printStackTrace();
            }
        }

        @Override
        public void destroy() {
            super.destroy();
            shutdown_engine();
        }

        public static class EngineImpl implements EngineInterface  {
            public Boolean create(Parameters message) {
                Boolean status = null;
                try {
                    status = engine.create_object(message);
                } catch (SemanticException | IOException e) {
                    e.printStackTrace();
                }
                return status;
            }

}

Answer 1

I am seeing a similar issue. If you look here there is a static ThreadLocal cache that gets filled up with no way of removing it:

https://github.com/apache/avro/blob/master/lang/java/avro/src/main/java/org/apache/avro/generic/GenericDatumReader.java#L106

  private static final ThreadLocal<Map<Schema, Map<Schema, ResolvingDecoder>>> RESOLVER_CACHE = ThreadLocal
      .withInitial(WeakIdentityHashMap::new);

In my case each request has a different schema so my Spring Boot service eventually runs out of memory and dies.

This is also an example of a classloader leak: ThreadLocal & Memory Leak

I can see a JIRA request here for an improvement: https://issues.apache.org/jira/browse/AVRO-1595

Update: I was able to work around the issue with the memory leak. There was an ExecutorService that was re-using threads. Now I shutdown the service which causes the threads to be finalized. This allows for the ThreadLocal memory to be GC'd.

As someone pointed out the RESOLVER_CACHE uses a WeakIdentityHashMap that should allow for the cache to be GC'd but this was not happening for me.