Tomcat Guice/JDBC Memory Leak

Question

I'm experiencing a memory leak due to orphaned threads in Tomcat. Particularly, it seems that Guice and the JDBC driver are not closing threads.

Aug 8, 2012 4:09:19 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: A web application appears to have started a thread named [com.google.inject.internal.util.$Finalizer] but has failed to stop it. This is very likely to create a memory leak.
Aug 8, 2012 4:09:19 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: A web application appears to have started a thread named [Abandoned connection cleanup thread] but has failed to stop it. This is very likely to create a memory leak.

I know this is similar to other questions (such as this one), but in my case, the answer of "don't worry about it" won't be sufficient, as it is causing problems for me. I have CI server which regularly updates this application, and after 6-10 reloads, the CI server will hang because Tomcat is out of memory.

I need to be able to clear up these orphaned threads so I can run my CI server more reliably. Any help would be appreciated!

Answer 1

I just dealt with this problem myself. Contrary to some other answers, I do not recommend issuing the t.stop() command. This method has been deprecated, and for good reason. Reference Oracle's reasons for doing this.

However there is a solution for removing this error without needing to resort to t.stop()...

You can use most of the code @Oso provided, just replace the following section

Set<Thread> threadSet = Thread.getAllStackTraces().keySet();
Thread[] threadArray = threadSet.toArray(new Thread[threadSet.size()]);
for(Thread t:threadArray) {
    if(t.getName().contains("Abandoned connection cleanup thread")) {
        synchronized(t) {
            t.stop(); //don't complain, it works
        }
    }
}

Replace it using the following method provided by the MySQL driver:

try {
    AbandonedConnectionCleanupThread.shutdown();
} catch (InterruptedException e) {
    logger.warn("SEVERE problem cleaning up: " + e.getMessage());
    e.printStackTrace();
}

This should properly shutdown the thread, and the error should go away.

Answer 2

I've had the same issue, and as Jeff says, the "don't worry about it approach" was not the way to go.

I did a ServletContextListener that stops the hung thread when the context is being closed, and then registered such ContextListener on the web.xml file.

I already know that stopping a thread is not an elegant way to deal with them, but otherwise the server keeps on crashing after two or three deploys (it is not always possible to restart the app server).

The class I created is:

public class ContextFinalizer implements ServletContextListener {

    private static final Logger LOGGER = LoggerFactory.getLogger(ContextFinalizer.class);

    @Override
    public void contextInitialized(ServletContextEvent sce) {
    }

    @Override
    public void contextDestroyed(ServletContextEvent sce) {
        Enumeration<Driver> drivers = DriverManager.getDrivers();
        Driver d = null;
        while(drivers.hasMoreElements()) {
            try {
                d = drivers.nextElement();
                DriverManager.deregisterDriver(d);
                LOGGER.warn(String.format("Driver %s deregistered", d));
            } catch (SQLException ex) {
                LOGGER.warn(String.format("Error deregistering driver %s", d), ex);
            }
        }
        Set<Thread> threadSet = Thread.getAllStackTraces().keySet();
        Thread[] threadArray = threadSet.toArray(new Thread[threadSet.size()]);
        for(Thread t:threadArray) {
            if(t.getName().contains("Abandoned connection cleanup thread")) {
                synchronized(t) {
                    t.stop(); //don't complain, it works
                }
            }
        }
    }

}

After creating the class, then register it on the web.xml file:

<web-app...
    <listener>
        <listener-class>path.to.ContextFinalizer</listener-class>
    </listener>
</web-app>

Answer 3

The least invasive workaround is to force initialisation of the MySQL JDBC driver from code outside of the webapp's classloader.

In tomcat/conf/server.xml, modify (inside the Server element):

<Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener" />

to

<Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener"
          classesToInitialize="com.mysql.jdbc.NonRegisteringDriver" />

• With mysql-connector-java-8.0.x use com.mysql.cj.jdbc.NonRegisteringDriver instead

This assumes you put the MySQL JDBC driver into tomcat's lib directory and not inside your webapp.war's WEB-INF/lib directory, as the whole point is to load the driver before and independently of your webapp.

References:

• http://bugs.mysql.com/bug.php?id=68556#c400606

• http://tomcat.apache.org/tomcat-7.0-doc/config/listeners.html#JRE_Memory_Leak_Prevention_Listener_-_org.apache.catalina.core.JreMemoryLeakPreventionListener

• http://markmail.org/message/dmvlkps7lbgpngil

• com.mysql.jdbc.NonRegisteringDriver source v5.1

• com.mysql.cj.jdbc.NonRegisteringDriver source v8.0
• Changes in connector/J v8.0