javax.xml.bind.DatatypeConverter leaking class loaders?

Question

I am currently investigating some class loader leaks of an application on Tomcat 7 (w/ Oracle JDK 7). One class that keeps a static reference to the web application class loader (and thus causes the class loader not to be released on redeploy/restart) is javax.xml.bind.DatatypeConverter, which lives in the system class loader and keeps a static reference via its theConverter field to com.sun.xml.bind.DatatypeConverterImpl from Sun's jaxb-impl package.

Has anyone ever observed this issue before? Any suggestions (except for using reflection to null the static field on application shutdown)?

Answer 1

Tomcat 8 issue warning on subsequent web application redeploy during development:

org.apache.catalina.loader.WebappClassLoaderBase checkThreadLocalMapForLeaks

SEVERE: The web application [rsnetlombard] created a ThreadLocal with key of type
[com.sun.xml.bind.v2.ClassFactory$1] (value [com.sun.xml.bind.v2.ClassFactory$1@79eb7926])
and a value of type [java.util.WeakHashMap]
(value [{class javax.xml.bind.annotation.W3CDomHandler=java.lang.ref.WeakReference@525eec52}])
but failed to remove it when the web application was stopped.
Threads are going to be renewed over time to try and avoid a probable memory leak.

I make heap dump within VisualVM and open it.

VisualVM find destroyed web application class loader in OQL tab by query:

select x from org.apache.catalina.loader.WebappClassLoader x where x.state.name.toString() == "DESTROYED"

Visiting pointed link to object in "instalce" tab allow call "Find nearest GC root" in "reference section" and copy textual representation to clipboard::

this     - value: org.apache.catalina.loader.WebappClassLoader #3
 <- <classLoader>     - class: com.sun.xml.bind.DatatypeConverterImpl, value: org.apache.catalina.loader.WebappClassLoader #3
  <- <class>     - class: com.sun.xml.bind.DatatypeConverterImpl, value: com.sun.xml.bind.DatatypeConverterImpl class DatatypeConverterImpl
   <- theConverter (sticky class)     - class: javax.xml.bind.DatatypeConverter, value: com.sun.xml.bind.DatatypeConverterImpl #1

javax.xml.bind.DatatypeConverter is from Java SE and that class loaded by system classloader (and so marked (sticky class)). But point to class that loaded by web application classloader.

Googling about com.sun.xml.bind.DatatypeConverterImpl leads to this SO post.

Supplied solution say that com.sun.jersey:jersey-json package request JAXB API implementation from com.sun.xml.bind:jaxb-impl package::

$ mvn dependency:tree
...
[INFO] +- com.sun.jersey:jersey-json:jar:1.8:compile
[INFO] |  +- org.codehaus.jettison:jettison:jar:1.1:compile
[INFO] |  |  \- stax:stax-api:jar:1.0.1:compile
[INFO] |  +- com.sun.xml.bind:jaxb-impl:jar:2.2.3-1:compile
[INFO] |  |  \- javax.xml.bind:jaxb-api:jar:2.2.2:compile
[INFO] |  |     \- javax.xml.stream:stax-api:jar:1.0-2:compile
[INFO] |  +- org.codehaus.jackson:jackson-core-asl:jar:1.7.1:compile
[INFO] |  +- org.codehaus.jackson:jackson-mapper-asl:jar:1.7.1:compile
[INFO] |  +- org.codehaus.jackson:jackson-jaxrs:jar:1.7.1:compile
[INFO] |  \- org.codehaus.jackson:jackson-xc:jar:1.7.1:compile

Because Java 7 comes with own JAXB implementation (JAXB RI by the fact) we don't need com.sun.xml.bind:jaxb-impl package. Add exclude to corresponding part of pom.xml::

    <dependency>
        <groupId>com.sun.jersey</groupId>
        <artifactId>jersey-json</artifactId>
        <version>${jersey.version}</version>
        <exclusions>
            <exclusion>
                <groupId>com.sun.xml.bind</groupId>
                <artifactId>jaxb-impl</artifactId>
            </exclusion>
        </exclusions>
    </dependency>

In order reach results quicker in test I reduce Tomcat memory::

JAVA_OPTS="-Djava.awt.headless=true -Xmx212m -XX:+UseConcMarkSweepGC -XX:MaxPermSize=66m"

Redeploing/using application 10 times give none for::

select x from org.apache.catalina.loader.WebappClassLoader x where x.state.name.toString() == "DESTROYED"

Under redeploys "Visual GC" plug-in shown PermGen cleanups.

Running with previous development setup require something::

JAVA_OPTS="-Djava.awt.headless=true -Xmx512m -XX:+UseConcMarkSweepGC -XX:MaxPermSize=256m"

to survive 4-5 redeploys. OQL query for larger PermGen give several Tomcat's WebappClassLoader but examining instances shown that there are no path to GC and they are cleaned when PermGen become full.

Answer 2

As it turned out, one of my dependencies (com.sun.jersey:jersey-json) pulled in com.sun.xml.bind:jaxb-impl, which was responsible for the System Classloader -> Application Classloader reference. Excluding that dependency solved the issue (as JDK 7 comes with a sensible JAXB implementation, which will be referenced within the System CL, which is fine).