Classloader leak because of invalid SSL certificate in SolrJ, HttpClient, JVM or my application?

Question

I've analyzed a classloader leak the last few days in a large application and I've worked out the problem.

My application uses SolrJ which will be initialilzed through a @Bean-Method:

@Bean(destroyMethod = "close")
public SolrClient solrClient() {
    return new HttpSolrClient(SOLR_URL);
}

SolrJ (org.apache.solr:solr-solrj:5.4.1) uses the Apache HttpClient (org.apache.httpcomponents:httpclient:4.4.1). The HttpClient initializes the SSL context by using normal java classes like javax.net.ssl.SSLSocketFactory. In this way java loads the trustManager and analyzes all trusted certificates. If there is an error the certificate (an instance of sun.security.x509.X509CertImpl) is stored in a list and gets enriched by the thrown exception. This exception is swallowed and my application remains unaware.

As far as I can see, the SSL Context is in the System / Root Classloader, my Application is in the dedicated WebappClassLoader and this is the problem because now there is an IOException inside of the SSL context which contains references in the stacktrace, backtrace and so on to classes in my application.

But now I don't know where this came from. Is it the SolrJ client, the Apache HttpClient, Java itself (the JVM) or is it my application?

I have made a small application to reproduce the problem which you can find here: https://github.com/CptS/solrj-classloader-leak This also contains a Workaround (a shutdown hook which removes the references which leads to the classloader leak).

If you disable the shutdown hook (e.g. by commenting it out) and start a clean Tomcat (see "Environment to reproduce" below) you can reproduce it by following this steps:

1. deploy the war of the demo project (A)
2. reload it (B)
3. reload it again (C)
4. Trigger GC (D)
5. undeploy
6. Trigger GC (E)
7. See that the metaspace gets not completly cleaned up (F)

I've create a heap dump and the shortest path to GC looks like this:

This was the same as in my large application. The mentioned Workaround (a little bit inspired by https://github.com/mjiderhamn/classloader-leak-prevention, but this unfortunately doesn't solve my problem) searches by using reflection for these unparseableExtensions and removes the Exception stored in the why field through this way: SSLContextImpl.DefaultSSLContext#defaultImpl -> SSLContextImpl#trustManager -> X509TrustManager#trustedCerts -> X509CertImpl#info -> X509CertInfo#extensions -> CertificateExtensions#unparseableExtensions -> UnparseableExtension#why

By doing this i've got the stacktrace of the exception if it helps someone:

java.io.IOException: No data available in passed DER encoded value.
    at sun.security.x509.GeneralNames.<init>(GeneralNames.java:61)
    at sun.security.x509.IssuerAlternativeNameExtension.<init>(IssuerAlternativeNameExtension.java:136)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at sun.security.x509.CertificateExtensions.parseExtension(CertificateExtensions.java:113)
    at sun.security.x509.CertificateExtensions.init(CertificateExtensions.java:88)
    at sun.security.x509.CertificateExtensions.<init>(CertificateExtensions.java:78)
    at sun.security.x509.X509CertInfo.parse(X509CertInfo.java:702)
    at sun.security.x509.X509CertInfo.<init>(X509CertInfo.java:167)
    at sun.security.x509.X509CertImpl.parse(X509CertImpl.java:1804)
    at sun.security.x509.X509CertImpl.<init>(X509CertImpl.java:195)
    at sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:100)
    at java.security.cert.CertificateFactory.generateCertificate(CertificateFactory.java:339)
    at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:755)
    at sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:56)
    at sun.security.provider.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:224)
    at sun.security.provider.JavaKeyStore$DualFormatJKS.engineLoad(JavaKeyStore.java:70)
    at java.security.KeyStore.load(KeyStore.java:1445)
    at sun.security.ssl.TrustManagerFactoryImpl.getCacertsKeyStore(TrustManagerFactoryImpl.java:226)
    at sun.security.ssl.SSLContextImpl$DefaultSSLContext.getDefaultTrustManager(SSLContextImpl.java:767)
    at sun.security.ssl.SSLContextImpl$DefaultSSLContext.<init>(SSLContextImpl.java:733)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at java.security.Provider$Service.newInstance(Provider.java:1595)
    at sun.security.jca.GetInstance.getInstance(GetInstance.java:236)
    at sun.security.jca.GetInstance.getInstance(GetInstance.java:164)
    at javax.net.ssl.SSLContext.getInstance(SSLContext.java:156)
    at javax.net.ssl.SSLContext.getDefault(SSLContext.java:96)
    at javax.net.ssl.SSLSocketFactory.getDefault(SSLSocketFactory.java:122)
    at org.apache.http.conn.ssl.SSLSocketFactory.getSystemSocketFactory(SSLSocketFactory.java:190)
    at org.apache.http.impl.conn.SchemeRegistryFactory.createSystemDefault(SchemeRegistryFactory.java:85)
    at org.apache.http.impl.client.SystemDefaultHttpClient.createClientConnectionManager(SystemDefaultHttpClient.java:121)
    at org.apache.http.impl.client.AbstractHttpClient.getConnectionManager(AbstractHttpClient.java:484)
    at org.apache.solr.client.solrj.impl.HttpClientUtil.setMaxConnections(HttpClientUtil.java:234)
    at org.apache.solr.client.solrj.impl.HttpClientConfigurer.configure(HttpClientConfigurer.java:40)
    at org.apache.solr.client.solrj.impl.HttpClientUtil.configureClient(HttpClientUtil.java:149)
    at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:125)
    at org.apache.solr.client.solrj.impl.HttpSolrClient.<init>(HttpSolrClient.java:189)
    at org.apache.solr.client.solrj.impl.HttpSolrClient.<init>(HttpSolrClient.java:162)
    at de.test.spring.SolrJConfig.solrClient(SolrJConfig.java:20)
    at de.test.spring.SolrJConfig$$EnhancerBySpringCGLIB$$dbd4362f.CGLIB$solrClient$0(<generated>)
    at de.test.spring.SolrJConfig$$EnhancerBySpringCGLIB$$dbd4362f$$FastClassBySpringCGLIB$$8e7566a6.invoke(<generated>)
    at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
    at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:309)
    at de.test.spring.SolrJConfig$$EnhancerBySpringCGLIB$$dbd4362f.solrClient(<generated>)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:162)
    at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:588)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1119)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1014)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:504)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:476)
    at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:303)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:299)
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:755)
    at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:757)
    at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:480)
    at de.test.WicketApplication.init(WicketApplication.java:32)
    at org.apache.wicket.Application.initApplication(Application.java:950)
    at org.apache.wicket.protocol.http.WicketFilter.init(WicketFilter.java:429)
    at org.apache.wicket.protocol.http.WicketFilter.init(WicketFilter.java:353)
    at org.apache.catalina.core.ApplicationFilterConfig.initFilter(ApplicationFilterConfig.java:279)
    at org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:260)
    at org.apache.catalina.core.ApplicationFilterConfig.<init>(ApplicationFilterConfig.java:105)
    at org.apache.catalina.core.StandardContext.filterStart(StandardContext.java:4640)
    at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5247)
    at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
    at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:724)
    at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:700)
    at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:714)
    at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:919)
    at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1703)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

My workaround solves the problem for now, but of course this is just a workaround.

I want to know and maybe someone can answer one or more of my question:

1. Is this a "bug" in SolrJ, HttpClient, Java or my application?
2. If it is my application, what I am doing wrong?
3. If it is not my application, is it a known problem? I can't find any information about this. (Where) should I create a bug ticket?
4. Why is there a "invalid" certificate? (BTW: Maybe the leak would also be solve if I remove this certificate from the trust store... I haven't tested but I think a invalid or damaged certificate should never lead to a classloader leak...)
5. Has anyone some more information on this? I can't belive that I'm the only one who detect this behaviour (except it is my appication... see my question 2).

At last but not least, my Environment to reproduce:

• Tomcat Version: Apache Tomcat/8.0.14 (Debian)
• JVM Version: 1.8.0_91-b14
• JVM Vendor: Oracle Corporation
• OS Name: Linux
• OS Version: 3.16.0-4-amd64
• Architecture: amd64

Answer 1

It's a bug in java, the bug ticket is here: http://bugs.java.com/bugdatabase/view_bug.do?bug_id=8168069

Many thanks to mjiderhamn (on GitHub). He is the developer of the great classloader-leak-prevention library and has now included a preventor for this issue (version 2.1.0).