Why does Tomcat start so much slower with shared libraries?

Question

The problem

we have a Tomcat 7.0.35 running with 25 webapps that all share the same libraries with the same versions (it is guaranteed that this will never change!). I figured instead of each project loading the same library, I could just move them to ${catalina.home}/lib which leads to the desired reduction of memory load.

However this comes at a price: The time for each webapp to be deployed increased from 5 seconds to 2 minutes, which is not viable!

I did some reading (e.g. Class Loader HOW-TO), unfortunately I was unable to find a way to decrease the startup time. Is there a way to solve this issue?

Requested profiling information

I ran jstack and jvisualvm on the two different scenarios, I will present some findings here (please let me know if you need additional information):

Libraries that are specific to each project seem to be loaded with: org.apache.catalina.loader.WebappClassLoader.findResourceInternal
jstack output for the thread running the method:

"localhost-startStop-1" daemon prio=10 tid=0x00007f17f8001800 nid=0x13b8 runnable [0x00007f183c800000]    java.lang.Thread.State: RUNNABLE         at java.text.MessageFormat.subformat(MessageFormat.java:1250)         at java.text.MessageFormat.format(MessageFormat.java:819)         at org.apache.naming.StringManager.getString(StringManager.java:145)         at org.apache.naming.resources.BaseDirContext.lookup(BaseDirContext.java:500)         at org.apache.naming.resources.ProxyDirContext.lookup(ProxyDirContext.java:310)         at org.apache.catalina.loader.WebappClassLoader.findResourceInternal(WebappClassLoader.java:3011)         at org.apache.catalina.loader.WebappClassLoader.findResource(WebappClassLoader.java:1262)         at org.apache.catalina.loader.WebappClassLoader.getResourceAsStream(WebappClassLoader.java:1499)         at org.apache.catalina.startup.ContextConfig.populateJavaClassCache(ContextConfig.java:2302)         at org.apache.catalina.startup.ContextConfig.populateJavaClassCache(ContextConfig.java:2294)         at org.apache.catalina.startup.ContextConfig.populateJavaClassCache(ContextConfig.java:2308)         at org.apache.catalina.startup.ContextConfig.populateJavaClassCache(ContextConfig.java:2291)         at org.apache.catalina.startup.ContextConfig.checkHandlesTypes(ContextConfig.java:2197)         at org.apache.catalina.startup.ContextConfig.processAnnotationsStream(ContextConfig.java:2154)         at org.apache.catalina.startup.ContextConfig.processAnnotationsJar(ContextConfig.java:2034)         at org.apache.catalina.startup.ContextConfig.processAnnotationsUrl(ContextConfig.java:1990)         at org.apache.catalina.startup.ContextConfig.processAnnotations(ContextConfig.java:1976)         at org.apache.catalina.startup.ContextConfig.webConfig(ContextConfig.java:1326)         at org.apache.catalina.startup.ContextConfig.configureStart(ContextConfig.java:882)         - locked <0x00000007f52f3ec0> (a org.apache.catalina.startup.ContextConfig)         at org.apache.catalina.startup.ContextConfig.lifecycleEvent(ContextConfig.java:379)         at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:119)         at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:90)         at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5343)         - locked <0x00000007f52f3bb8> (a org.apache.catalina.core.StandardContext)         at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)         - locked <0x00000007f52f3bb8> (a org.apache.catalina.core.StandardContext)         at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:902)         at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:879)         at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:636)         at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1236)         at org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:1877)         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)     Locked ownable synchronizers:         - <0x00000007846dfc70> (a java.util.concurrent.ThreadPoolExecutor$Worker) 

Libraries that are shared between projects seem to be loaded with:
at org.apache.catalina.loader.WebappClassLoader.getResourceAsStream which causes the incredible deployment times. JVisualVM displays 98%+ "Self time" for this method during project deployment.
jstack output for the thread running the method:

"localhost-startStop-1" daemon prio=10 tid=0x00007f6280001000 nid=0x6ba3 runnable [0x00007f62c4950000]    java.lang.Thread.State: RUNNABLE         at java.lang.String.intern(Native Method)         at java.util.jar.Attributes$Name.<init>(Attributes.java:466)         at java.util.jar.Attributes.putValue(Attributes.java:168)         at java.util.jar.Attributes.read(Attributes.java:421)         at java.util.jar.Manifest.read(Manifest.java:251)         at sun.security.util.SignatureFileVerifier.processImpl(SignatureFileVerifier.java:252)         at sun.security.util.SignatureFileVerifier.process(SignatureFileVerifier.java:239)         at java.util.jar.JarVerifier.processEntry(JarVerifier.java:307)         at java.util.jar.JarVerifier.update(JarVerifier.java:218)         at java.util.jar.JarFile.initializeVerifier(JarFile.java:345)         at java.util.jar.JarFile.getInputStream(JarFile.java:412)         - locked <0x00000007e8e9d890> (a sun.net.www.protocol.jar.URLJarFile)         at sun.net.www.protocol.jar.JarURLConnection.getInputStream(JarURLConnection.java:162)         at java.net.URLClassLoader.getResourceAsStream(URLClassLoader.java:233)         at org.apache.catalina.loader.WebappClassLoader.getResourceAsStream(WebappClassLoader.java:1528)         at org.apache.catalina.startup.ContextConfig.populateJavaClassCache(ContextConfig.java:2302)         at org.apache.catalina.startup.ContextConfig.populateJavaClassCache(ContextConfig.java:2294)         at org.apache.catalina.startup.ContextConfig.populateJavaClassCache(ContextConfig.java:2308)         at org.apache.catalina.startup.ContextConfig.populateJavaClassCache(ContextConfig.java:2294)         at org.apache.catalina.startup.ContextConfig.checkHandlesTypes(ContextConfig.java:2197)         at org.apache.catalina.startup.ContextConfig.processAnnotationsStream(ContextConfig.java:2154)         at org.apache.catalina.startup.ContextConfig.processAnnotationsJar(ContextConfig.java:2034)         at org.apache.catalina.startup.ContextConfig.processAnnotationsUrl(ContextConfig.java:1990)         at org.apache.catalina.startup.ContextConfig.processAnnotations(ContextConfig.java:1976)         at org.apache.catalina.startup.ContextConfig.webConfig(ContextConfig.java:1326)         at org.apache.catalina.startup.ContextConfig.configureStart(ContextConfig.java:882)         - locked <0x00000007858db6e0> (a org.apache.catalina.startup.ContextConfig)         at org.apache.catalina.startup.ContextConfig.lifecycleEvent(ContextConfig.java:379)         at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:118)         at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:91)         at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5346)         - locked <0x00000007858db318> (a org.apache.catalina.core.StandardContext)         at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:153)         - locked <0x00000007858db318> (a org.apache.catalina.core.StandardContext)         at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:902)         at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:879)         at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:636)         at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1236)         at org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:1877)         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)     Locked ownable synchronizers:         - <0x000000078460e618> (a java.util.concurrent.ThreadPoolExecutor$Worker) 

Answer 1

The WebappClassLoader searches on his own classpath first. So for each class not in the local classpath, all jar files will be looked into. Only then it will look at the delegate to the common classloader.

Say that looking into all of the war's jars takes x amount of time, for the jars its parent classloader it takes y amount of time.

On average loading a class from the correct classloader will take half of the time it takes to scan them everything.

Loading a class from the WebappClassLoader will take x/2 time.

Loading a class from the parent classloader will take x + y/2.

Depending on x and y you could change the order in which the classloaders are used. If y is larger than x, you should keep thee default setup, if x is larger than you can try to reverse the lookup order. If you do so loading a class from the common classloader will take y/2 time but a class from the WebappClassLoader will take y + x/2.

The WebbappClassLoader has a delegate property for this setting it to true should do the trick. See Common Attributes