Springboot embedded Tomcat classloader slowness

Question

I have built a web application that uses SpringBoot v1.3.6.RELEASE Tomcat 8.0.36 Java 1.8u101 on CentOS 7.2

The web application is also a SOAP client that calls out to another web application.(JAX-WS RI 2.2.9) If the applications remains idle for 15 seconds the first webservice call stalls for nearly 2 seconds. It appears that the stall happens in o.a.c.loader.WebappClassLoaderBase.

After idle 15 seconds

16:02:36.165 : Delegating to parent classloader org.springframework.boot.loader.LaunchedURLClassLoader@45283ce2

16:02:36.170 : Searching local repositories

16:02:36.170 : findResource(META-INF/services/javax.xml.soap.MetaFactory)

16:02:38.533 : --> Resource not found, returning null

16:02:38.533 : --> Resource not found, returning null

Next request no idle time

16:07:09.981 : Delegating to parent classloader org.springframework.boot.loader.LaunchedURLClassLoader@45283ce2

16:07:09.984 : Searching local repositories

16:07:09.985 : findResource(META-INF/services/javax.xml.soap.MetaFactory)

16:07:09.986 : --> Resource not found, returning null

16:07:09.986 : --> Resource not found, returning null

16:07:09.988 : findResources(META-INF/services

All above messages produced by o.a.c.loader.WebappClassLoaderBase and they are apparently being caused by ClientSOAPHandlerTube.processRequest which is from JAX-WS RI.

You'll notice the first call takes over 2 seconds but subsequent calls take only milliseconds. I'm wondering if anyone has experienced this behavior?

Possible solutions: Is it possible to change out the classloader used by tomcat in springboot to use ParallelWebappClassLoader

Or maybe this is a product of the reloadable flag on the classloader but I don't see how to change that flag in springboot.

When run using Jetty as the container this does not occur.

Final Solution: (thanks to Gergely Bacso)

    @Bean
public EmbeddedServletContainerCustomizer servletContainerCustomizer() {
    return new EmbeddedServletContainerCustomizer() {

        @Override
        public void customize(ConfigurableEmbeddedServletContainer container) {
             if (container instanceof TomcatEmbeddedServletContainerFactory) {
                customizeTomcat((TomcatEmbeddedServletContainerFactory) container);
            }
        }
        private void customizeTomcat(TomcatEmbeddedServletContainerFactory tomcatEmbeddedServletContainerFactory) {
            tomcatEmbeddedServletContainerFactory.addContextCustomizers(new TomcatContextCustomizer() {
                @Override
                public void customize(Context cntxt) {
                    cntxt.setReloadable(false);
                }
            });
        }
    };
}

Answer 1

Actually your findings are quite good and you have 90% answered your question already. These two facts:

1. "it appears that the stall happens in o.a.c.loader.WebappClassLoaderBase"
2. "when run using Jetty as the container this does not occur."

show that it is going be a Tomcat-related problem because:

1. o.a.c. stands for org.apache.catalina
2. Your code works well on another container. (Jetty)

You also observed, that the issue is happening after 15 seconds of idle time. This perfectly corresponds to Tomcat's default checkInterval setting, which is:

The number of seconds between checks for modified classes and resources, if reloadable has been set to true. The default is 15 seconds.

So in short: currently your reloadable flag is ON, and Tomcat tries to reload your classes which is handy during development, but unacceptable in any other case. The way to switch it off is not via Spring-boot though.

SOLUTION:
You need to locate your context.xml / server.xml where you will find your Context defined like this:

<Context ... reloadable="true">

Remove the reloadable flag, and you have solved the problem. The file itself can be either in $CATALINA_BASE/conf of $CATALINE_HOME/conf, but in reality these locations can be a bit tricky to find if you are using some IDE to manage Tomcat for you.

In case of embedded Tomcat with Spring-boot:

The class you can use to manipulate Tomcat settings is: EmbeddedServletContainerCustomizer.

Through this you can add a TomcatContextCustomizer (addContextCustomizers) so that you can call setReloadable on the context itself.

I do not see any reason for Spring-boot needing this flag on true.