What is taking long on Jetty startup?

Question

Starting Jetty there's a long delay (8s) before my web application starts loading

13:50:10 [INFO] jetty-9.4.5.v20170502
13:50:18 [INFO] Scanning elapsed time=146ms

With debug logging turned on there are two interesting steps

• Extracting dependent war-application, which takes time after all (3s)

  10:03:13 [DEBUG] Extracting entry = null from jar file:[..]/application-1.0.war
  10:03:16 [DEBUG] Unpacked overlay: jar:file:[..]/application-1.0.war!/ to file:[..]
  

• and the below 4s delay:

  10:03:16 [DEBUG] Service loaders found in 0ms
  10:03:20 [DEBUG] loaded interface javax.servlet.ServletContainerInitializer
  

How can I debug or affect what's causing above 4s delay?

---

Configuration

pom.xml

<dependency>
    <groupId>com.company</groupId>
    <artifactId>application</artifactId>
    <version>1.0</version>
    <type>war</type>
</dependency>

[...]

<plugin>
    <groupId>org.eclipse.jetty</groupId>
    <artifactId>jetty-maven-plugin</artifactId>
    <version>9.4.5.v20170502</version>
    <configuration>
        <webApp>
            <webInfIncludeJarPattern>empty</webInfIncludeJarPattern>
            <containerIncludeJarPattern>empty</containerIncludeJarPattern>
        </webApp>
    </configuration>
</plugin>

web.xml

<web-app xmlns="http://xmlns.jcp.org/xml/ns/javaee"
     xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
     xsi:schemaLocation="http://xmlns.jcp.org/xml/ns/javaee http://xmlns.jcp.org/xml/ns/javaee/web-app_3_1.xsd" 
     version="3.1" 
     metadata-complete="true">

   <context-param>
       <param-name>contextClass</param-name>
       <param-value>org.springframework.web.context.support.AnnotationConfigWebApplicationContext</param-value>
   </context-param>

   <context-param>
       <param-name>contextConfigLocation</param-name>
       <param-value>com.app.AnnotationConfig</param-value>
   </context-param>

   <listener>
       <listener-class>org.springframework.web.context.ContextLoaderListener</listener-class>
   </listener>

   [...]

</web-app>

Answer 1

You have a bigger then average webapp (70MB of WEB-INF/lib jars), but nothing colossally big (I've seen 800MB war files)

There's a few things that occur that could be cause for slow down.

1. Unpacking the war into the WebApp temp directory
2. Unpacking resource jars (WEB-INF/lib/*.jar!/META-INF/resources/) into the WebApp temp directory
3. Jar bytecode scanning (for annotations and types declared in @HandlesType annotations on javax.servlet.ServletContainerInitializer classes)

If your filesystem is slow, then any of the above can slow you down.

Note: the DEBUG logging of Jetty will tell you the timing of each of those (even breaking down the timing of bytecode scanning timing to the individual jar)

The bytecode scanning step is the most often place where startup times take a hit.

Configuring the <containerIncludeJarPattern> for "empty" is not recommended, that is necessary for Servlets, JSPs, Taglibs to function.

The containerIncludeJarPattern default is only the servlet / jsp / taglib jars anyway. (which take microseconds to scan)

The <webInfIncludeJarPattern> should also not be just "empty", it should at a minimum include your WEB-INF/classes content (aka .*/classes/.*). Consider setting it up to only scan those WEB-INF/lib jars that you need. (something like .*/lib/spring-.*\.jar$|.*/classes/.*)

The size in bytes of the classes in your WEB-INF/lib/*.jar and WEB-INF/classes is actually irrelevant. What's more relevant to timing is the number of files found (even non-class files).

If you use resource jars (WEB-INF/lib/*.jar!/META-INF/resources/) then this is a significant penalty/source of your slow startup.

What you can do:

Start with looking at your DEBUG logs, that will tell you where things are taking their times.

Next, investigate using the quickstart features of Jetty if startup timing is important (this has 2 parts, a build-time component that scans and builds a jetty-quickstart.xml that is included in your war, and a runtime module that looks for and uses that jetty-quickstart.xml if found)

Finally, If you are using resource jars (WEB-INF/lib/*.jar!/META-INF/resources/), consider moving those contents out of WEB-INF/lib and into the normal places in your war during your build (package phase in maven). These are convenient, but have a bunch of side effects that you don't appear to like. (also consider the issue of conflicting resource resolution at runtime).