Performance - Spring Boot - Server Response Time

Question

I have a strange behaviour on our spring boot application:

• Frontend/Client - angular 6
• Backend - spring boot - spring MVC - embedded tomcat - Linux

After restarting the backend the first calls to a controller needs about 5 seconds, the following same request only takes 50ms. This is reproducible in 90% of the cases, sometimes even the first call is fast.

I am sure, the problem is on the server not on the client. On the browser, I see the TTFB time (time to the first byte) is increasing to 5 seconds. The following requests only need 10ms for TTFB.

With monitoring tools on the server (app dynamics) i can collect such slow server calls and on the call graph i can see that:

org.apache.catalina.webresources.JarWarResourceSet:getArchiveEntries:117

needs 4916ms. So here is my bottleneck, I think. But I don't know how to fix it.

What i have already tryed:

• Switched from hikaricp to apache tomcat jdbc connection pool
• Ugraded spring boot from 2.0.0 to 2.0.5
• Upgraded java to 1.8.0_181
• Propertie spring.jpa.tomcat.testOnBorrow = true
• Propertie spring.jpa.tomcat.validationQuery = select 1

Everything without influence to the server delay.

Update

The time is lost cause the war file is scanned multiple times.

org.apache.catalina.webresources.CachedResource.validateResource is checking if we have a war file (isPackedWarFile) and this check return false. Even though it is a war file. For this misbehave I have a workaround. I set tomcat.resource.cache-tt to a high value.

But now org.apache.catalina.webresources.Cache.getResource has a noCache method. And in this method class and jar files are excluded from caching. And this is the reason why the war file is scanned again.

Scanning the whole war file takes roughly 5 seconds. And this break is a stop the world break. And this scanning is absolutely needless cause the war file is not exploded and so its contents can not be changed.

Update

If I put the war file into a tomcat installation everything is fast. The embedded tomcat is the problem.