Tomcat sporadic latency

Question

I have been micro-optimizing our page response time on Tomcat and in almost all cases I will see a response time of 50ms if I refresh over and over again but if the page is not hit for a second or two the response time jumps back up to 500ms.

I have seen this same behavior regardless of local, not local, APR, NIO, JIO, static or dynamic responses (ie serving static file or handing the response dynamically). So far I have yet to see this behavior not happen on Tomcat (that is consistent sub 400ms regardless of frequency).

I have used Visual VM also to see if there were any clues.

I thought it was some sort of keep alive but when I run Apache Bench I get even faster (sub 50ms) response times (obviously because its hitting it frequently).

So how do you keep a low latency not frequently hit URL in Tomcat? Perhaps this question is better for ServerFault?

UPDATE: I'm almost positive its a Tomcat 6 issue. I thought I had tested on Tomcat 7 but I just tested again on it and had no issue (see results below). Even the latest Tomcat 6 still has this problem.

Here is the ab output for Tomcat 6 (notice the max):

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       0
Processing:    14   39  45.2     30     314
Waiting:       14   38  45.2     30     314
Total:         14   39  45.2     30     314

Here is ab output for Tomcat 7 notice the max:

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       0
Processing:    25   38   8.8     37      67
Waiting:       25   37   8.7     36      66
Total:         25   38   8.8     37      67

The Tomcat versions are the only difference (same machine, same JDK, etc...). I thought for sure the latest Tomcat 6 would be fine but it has similar latency on first request.

Answer 1

Peeking at the tomcat code I decided to search on the word "Weak" on the theory that your problem is that something in a weak reference is collected when you don't re-request quickly.

I came up with the following guess... I found this interesting Class:

http://svn.apache.org/viewvc/tomcat/tc6.0.x/trunk/java/javax/el/BeanELResolver.java?revision=1027187&view=markup

It seems to maintain a cache of BeanProperties objects, a portion of which is handled by a WeakHashMap, and whenever the cache fills up, everything is put in a WeakHashMap and can be garbage collected. If items in the weak map are requested, they are put back in the main map (which is not weak). If your page trips this behavior right at the end of your processing (by causing something like the size of the cache in BeanProperties to be added you could wind up throwing away almost all of the cached bean descriptions.

Conveniently there is a property for tuning this:

private static final String CACHE_SIZE_PROP =
    "org.apache.el.BeanELResolver.CACHE_SIZE";

So maybe try playing with that and see if it influences the behavior? This might not be it however since I didn't see a big change (quick look) in this class in Tomcat 7 where you say your problem disappears. (have you been tweaking this property in your prior tuning efforts?)