Tomcat process time is small but nginx shows it is big

Question

I have five tomcat instances behind nginx.

Sometimes the nginx upstream_response_time is very big, more than 1 second, while the tomcat local access log shows the process time is only 50ms(I use %D to log process time).

What is the possible reason and how to fix it? It does not seems the network is slow since other applications run fast.

Update:

Seems the nginx upstream_response_time = %D + 1 sec.

Answer 1

My hypothesis for your observation are packet losses. This seems the most probable problem to me since you are saying that it happens when they are many requests. To test this, you can monitor the traffic e.g. with tcpdump or iftop. If you are under Ubuntu, you can install and start the latter with

sudo apt-get install iftop
sudo iftop

There are many other network monitoring solutions in Linux, the amazing Wireshark works for all operating systems.

One reason for package losses can be collisions, you can check that with a ifconfig [interface] if you are under Linux:

me@mymahine:~$ ifconfig eth1
eth1      Link encap:Ethernet  HWaddrf f:41:8d:ef:41:8d  
          BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 <-------------------------- check here ---
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

Are the Tomcats and the Nginx on the same physical (or virtual) machine?

Further reading

• http://mailman.nginx.org/pipermail/nginx/2008-October/008025.html