Tomcat 7 shuts down abruptly with JDK 7

Question

In our development environment, we have 2 Tomcat 7.0.22 nodes on Cent OS 5.5 handled by F5 load balancer with sticky sessions. We recently upgraded to JDK 7 and have started observing random tomcat shutdowns. Multiple wars are deployed from common shared loader directory. The pattern during shutdown is as follows

• In majority cases, both nodes shutdown one after the other. In one scenario, one tomcat died while other was still up.
• catalina logs indicate shutdown was graceful after printing "INFO: Stopping service Catalina"
• We couldn't find any System.exit calls
• No indication of OOM
• We added a shutdown hook to our spring container and the thread dumps didn't raise any suspicious thread.
• No watchdog service being run in background
• Observed one script under etc/lograte.d that rotated logs
• HTTP 1.1 connector for 8080, redirect 8443
• AJP for 8009, redirect 8443

I have run out of ideas to debug this issue. Basically, I'm looking out for a hook or debugger which lets me know the exact process or thread which has invoked shutdown. This should be possible since its a graceful shutdown and Tomcat handled it.

We will mostly revert to JDK 1.6 and test for few days. Till then, any help or pointers will be appreciated

Thanks in advance!

Answer 1

Since you mention logrotate, have you tried manually running logrotate [ /usr/sbin/logrotate /etc/logrotate.conf ] and watching to see if it is responsible for the shutdown?