Tomcat stopped without any log or any stack

Question

We have trouble with Tomcat 5.5 which stops at night on our production servers (Linux CentOS 4.8) and we have no idea why it stops... There is no Tomcat's log in catalina.out or any application's log.

We tried different things to find why the server stops:

• configure Tomcat to be able to generate a core dump
• instrument System.exit() method with javassist to find if the method was called
• add a shutdown hook to the JVM (with Runtime.getRuntime().addShutdownHook())

None of them worked, we have no core dump, the Exit method and the shutdown hook are not called. My conclusions are:

• The VM is not terminated properly but crash without any log.

Any idea or log to read to find why Tomcat stops?

Answer 1

1) Make sure you know where stderr is redirected and check if anything got printed there.

2) Check the memory limits on Tomcat and how much free memory does the system have. Review the Linux system logs under /var/log to see if anything suspicious happened during the time. For example, kernel can randomly kill a process (almost) without a trace if the system is running low on memory.

We've ran 5.5 in production for years and never had any unexplained shutdowns, FWIW.