Tomcat shutdown VERY slow after calling shutdown.sh?

Question

My situation is like this:

Everytime before uploading war file to web-app folder, I stop Tomcat by calling sh shutdown.sh. It used to take about 30 seconds for a total shutdown. But now it doesn't work well anymore.

Actually, it did some work, because when I access the application from web-page it throws 503 error (Under Maintenance). But when I use ps aux | grep tomcat to check, the tomcat process is still there. And it will be there for around 5 - 10 mins.

I understand that it may need to take extra times to complete all the tasks, but it is way too slow (5 - 10 minutes), before it is stop totally. I don't understand why this happens, but there must be some reason. Maybe there's something to do with the code, or the new script of deployment we used recently. I just have almost no clue about where to check.

This is important to our team because we are using "auto-deployment", in which we use a script to auto-package war file, uploading and deploy on a specific time. If we started a new tomcat instance before the old one successfully shutdown, it will hang there for eternal, and cleaning up task by "kill -9" is daunting.

Is there anyone who has experimented this issue? Any clue would be appreciated.

Answer 1

Hoàng Long -

Thank you for the update.

1) The fact that you see your Quartz jobs running, and the error message, are both significant:

SEVERE: The web application [/project] appears to have started a thread named [Resource Destroyer in BasicResourcePool.close()] but has failed to stop it. This is very likely to create a memory leak.

2) One suggestion is configuration:

http://forum.springsource.org/showthread.php?17833-Spring-Quartz-Tomcat-no-shutdown

I had the same problem. I fixed it by adding destroy-method="destroy" to the SchedulerFactoryBean definition. This way spring closes down the scheduler when the application is stopped.

3) Another suggestion is to add a shutdown listener:

http://forums.terracotta.org/forums/posts/list/15/4341.page

Using a context listener and introducing a timeout on shutdown solves the issue for me. I just wait a second after shutting down:

  public void contextDestroyed(ServletContextEvent sce) {
    try {
      factory.getScheduler().shutdown();
      Thread.sleep(1000);

Answer 2

If this is something that mystically started to happen within the last few days, perhaps you're running into the Linux leap second bug? For more information, see

https://serverfault.com/questions/403732/anyone-else-experiencing-high-rates-of-linux-server-crashes-during-a-leap-second

https://access.redhat.com/knowledge/articles/15145

http://pedroalves-bi.blogspot.fi/2012/07/java-leap-second-bug-how-to-fix-your.html