Java appears hung

Question

I've been using the Java Service wrapper in a custom application for quite a while and it's been working fine. Since updating our application to a new version in the last few days the JVM started hanging and then wrapper prints this in the log: JVM appears hung: Timed out waiting for signal from JVM.

It then automatically terminates the JVM and starts the app again. This happens after about 10 hours of running, which just makes it harder to debug.

Of course I am going to look through the changes that we've made, but no major changes were made that I would suspect is causing this type of problem.

Where can I look to try and figure out what is happening? Debug messages from the application don't indicate anything interesting. If the JVM just crashes it will usually create a dump, which can help in debugging it, but it's hanging, so it's not creating a dump. If I make it not restart the service automatically is there anything I can do to get some useful information out of the JVM before restarting it?

It seems to me that the JVM shouldn't hang from typical programming errors. What have you run into before that can cause the JVM to hang?

Answer 1

Read up on the wrapper.ping.timeout property. The wrapper software communicates with your JVM every so often to make sure that it is alive. If that communication fails for whatever reason, the wrapper deems the process hung and attempts to restart it.

Depending on how your application is architected, your JVM might be busy processing something else when the wrapper tries to "ping" it.

Answer 2

See if you can use the Visual VM to see what is going on. Have the Visual VM monitor the app for the whole time and when it stops working perhaps you can determine what is wrong.

If the VM hangs you can get the state of the threads... I think the Visual VM will make it a bit easier given your setup than the usual ctrl-break (or whatver the key combo is).

(Edit based on comment)

Tried this. Last time it hung the number of threads and the amount of memory in use were quite low, so neither of those are causing the problem. Unfortunately after it hangs and wrapper terminates it you can't get a thread dump.

Is there any way you can run it without the wrapper to debug it? Also if you use the NetBeans profiler it might give you a chance to deal with it when it stops (I'll check later today and see if I can find out if that would behave differently).

Answer 3

What environment are you in? OS, JVM version, hardware architecture?

That does sound like a bug, and given that it takes many hours, it sounds like a resource-exhaustion bug of some sort.

Answer 4

I had a couple different versions of a library on the classpath (JBPM). With wrapper you can use wildcards to include jars. Be careful with this though as you may accidentally include more than you should.

Here is an IBM article that gives information on debugging hangs in Java. It basically says that there are two things that can cause hangs:

1. An infinite loop,
2. A deadlock.

Since then I've had to debug other hanging issues. On linux you can send the JVM the QUIT signal to make it do a thread dump to the console. This really helps figuring out where the issue is. Use this command to do that: kill -QUIT

Edit 6/13/2017

These days I use jmap included in the JDK to dump the entire memory of the program. Then I use Eclipse Memory Analyzer to see the exact state of the program when it crashed. You can look at the list of threads that are active and then inspect the variables in each stack frame.

/usr/java/latest/bin/jmap -dump:file=/tmp/app-crash.hprof <PID>

Where PID is the process ID of the java process.