Multiple threads stuck in native calls (Java)

Question

I have a problem with an application running on Fedora Core 6 with JDK 1.5.0_08.

After some amount of uptime (usually some days) threads begin getting stuck in native methods.

The threads are locked in something like this:

"pool-2-thread-2571" prio=1 tid=0x08dd0b28 nid=0x319e waiting for monitor entry [0xb91fe000..0xb91ff7d4]
at java.lang.Class.getDeclaredConstructors0(Native Method)

or

"pool-2-thread-2547" prio=1 tid=0x75641620 nid=0x1745 waiting for monitor entry [0xbc7fe000..0xbc7ff554]
at sun.misc.Unsafe.defineClass(Native Method)

Especially puzzling to me is this one:

"HealthMonitor-10" daemon prio=1 tid=0x0868d1c0 nid=0x2b72 waiting for monitor entry [0xbe5ff000..0xbe5ff4d4]
at java.lang.Thread.dumpThreads(Native Method)
at java.lang.Thread.getStackTrace(Thread.java:1383)

The threads remain stuck until the VM is restarted.

Can anyone give me an idea as to what is happening here, what might be causing the native methods to block? The monitor entry address range at the top of each stuck thread is different. How can I figure out what is holding this monitor?

Answer 1

Can you find out which thread is actually synchronizing on the monitor on which the native method is waiting? At least the thread-dump you get from the VM when you send it a SIGQUIT (kill -3) should show this information, as in

"Thread-0" prio=5 tid=0x0100b060 nid=0x84c000 waiting for monitor entry [0xb0c8a000..0xb0c8ad90]
    at Deadlock$1.run(Deadlock.java:8)
    - waiting to lock <0x255e5b38> (a java.lang.Object)
...
"main" prio=5 tid=0x01001350 nid=0xb0801000 waiting on condition [0xb07ff000..0xb0800148]
    at java.lang.Thread.sleep(Native Method)
    at Deadlock.main(Deadlock.java:21)
- locked <0x255e5b38> (a java.lang.Object)

In the dumps you've posted so far, I can't see any thread that is actually waiting to lock a specific monitor...

Answer 2

I was having similar problems a few months ago and found the jthread(?) utility to be invaluable. You give it the process ID for your Java application and it will dump the entire stack for each thread in your process.

From the output of jthread, I could see one thread was trying to obtain a lock after having entered a monitor and another thread was trying to enter the monitor after obtaining the lock. A recipe for deadlock.

I was also wondering if your application was running into a garbage collection issue. You say it runs for a couple days before it stops like this. How long have you let it sit in the stuck state to see if maybe the GC ever finishes?