Analyzing thread dump of a java process

Question

I have Java EE based application running on tomcat and I am seeing that all of a sudden the application hangs after running for couple of hours.

I collected the thread dump from the application just before it hangs and put it in TDA for analysis:

TDA (Thread Dump Analyzer) gives the following message for the above monitor:

A lot of threads are waiting for this monitor to become available again.
This might indicate a congestion. You also should analyze other locks 
blocked by threads waiting for this monitor as there might be much more 
threads waiting for it.

And here is the stacktrace of the thread highlighted above:

"MY_THREAD" prio=10 tid=0x00007f97f1918800 nid=0x776a 
             waiting for monitor entry [0x00007f9819560000]
   java.lang.Thread.State: BLOCKED (on object monitor)
    at java.util.Hashtable.get(Hashtable.java:356)
    - locked <0x0000000680038b68> (a java.util.Properties)
    at java.util.Properties.getProperty(Properties.java:951)
    at java.lang.System.getProperty(System.java:709)
    at com.MyClass.myMethod(MyClass.java:344)

I want to know what does the "waiting for monitor entry" state means? And also would appreciate any pointers to help me debug this issue.

Answer 1

One of your threads acquired a monitor object (an exclusive lock on a object). That means the thread is executing synchronized code and for whatever reason stuck there, possibly waiting for other threads. But the other threads cannot continue their execution because they encountered a synchronized block and asked for a lock (monitor object), however they cannot get it until it is released by other thread. So... probably deadlock.

Answer 2

Please look for this string from the whole thread dump

- locked <0x00007f9819560000>

If you can find it, the thread is deadlock with thread "tid=0x00007f97f1918800"