Java thread dump: WAITING (on object monitor) - what is it waiting on?

Question

there was a similar question asked java-thread-dump-waiting-on-object-monitor-line-not-followed-by-waiting-on, but there was no concrete answer, so I will ask my question in hopes to get more info...

In the following thread dump I see that the thread is in the "WAITING (on object monitor)" state - but there is no line with "waiting on " that would indicate what it is waiting for. How do I interpret this thread stack and find out why (and what resource) this thread is waiting on?

"eventTaskExecutor-50" prio=10 tid=0x0000000004117000 nid=0xd8dd in Object.wait() [0x00007f8f457ad000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:503)
at com.tibco.tibjms.TibjmsxLink.sendRequest(TibjmsxLink.java:359)
- locked <0x00007f98cbebe5d8> (a com.tibco.tibjms.TibjmsxResponse)
at com.tibco.tibjms.TibjmsxSessionImp._confirmTransacted(TibjmsxSessionImp.java:2934)
at com.tibco.tibjms.TibjmsxSessionImp._confirm(TibjmsxSessionImp.java:3333)
- locked <0x00007f90101399b8> (a java.lang.Object)
at com.tibco.tibjms.TibjmsxSessionImp._commit(TibjmsxSessionImp.java:2666)
at com.tibco.tibjms.TibjmsxSessionImp.commit(TibjmsxSessionImp.java:4516)
at org.springframework.jms.support.JmsUtils.commitIfNecessary(JmsUtils.java:217)
at org.springframework.jms.listener.AbstractMessageListenerContainer.commitIfNecessary(AbstractMessageListenerContainer.java:577)
at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:482)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:325)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:263)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1102)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:996)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)

Locked ownable synchronizers:
- <0x00007f901011ca88> (a java.util.concurrent.ThreadPoolExecutor$Worker)

This thread is one of the listener threads configured to accept messages from the Tibco bus.

thanks!

Marina

Answer 1

It's a peculiarity of HotSpot JVM. When dumping a stack, JVM recovers the wait object from the method local variables. This info is available for interpreted methods, but not for compiled native wrappers.

When Object.wait is executed frequently enough, it gets JIT-compiled.
After that there will be no "waiting on" line in a thread dump.

1. Since wait() must be called on a synchronized object, most often the wait object is the last locked object in the stack trace. In your case it is

   - locked <0x00007f98cbebe5d8> (a com.tibco.tibjms.TibjmsxResponse)
   

2. To prevent Object.wait from being JIT-compiled (and thus making wait info always available) use the following JVM option

   -XX:CompileCommand="exclude,java/lang/Object.wait" 
   

Answer 2

This thread is waiting the notification from another thread (thread name is TCPLinkReader, if you look over the full thread dump you should be able to find it) which is created by the TIBCO EMS client library.

The stacktrace shows that the Spring application is trying to commit a session. To commit the session EMS client needs to send some data to server and waiting for the confirmation from server that the session is committed successfully or not.

TCPLinkReader thread is a dedicated thread that EMS client use to receive downstream (from server to client) TCP packets.

If you are seeing this thread lasts for long, there are 2 scenarios:

• Something wrong on the EMS server side, possibly hung

• there are some defects in the client library that caused deadlock, so server did send the response back but the TCPLinkReader thread did not notify the caller thread.

Last, post the full thread dump if problem persists.