Thread dump showing Runnable state, but its hung for quite a long time

Question

We are facing an unusual problem in our application, in the last one month our application reached an unrecoverable state, It was recovered post application restart.

Background : Our application makes a DB query to fetch some information and this Database is hosted on a separate node.

Problematic case : When the thread dump was analyzed we see all the threads are in runnable state fetching the data from the database, but it didn't finished even after 20 minutes.

Post the application restart as expected all threads recovered. And the CPU usage was also normal.

Below is the thread dump

ThreadPool:2:47" prio=3 tid=0x0000000007334000 nid=0x5f runnable [0xfffffd7fe9f54000] java.lang.Thread.State: RUNNABLE at oracle.jdbc.driver.T2CStatement.t2cParseExecuteDescribe(Native Method) at oracle.jdbc.driver.T2CPreparedStatement.executeForDescribe(T2CPreparedStatement.java:518) at oracle.jdbc.driver.T2CPreparedStatement.executeForRows(T2CPreparedStatement.java:764) at ora

All threads in the same state.

Questions:

1. what could be the reason for this state?
2. how to recover under this case ?

Answer 1

It's probably waiting for network data from the database server. Java threads waiting (blocked) on I/O are described by the JVM as being in the state RUNNABLE even though from the program's point of view they're blocked.

Answer 2

1. Does your code manually handle transaction? If then, maybe some of the code didn't commit() after changing data. Or maybe someone ran data modification query directly through PLSQL or something and didn't commit, and that leads all reading operation to be hung.

2. When you experienced that "hung" and DB has recovered from the status, did you check the data if some of them were rolled back? Asking this since you said "It was recovered post application restart.". It's happening when JDBC driver changed stuff but didn't commit, and timeout happened... DB operation will be rolled back. ( can be different based on the configuration though )