ThreadDump issues when almost all threads are in NATIVE state

Question

I took thread dump of a host in my application where most of the threads are in (state = IN_NATIVE) state.

Only couple of threads were blocked at these instances:

Thread 31681: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Interpreted frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=1987 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Compiled frame)
 - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)

and

Thread 20966: (state = BLOCKED)
 - java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
 - com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run() @bci=34, line=534 (Compiled frame)

and

Thread 31682: (state = BLOCKED)
 - java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
 - java.lang.ref.ReferenceQueue.remove(long) @bci=44, line=118 (Interpreted frame)
 - java.lang.ref.ReferenceQueue.remove() @bci=2, line=134 (Interpreted frame)
 - org.apache.commons.httpclient.JakartaHttpConnectionManager$ReferenceQueueThread.run() @bci=10, line=1104 (Interpreted frame)

We make some HttpConnections and requests were timing out after a configurable time. When I restarted the host, everything was back to normal.

So, I assume that thread should have been blocked at some point.

Threads running in NATIVE code were pointing here :

         Thread 29587: (state = IN_NATIVE)
     - java.net.SocketInputStream.socketRead0(java.io.FileDescriptor, byte[], int, int, int) @bci=0 (Compiled frame; information may be imprecise)
     - java.net.SocketInputStream.read(byte[], int, int) @bci=84, line=129 (Compiled frame)
     - java.io.BufferedInputStream.fill() @bci=175, line=218 (Compiled frame)
     - java.io.BufferedInputStream.read1(byte[], int, int) @bci=44, line=258 (Compiled frame)
     - java.io.BufferedInputStream.read(byte[], int, int) @bci=49, line=317 (Compiled frame)
     - sun.net.www.MeteredStream.read(byte[], int, int) @bci=16, line=116 (Compiled frame)
     - java.io.FilterInputStream.read(byte[], int, int) @bci=7, line=116 (Compiled frame)
     - sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(byte[], int, int) @bci=4, line=2672 (Compiled frame)
     - javax.imageio.stream.FileCacheImageInputStream.readUntil(long) @bci=64, line=121 (Compiled frame)
     - javax.imageio.stream.FileCacheImageInputStream.read(byte[], int, int) @bci=69, line=167 (Compiled frame)
     - com.sun.imageio.plugins.jpeg.JPEGImageReader.readImageHeader(long, boolean, boolean) @bci=0 (Compiled frame)
     - com.sun.imageio.plugins.jpeg.JPEGImageReader.readNativeHeader(boolean) @bci=12, line=532 (Compiled frame)
     - com.sun.imageio.plugins.jpeg.JPEGImageReader.checkTablesOnly() @bci=92, line=277 (Compiled frame)
     - com.sun.imageio.plugins.jpeg.JPEGImageReader.gotoImage(int) @bci=41, line=409 (Compiled frame)
     - com.sun.imageio.plugins.jpeg.JPEGImageReader.readHeader(int, boolean) @bci=2, line=525 (Compiled frame)
     - com.sun.imageio.plugins.jpeg.JPEGImageReader.readInternal(int, javax.imageio.ImageReadParam, boolean) @bci=3, line=968 (Compiled frame)
     - com.sun.imageio.plugins.jpeg.JPEGImageReader.read(int, javax.imageio.ImageReadParam) @bci=8, line=948 (Compiled frame)
     - javax.imageio.ImageIO.read(javax.imageio.stream.ImageInputStream) @bci=55, line=1422 (Compiled frame)
     - javax.imageio.ImageIO.read(java.net.URL) @bci=42, line=1374 (Compiled frame)
..............
.............    
 - java.util.concurrent.Executors$RunnableAdapter.call() @bci=4, line=441 (Interpreted frame)
     - java.util.concurrent.FutureTask$Sync.innerRun() @bci=30, line=303 (Interpreted frame)
     - java.util.concurrent.FutureTask.run() @bci=4, line=138 (Interpreted frame)
     - java.util.concurrent.ThreadPoolExecutor$Worker.runTask(java.lang.Runnable) @bci=59, line=886 (Interpreted frame)
     - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=28, line=908 (Interpreted frame)
     - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)

which I assume is the culprit. We internally use ImageIO and Jsoup libraries which make these calls.

I don't have much idea about reading thread dumps and I am new to it. But any help would be appreciated.

Is it possible that all natively running threads are causing problems? How should I proceed with it?

Thanks,

Answer 1

The first three thread dumps are idle threads waiting for a new task to perform. You can ignore them.

I suspect the last one is waiting on IO, rather than consuming CPU but I don't know for sure.