On my busiest production installation, on occasion I get a single thread that seems to get stuck in an infinite loop. I've not managed to figure out who is the culprit, after much research and debugging, but it seems like it should be possible. Here are the gory details:
Current debugging notes:
1) ps -eL 18975 shows me the the Linux pid the problem child thread, 19269
$ps -eL | grep 18975
...
PID LWP TTY TIME CMD
18975 18994 ? 00:00:05 java
18975 19268 ? 00:00:00 java
18975 19269 ? 05:16:49 java
18975 19271 ? 00:01:22 java
18975 19273 ? 00:00:00 java
...
2) jstack -l 18975 says there are no deadlocks, jstack -m 18975 does not work
3) jstack -l 18975 does give me the stack trace for all my threads (~400). Example thread stack (and not the problem):
"http-342.877.573.944-8080-360" daemon prio=10 tid=0x0000002adaba9c00 nid=0x754c in Object.wait() [0x00000000595bc000..0x00000000595bccb0] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on (a org.apache.tomcat.util.net.JIoEndpoint$Worker) at java.lang.Object.wait(Object.java:485) at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:416) - locked (a org.apache.tomcat.util.net.JIoEndpoint$Worker) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:442) at java.lang.Thread.run(Thread.java:619)
4) The ps -eL output's thread ID does not match the output from jstack, or at least I cannot see it. (jstack documentation is a bit sparse.)
5) There are no heavy IO, memory usage or other corresponding activity clues to work with.
Platform:
Does anybody know how I can make that connection from the linux ps output to my problem child java thread? So close, yet so far...
You can use Thread. currentThread(). getStackTrace() . That returns an array of StackTraceElement s that represent the current stack trace of a program.
Ctrl + Break (Windows) In Windows operating systems, we can capture a thread dump using the CTRL and Break key combination. To take a thread dump, navigate to the console used to launch the Java application, and press the CTRL and Break keys together.
currentThread() is a static method , returns a reference to the currently executing thread object. After getting the reference of the current thread , we will call the getId() method on Thread class. getId() method in Java. getId() will return the unique identifier of current thread which is a positive long value.
A stack trace is a user-friendly snapshot of the threads and monitors in a Virtual Machine for the Java platform (Java Virtual Machine or JVM machine). A thread dump shows what every thread in a JVM is doing at a given time and is useful in debugging.
It looks like the nid in the jstack output is the Linux LWP id.
"http-342.877.573.944-8080-360" daemon prio=10 tid=0x0000002adaba9c00 nid=0x754c in Object.wait() [0x00000000595bc000..0x00000000595bccb0]
Convert the nid to decimal and you have the LWP id. In your case 0x754c is 30028. This process is not shown in our ps output, but it was probably one of the LWPs you have omitted to save space.
Here's a little a Perl snippet you can use to pipe the output of jstack to:
#!/usr/bin/perl -w
while (<>) {
if (/nid=(0x[[:xdigit:]]+)/) {
$lwp = hex($1);
s/nid=/lwp=$lwp nid=/;
}
print;
}
You can use JConsole to view the thread's stack trace.
If your using JDK 1.6.0_07 or above, you can also use visualvm.
Both tools provide a nice view of all the running threads in an application. The visualvm is quite a bit nicer, but hopefully seeing all the threads can help you track down the run-away thread.
Check for threads that are always in a state of RUNNING. When we had a run-away thread, the stack trace would constantly change. So we were able to tell which methods the loop was calling, and track down the loop.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With