Menu
I am trying to investigate why my jboss is taking up 600% CPU (running on CentOS 5).
I run top -H and try to compare the result with jstack output. Most of the thread IDs that come from top appear in the output of jstack but unfortunately the threads that take up most of the CPU are not listed there.
Because the output is very long I am only listing part of it here.
$top -b -n1 -H -p <jboss pid>
top - 08:04:58 up 73 days, 19:50, 1 user, load average: 5.72, 5.97, 5.95
Tasks: 128 total, 6 running, 122 sleeping, 0 stopped, 0 zombie
Cpu(s): 63.6%us, 1.7%sy, 0.0%ni, 34.2%id, 0.3%wa, 0.0%hi, 0.2%si, 0.0%st
Mem: 32946160k total, 32545228k used, 400932k free, 86776k buffers
Swap: 34996216k total, 136k used, 34996080k free, 21577176k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
12908 jboss 16 0 9488m 8.6g 18m S 71.5 27.4 146:45.68 java
12903 jboss 16 0 9488m 8.6g 18m S 69.5 27.4 146:36.08 java
12904 jboss 16 0 9488m 8.6g 18m S 69.5 27.4 146:39.63 java
12905 jboss 16 0 9488m 8.6g 18m S 69.5 27.4 146:44.46 java
12906 jboss 16 0 9488m 8.6g 18m S 69.5 27.4 146:53.94 java
12907 jboss 16 0 9488m 8.6g 18m S 69.5 27.4 146:39.18 java
12914 jboss 16 0 9488m 8.6g 18m R 27.8 27.4 36:26.64 java
12901 jboss 25 0 9488m 8.6g 18m S 0.0 27.4 0:00.00 java
12902 jboss 25 0 9488m 8.6g 18m S 0.0 27.4 0:00.05 java
12909 jboss 23 0 9488m 8.6g 18m S 0.0 27.4 76:34.09 java
12910 jboss 23 0 9488m 8.6g 18m S 0.0 27.4 76:33.49 java
12911 jboss 15 0 9488m 8.6g 18m S 0.0 27.4 257:10.35 java
12915 jboss 15 0 9488m 8.6g 18m S 0.0 27.4 0:06.63 java
12916 jboss 15 0 9488m 8.6g 18m S 0.0 27.4 0:19.61 java
more lines....
The jstack outpout is as follows
# /usr/java/jdk1.6.0_33/bin/jstack -F 3362
Deadlock Detection:
No deadlocks found.
Thread 13055: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=485 (Interpreted frame)
- org.apache.tomcat.util.net.JIoEndpoint$Worker.await() @bci=8, line=415 (Interpreted frame)
- org.apache.tomcat.util.net.JIoEndpoint$Worker.run() @bci=11, line=441 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
Thread 12996: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=485 (Interpreted frame)
- org.apache.tomcat.util.net.JIoEndpoint$Worker.await() @bci=8, line=415 (Interpreted frame)
- org.apache.tomcat.util.net.JIoEndpoint$Worker.run() @bci=11, line=441 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
Thread 12994: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=485 (Interpreted frame)
- org.apache.tomcat.util.net.JIoEndpoint$Worker.await() @bci=8, line=415 (Interpreted frame)
- org.apache.tomcat.util.net.JIoEndpoint$Worker.run() @bci=11, line=441 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
more stacks....
The thread IDs in the output of jstack also appear in the output of top but the thread IDs from top that are most CPU intensive (1903-1908) do not appear in the output of jstack.
I thought these can be GC threads but I can't find anything to prove this.
Any idea why I don't see all the threads in jstack or what I can do to see the missing threads?
712
Try to run jstack under the jboss owner. This way you'll get more threads as an output. By issuing
jstack -l 12345
I got
"Service Thread" #9 daemon prio=9 os_prio=0 tid=0x00007f976c0d3000 nid=0x1f56a runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
note the nid is the hex representation of thread id got from top -H
200
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
