JBoss CPU utilization issue

Question

I am using JBoss AS 4.2.3 along with the seam framework. My CPU usage increases as the number of users increase and it hits 99% for just 80 users. We also use Hibernate, EJB3 and Apache with mod_jk for loadbalancing.

When I took the thread dump all the runnable threads are doing a same activity with the following trace:

at java.net.SocketInputStream.socketRead0(Native Method) 
at java.net.SocketInputStream.read(SocketInputStream.java:129) 
at org.apache.coyote.ajp.AjpProcessor.read(AjpProcessor.java:1012) 
at org.apache.coyote.ajp.AjpProcessor.readMessage(AjpProcessor.java:1091) 
at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:384) 
at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:366) 
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:446) 
at java.lang.Thread.run(Thread.java:662)

I am not able to interpret this with the stack trace. Also I find that even when the users have logged out, the CPU utilization still continues to be the same with threads in the same state.

Answer 1

These threads are attempting to read from a Socket connection. In this case they are waiting for the next request to be sent to the server from mod_jk in Apache. This is quite normal and they probably are not the reason for your CPU usage.

At this point you really need to go and run your application through a profiler.

If you are unable to run a profiler on the system (i.e. it's a production box) the next best thing is to start to take many stack dumps each a couple of seconds apart and then go though them by hand matching up the thread IDs. You need to look for the threads that are running your code and don't seem to have changed between dumps.

It is a very tedious task and doesn't always get clear results, but without a profiler or some sort of instrumentation you won't be able to find where all that CPU is going.