Profiling Java: find out where threads spend time blocked

Question

I have a multi-threaded application which scales well to begin with, but running on a 16-cpu server, once I exceed 5 or 6 hardware threads the performance levels off. I suspect that the bottleneck surrounds one of the synchronized methods. However, I need to be sure it's the guilty method before I start diving into the code and trying to replace the algorithm with a non-blocking one.

Running Java with the -Xprof argument tells me that, as I expected, the threads are spending most of their time blocked. Is there a way that I can break that down into how much time they spend blocked at a particular method?

Answer 1

http://yourkit.com the monitor view will tell you which lock classes are hot, who is holding the contended locks and breakdown by lock instance and caller stack. There is 30 day evaluation period of the tool.

Answer 2

The jvisualvm tool that comes with the JDK can help you a little bit, although its CPU profiling information is rather limited (more of a visualizer for Xprof data). I generally find it more useful for memory profiling.

JProfiler has a pretty nice CPU profiler with some really cool functionality that might help you, but it's commercial.

Or, you could add statistics gathering to your code (e.g., measuring how much time it takes to execute each synchronized method you suspect, breaking it into time waiting for sync / time executing the method), although that's a lot more work.

Answer 3

Could you try this method? If it works across multiple CPUs it should find the problem, but that's a big "if".

Basically, when you see that a thread is blocked, the call stack tells you exactly why. If you're not sure if you're seeing the real problem, do it a few times.