How to detect the cause of OutofMemoryError?

Question

I have a complaint that a server application of mine is crashing on high load.
It is a web app running in Tomcat 5.
I see the thread dumps and I see that there is an OutOfMemory error

1TISIGINFO Dump Event "systhrow" (00040000) Detail
"java/lang/OutOfMemoryError" "Failed to create thread: retVal -1073741830, errno 12" >received 1TIDATETIME Date: 2012/07/17 at 20:03:17 1TIFILENAME >Javacore filename:C:\ServerApplication\Tomcat5\bin\javacore.34545719.4646464.4172.0003.txt

The heap information is the following:

Maximum Java heap size : 1500m    
Initial Java heap size : 256m

This is the configuration for initial and Max heap size (32 bit java)

I also see that there is available free heap space

1STHEAPFREE    Bytes of Heap Space Free: 2D19F3C0   
1STHEAPALLOC   Bytes of Heap Space Allocated: 5DC00000

This is arround 750MB free space, right?

And from the thread method analysis I see that the number of Threads is 695 of which 49% is java/lang/Object.wait(Native Method) and 39% is in sun/misc/Unsafe.park(Native Method)
Also I see this NO JAVA STACK 1% not sure what that means.
Also 0 treads deadlocked and 2% is Runnable.

I am not sure how to interpret this information or how to go on from here to detect the underlying cause.
Any help on this?

Answer 1

According to this post:

There are two possible causes of the java.lang.OutOfMemoryError: Failed to create a thread message:

• There are too many threads running and the system has run out of internal resources to create new threads.
• The system has run out of native memory to use for the new thread. Threads require a native memory for internal JVM structures, a Java™ stack, and a native stack.

So this error may well be completely unrelated to memory, just that too many threads are created...

EDIT:

As you've got 695 threads, you would need 695 times the stack size as memory. Considering this post on thread limits, I suspect that you are trying to create too many threads for the available virtual memory space.

Answer 2

You should start the JVM with the -XX:+HeapDumpOnOutOfMemoryError flag. This will produce a heap dump when the OutOfMemoryError is generated.

Then, as @Steve said, you can use a tool like MAT to analyze the dump and see which objects are allocated, and who is keeping references to them. This usually will give you some insight on why your JVM is exhausting its memory.