Tomcat crash: There is insufficient memory for the Java Runtime Environment to continue

Question

We are running a webapplication with low number of users, about 7, but with high load of showing images. Recently we upgraded to Java 6u43 and Tomcat 6.036 and after some usage (about a week) tomcat services crashes. After a restart it can run for a few days, or just a few hours and crashes again. Even after reboot of the server it crashes again, we even saw that after 10 minutes of being active.

it is always with the same message in a hs_err_pid file in the bin folder of tomcat. See below an example of the full error message. It is always insufficient native memory and the main active thread is always JavaThread "C2 CompilerThread1", but always in a different method.

We are running on windows 2008 service pack 2, 32 bit, 2 CPU, 3GB. For tomcat xms set to 512KB and xmx to 1536KB. no other specific settings. When it crashes the Java heap is far below full usage of the 1536.

Any idea, suggestions what this could be?

(before the tomcat upgrade we used tomcat 6.014 and Java 6u02 for long time without any issue)

#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 629784 bytes for Chunk::new
# Possible reasons:
#   The system is out of physical RAM or swap space
#   In 32 bit mode, the process size limit was hit
# Possible solutions:
#   Reduce memory load on the system
#   Increase physical memory or swap space
#   Check if swap backing store is full
#   Use 64 bit Java on a 64 bit OS
#   Decrease Java heap size (-Xmx/-Xms)
#   Decrease number of Java threads
#   Decrease Java thread stack sizes (-Xss)
#   Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
#  Out of Memory Error (allocation.cpp:317), pid=5156, tid=1820
#
# JRE version: 6.0_43-b01
# Java VM: Java HotSpot(TM) Server VM (20.14-b01 mixed mode windows-x86 )

---------------  T H R E A D  ---------------

Current thread (0x00f99800):  JavaThread "C2 CompilerThread1" daemon [_thread_in_native, id=1820, stack(0x66680000,0x666f0000)]

Stack: [0x66680000,0x666f0000],  sp=0x666ee694,  free space=441k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)

[error occurred during error reporting (printing native stack), id 0xc0000005]


Current CompileTask:
C2:398266147 1679  !   org.apache.jasper.compiler.Compiler.isOutDated(Z)Z (485 bytes)


---------------  P R O C E S S  ---------------

Java Threads: ( => current thread )
  0x66870c00 JavaThread "http-8080-4" daemon [_thread_blocked, id=4856, stack(0x6b6a0000,0x6b710000)]
  0x66870800 JavaThread "http-8080-3" daemon [_thread_blocked, id=5684, stack(0x6b170000,0x6b1e0000)]
  0x66870000 JavaThread "ajp-8018-5" daemon [_thread_blocked, id=3532, stack(0x6b100000,0x6b170000)]
  0x6686fc00 JavaThread "ajp-8018-4" daemon [_thread_blocked, id=1400, stack(0x6b050000,0x6b0c0000)]
  0x6686f400 JavaThread "ajp-8018-3" daemon [_thread_blocked, id=2804, stack(0x6afe0000,0x6b050000)]
  0x6686ec00 JavaThread "http-8080-2" daemon [_thread_blocked, id=3940, stack(0x68fd0000,0x69040000)]
  0x6686e800 JavaThread "http-8080-1" daemon [_thread_blocked, id=5168, stack(0x68f60000,0x68fd0000)]
  0x6686e000 JavaThread "ajp-8018-2" daemon [_thread_blocked, id=4260, stack(0x68ef0000,0x68f60000)]
  0x6686dc00 JavaThread "ajp-8018-1" daemon [_thread_blocked, id=5476, stack(0x68e80000,0x68ef0000)]
  0x6686d400 JavaThread "ajp-8018-Acceptor-0" daemon [_thread_in_native, id=2640, stack(0x68e10000,0x68e80000)]
  0x6686d000 JavaThread "ajp-8018-CometPoller-7" daemon [_thread_blocked, id=4672, stack(0x68da0000,0x68e10000)]
  0x6686c800 JavaThread "ajp-8018-CometPoller-6" daemon [_thread_blocked, id=4892, stack(0x68d30000,0x68da0000)]
  0x6686c000 JavaThread "ajp-8018-CometPoller-5" daemon [_thread_blocked, id=1836, stack(0x68cc0000,0x68d30000)]
  0x6686bc00 JavaThread "ajp-8018-CometPoller-4" daemon [_thread_blocked, id=5452, stack(0x68c50000,0x68cc0000)]
  0x6686b400 JavaThread "ajp-8018-CometPoller-3" daemon [_thread_blocked, id=584, stack(0x68be0000,0x68c50000)]
  0x66fc7000 JavaThread "ajp-8018-CometPoller-2" daemon [_thread_blocked, id=1932, stack(0x68b70000,0x68be0000)]
  0x66fc6800 JavaThread "ajp-8018-CometPoller-1" daemon [_thread_blocked, id=5540, stack(0x68b00000,0x68b70000)]
  0x66fc6000 JavaThread "ajp-8018-CometPoller-0" daemon [_thread_blocked, id=2728, stack(0x68a90000,0x68b00000)]
  0x66fc5c00 JavaThread "ajp-8018-Poller-7" daemon [_thread_blocked, id=5104, stack(0x68a20000,0x68a90000)]
  0x66fc5400 JavaThread "ajp-8018-Poller-6" daemon [_thread_blocked, id=3756, stack(0x689b0000,0x68a20000)]
  0x66fc5000 JavaThread "ajp-8018-Poller-5" daemon [_thread_blocked, id=4888, stack(0x68940000,0x689b0000)]
  0x66fc4800 JavaThread "ajp-8018-Poller-4" daemon [_thread_in_native, id=3924, stack(0x688d0000,0x68940000)]
  0x66fc4400 JavaThread "ajp-8018-Poller-3" daemon [_thread_blocked, id=2632, stack(0x68860000,0x688d0000)]
  0x66fc3c00 JavaThread "ajp-8018-Poller-2" daemon [_thread_blocked, id=3792, stack(0x687f0000,0x68860000)]
  0x66fc3400 JavaThread "ajp-8018-Poller-1" daemon [_thread_blocked, id=4840, stack(0x68380000,0x683f0000)]
  0x66fc3000 JavaThread "ajp-8018-Poller-0" daemon [_thread_blocked, id=5424, stack(0x68310000,0x68380000)]
  0x66fc2800 JavaThread "http-8080-Acceptor-0" daemon [_thread_in_native, id=5532, stack(0x682a0000,0x68310000)]
  0x66fc2400 JavaThread "http-8080-Sendfile-0" daemon [_thread_blocked, id=2560, stack(0x68230000,0x682a0000)]
  0x66fc1c00 JavaThread "http-8080-CometPoller-7" daemon [_thread_blocked, id=4264, stack(0x681c0000,0x68230000)]
  0x66fc1800 JavaThread "http-8080-CometPoller-6" daemon [_thread_blocked, id=5948, stack(0x68150000,0x681c0000)]
  0x66fc1000 JavaThread "http-8080-CometPoller-5" daemon [_thread_blocked, id=3768, stack(0x680e0000,0x68150000)]
  0x66fc0800 JavaThread "http-8080-CometPoller-4" daemon [_thread_blocked, id=5772, stack(0x68070000,0x680e0000)]
  0x66fc0400 JavaThread "http-8080-CometPoller-3" daemon [_thread_blocked, id=564, stack(0x68000000,0x68070000)]
  0x66fbfc00 JavaThread "http-8080-CometPoller-2" daemon [_thread_blocked, id=1192, stack(0x67f90000,0x68000000)]
  0x66fbf800 JavaThread "http-8080-CometPoller-1" daemon [_thread_blocked, id=2776, stack(0x67f20000,0x67f90000)]
  0x66c2f800 JavaThread "http-8080-CometPoller-0" daemon [_thread_blocked, id=1012, stack(0x67eb0000,0x67f20000)]
  0x66dbe400 JavaThread "http-8080-Poller-7" daemon [_thread_blocked, id=3288, stack(0x67e40000,0x67eb0000)]
  0x66d9d800 JavaThread "http-8080-Poller-6" daemon [_thread_blocked, id=3684, stack(0x67dd0000,0x67e40000)]
  0x669af000 JavaThread "http-8080-Poller-5" daemon [_thread_blocked, id=4244, stack(0x67d60000,0x67dd0000)]
  0x6701b000 JavaThread "http-8080-Poller-4" daemon [_thread_blocked, id=2604, stack(0x67a90000,0x67b00000)]
  0x66f4f000 JavaThread "http-8080-Poller-3" daemon [_thread_blocked, id=2276, stack(0x67a20000,0x67a90000)]
  0x66509000 JavaThread "http-8080-Poller-2" daemon [_thread_blocked, id=5324, stack(0x679b0000,0x67a20000)]
  0x66c5a400 JavaThread "http-8080-Poller-1" daemon [_thread_blocked, id=2628, stack(0x678f0000,0x67960000)]
  0x66b9a800 JavaThread "http-8080-Poller-0" daemon [_thread_blocked, id=5576, stack(0x67850000,0x678c0000)]
  0x668b4000 JavaThread "ContainerBackgroundProcessor[StandardEngine[Catalina]]" daemon [_thread_blocked, id=3152, stack(0x677e0000,0x67850000)]
  0x66853000 JavaThread "GC Daemon" daemon [_thread_blocked, id=1636, stack(0x675e0000,0x67650000)]
  0x00f9c800 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=3184, stack(0x666f0000,0x66760000)]
=>0x00f99800 JavaThread "C2 CompilerThread1" daemon [_thread_in_native, id=1820, stack(0x66680000,0x666f0000)]
  0x00f98400 JavaThread "C2 CompilerThread0" daemon [_thread_blocked, id=3932, stack(0x66610000,0x66680000)]
  0x00f96c00 JavaThread "Attach Listener" daemon [_thread_blocked, id=2612, stack(0x663a0000,0x66410000)]
  0x00f95c00 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=5128, stack(0x66330000,0x663a0000)]
  0x00f68800 JavaThread "Finalizer" daemon [_thread_blocked, id=4868, stack(0x662c0000,0x66330000)]
  0x00f62000 JavaThread "Reference Handler" daemon [_thread_blocked, id=5500, stack(0x66250000,0x662c0000)]
  0x00e5a000 JavaThread "main" [_thread_in_native, id=2800, stack(0x00d70000,0x00de0000)]

Other Threads:
  0x00f5e800 VMThread [stack: 0x661e0000,0x66250000] [id=4608]
  0x00f9d000 WatcherThread [stack: 0x66760000,0x667d0000] [id=3600]

VM state:not at safepoint (normal execution)

VM Mutex/Monitor currently owned by a thread: None

Heap
 PSYoungGen      total 368192K, used 215481K [0x46860000, 0x65730000, 0x65c60000)
  eden space 230016K, 84% used [0x46860000,0x5263eae8,0x54900000)
  from space 138176K, 15% used [0x5cfb0000,0x5e43f968,0x656a0000)
  to   space 137920K, 0% used [0x54900000,0x54900000,0x5cfb0000)
 PSOldGen        total 349568K, used 320646K [0x08060000, 0x1d5c0000, 0x46860000)
  object space 349568K, 91% used [0x08060000,0x1b981b60,0x1d5c0000)
 PSPermGen       total 33664K, used 33565K [0x04060000, 0x06140000, 0x08060000)
  object space 33664K, 99% used [0x04060000,0x061277e8,0x06140000)

Code Cache  [0x00fa0000, 0x01570000, 0x03fa0000)
 total_blobs=1881 nmethods=1658 adapters=176 free_code_cache=44292736 largest_free_block=10112

Dynamic libraries:
0x00400000 - 0x00417000     D:\Kappa\apache-tomcat-6.0.36\bin\tomcat6.exe
0x77460000 - 0x77587000     C:\Windows\system32\ntdll.dll
0x76490000 - 0x7656c000     C:\Windows\system32\kernel32.dll
0x75ef0000 - 0x75fb6000     C:\Windows\system32\ADVAPI32.dll
0x75fc0000 - 0x76083000     C:\Windows\system32\RPCRT4.dll
0x76570000 - 0x77080000     C:\Windows\system32\SHELL32.dll
0x762d0000 - 0x7637a000     C:\Windows\system32\msvcrt.dll
0x75d30000 - 0x75d7b000     C:\Windows\system32\GDI32.dll
0x75e50000 - 0x75eed000     C:\Windows\system32\USER32.dll
0x76270000 - 0x762c9000     C:\Windows\system32\SHLWAPI.dll
0x775d0000 - 0x775ee000     C:\Windows\system32\IMM32.DLL
0x775f0000 - 0x776b8000     C:\Windows\system32\MSCTF.dll
0x77590000 - 0x77599000     C:\Windows\system32\LPK.DLL
0x75d80000 - 0x75dfd000     C:\Windows\system32\USP10.dll
0x748d0000 - 0x74a6e000     C:\Windows\WinSxS\x86_microsoft.windows.common-controls_6595b64144ccf1df_6.0.6002.18005_none_5cb72f96088b0de0\comctl32.dll
0x6db90000 - 0x6dfea000     D:\Kappa\Java\jdk1.6.0_43\jre\bin\server\jvm.dll
0x743e0000 - 0x74412000     C:\Windows\system32\WINMM.dll
0x77240000 - 0x77385000     C:\Windows\system32\ole32.dll
0x771b0000 - 0x7723d000     C:\Windows\system32\OLEAUT32.dll
0x74380000 - 0x743b9000     C:\Windows\system32\OLEACC.dll
0x7c340000 - 0x7c396000     C:\Windows\system32\MSVCR71.dll
0x75ae0000 - 0x75b0c000     C:\Windows\system32\apphelp.dll
0x6d870000 - 0x6d87c000     D:\Kappa\Java\jdk1.6.0_43\jre\bin\verify.dll
0x6d3d0000 - 0x6d3ef000     D:\Kappa\Java\jdk1.6.0_43\jre\bin\java.dll
0x6d8b0000 - 0x6d8bf000     D:\Kappa\Java\jdk1.6.0_43\jre\bin\zip.dll
0x6d600000 - 0x6d609000     D:\Kappa\Java\jdk1.6.0_43\jre\bin\management.dll
0x673d0000 - 0x674d7000     D:\Kappa\apache-tomcat-6.0.36\bin\tcnative-1.dll
0x75cd0000 - 0x75cd7000     C:\Windows\system32\PSAPI.DLL
0x775a0000 - 0x775cd000     C:\Windows\system32\WS2_32.dll
0x760a0000 - 0x760a6000     C:\Windows\system32\NSI.dll
0x75300000 - 0x7533b000     C:\Windows\system32\MSWSOCK.dll
0x75060000 - 0x7509b000     C:\Windows\system32\rsaenh.dll
0x6d850000 - 0x6d858000     D:\Kappa\Java\jdk1.6.0_43\jre\bin\sunmscapi.dll
0x755e0000 - 0x756d2000     C:\Windows\system32\CRYPT32.dll
0x75740000 - 0x75752000     C:\Windows\system32\MSASN1.dll
0x75b90000 - 0x75bae000     C:\Windows\system32\USERENV.dll
0x75b70000 - 0x75b84000     C:\Windows\system32\Secur32.dll
0x74f00000 - 0x74f05000     C:\Windows\System32\wshtcpip.dll
0x6d6b0000 - 0x6d6c3000     D:\Kappa\Java\jdk1.6.0_43\jre\bin\net.dll
0x75360000 - 0x75365000     C:\Windows\System32\wship6.dll
0x74c20000 - 0x74c2f000     C:\Windows\system32\NLAapi.dll
0x75540000 - 0x75559000     C:\Windows\system32\IPHLPAPI.DLL
0x75500000 - 0x75535000     C:\Windows\system32\dhcpcsvc.DLL
0x75780000 - 0x757ac000     C:\Windows\system32\DNSAPI.dll
0x754f0000 - 0x754f7000     C:\Windows\system32\WINNSI.DLL
0x754c0000 - 0x754e2000     C:\Windows\system32\dhcpcsvc6.DLL
0x74290000 - 0x7429f000     C:\Windows\system32\napinsp.dll
0x74270000 - 0x74278000     C:\Windows\System32\winrnr.dll
0x75e00000 - 0x75e49000     C:\Windows\system32\WLDAP32.dll
0x74280000 - 0x74286000     C:\Windows\system32\rasadhlp.dll
0x003d0000 - 0x003dd000     D:\Kappa\Java\jdk1.6.0_43\bin\ntlmauth.dll
0x75410000 - 0x75417000     C:\Windows\system32\credssp.dll
0x750d0000 - 0x75114000     C:\Windows\system32\schannel.dll
0x759a0000 - 0x75a16000     C:\Windows\system32\NETAPI32.dll
0x752c0000 - 0x752f7000     C:\Windows\system32\msv1_0.dll
0x757b0000 - 0x757c1000     C:\Windows\system32\cryptdll.dll
0x6d0b0000 - 0x6d1fc000     D:\Kappa\Java\jdk1.6.0_43\jre\bin\awt.dll
0x707c0000 - 0x70802000     C:\Windows\system32\WINSPOOL.DRV
0x740d0000 - 0x74155000     C:\Windows\WinSxS\x86_microsoft.windows.common-controls_6595b64144ccf1df_5.82.6001.18000_none_886786f450a74a05\COMCTL32.dll
0x00920000 - 0x0092b000     D:\Kappa\Java\jdk1.6.0_43\jre\bin\mlib_jai_util.dll
0x6b250000 - 0x6b69e000     D:\Kappa\Java\jdk1.6.0_43\jre\bin\mlib_jai_mmx.dll
0x6d220000 - 0x6d24f000     D:\Kappa\Java\jdk1.6.0_43\jre\bin\cmm.dll

VM Arguments:
jvm_args: -Dcatalina.base=D:\Kappa\apache-tomcat-6.0.36 -Dcatalina.home=D:\Kappa\apache-tomcat-6.0.36 -Djava.endorsed.dirs=D:\Kappa\apache-tomcat-6.0.36\endorsed -Djava.io.tmpdir=D:\Kappa\apache-tomcat-6.0.36\temp -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.util.logging.config.file=D:\Kappa\apache-tomcat-6.0.36\conf\logging.properties -Xms512m -Xmx1500m 
java_command: <unknown>
Launcher Type: generic

Environment Variables:
JAVA_HOME=D:\Kappa\Java\jdk1.6.0_43
CLASSPATH=D:\Kappa\apache-tomcat-6.0.36\lib\servlet-api.jar
PATH=C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;D:\Kappa\Java\jdk1.6.0_43\bin;C:\Program Files\Windows Imaging\
USERNAME=SVC-UPSLKappa-DIA
OS=Windows_NT
PROCESSOR_IDENTIFIER=x86 Family 6 Model 23 Stepping 6, GenuineIntel



---------------  S Y S T E M  ---------------

OS: Windows Server 2008 Build 6002 Service Pack 2

CPU:total 2 (1 cores per cpu, 1 threads per core) family 6 model 23 stepping 6, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1

Memory: 4k page, physical 2095900k(292724k free), swap 4467300k(2557472k free)

vm_info: Java HotSpot(TM) Server VM (20.14-b01) for windows-x86 JRE (1.6.0_43-b01), built on Mar  1 2013 02:47:24 by "java_re" with MS VC++ 7.1 (VS2003)

time: Tue Oct 01 08:51:47 2013
elapsed time: 398205 seconds

Answer 1

The process heap != the java heap.

On 32-bit Windows you have (unless you set some boot switches) a 2GB maximum per process. This 2GB has to support:

• Java Heap (set with Xms, Xmx)
• PermGen
• Thread stacks
• GC
• Code cache
• Native JVM code

and probably a couple of things I have forgotten.

If the total of all of the above exceeds 2GB the process crashes. This is what has happended in your case.

The error message tells you quite clearly what you need to do. Pick one of the options and implement it.

Answer 2

Use a profiler like VisualVM. Connect it to monitor the memory use of your tomcat server (remote configuration). It will constantly monitor and show you your heap allocation. You can also see the objects being hold in memory.

If this happens you have either hit the memory limit of the process (you will see the total used amount in visual vm). It might be even a memory leak or a bug in the JVM. Check if you are using the 64 bit version. But first check out the object creation and heap using VisualVM (or similar). You can tell by the graph if your consumption goes up in a different way.

You can even compare the heap usage pattern you gain using the previous used Java version. This way you can compare the memory consumption of the old working setup with the new problematic one.