Menu
I have a jar file running on an amazon-ec2-m1.large instance with linux 64bit operating system. I run out of memory after different hours, usually between 2-4, although in my last log write (before the error file created ) I see the following:
Memory -
totalHeapSize: 264 MB,
maxHeapSize: 1656 MB,
freeHeapSize: 220 MB
When I look from the jar start until in run out of memory - all values are the same and the freeHeapSize is between 200-230 MB (i check every 30 seconds and do System.gc() )
in the hs_err_pid2250.log wrriten:
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Cannot create GC thread. Out of system resources.
# Possible reasons:
# The system is out of physical RAM or swap space
# In 32 bit mode, the process size limit was hit
# Possible solutions:
# Reduce memory load on the system
# Increase physical memory or swap space
# Check if swap backing store is full
# Use 64 bit Java on a 64 bit OS
# Decrease Java heap size (-Xmx/-Xms)
# Decrease number of Java threads
# Decrease Java thread stack sizes (-Xss)
# Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
# Out of Memory Error (gcTaskThread.cpp:46), pid=2250, tid=140227064182528
#
# JRE version: 7.0_25-b15
# Java VM: Java HotSpot(TM) 64-Bit Server VM (23.25-b01 mixed mode linux-amd64 compressed oops)
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
--------------- T H R E A D ---------------
Current thread (0x00007f8920007800): JavaThread "Unknown thread" [_thread_in_vm, id=2251, stack(0x00007f8928493000,0x00007f8928594000)]
Stack: [0x00007f8928493000,0x00007f8928594000], sp=0x00007f89285925d0, free space=1021k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0x8a1d3a] VMError::report_and_die()+0x2ea
V [libjvm.so+0x4071fb] report_vm_out_of_memory(char const*, int, unsigned long, char const*)+0x9b
V [libjvm.so+0x4b3e2a] GCTaskThread::GCTaskThread(GCTaskManager*, unsigned int, unsigned int)+0xfa
V [libjvm.so+0x4b2896] GCTaskManager::initialize()+0x266
V [libjvm.so+0x75dc4f] ParallelScavengeHeap::initialize()+0x6cf
V [libjvm.so+0x87a078] Universe::initialize_heap()+0xb8
V [libjvm.so+0x87c7ed] universe_init()+0x7d
V [libjvm.so+0x500315] init_globals()+0x65
V [libjvm.so+0x863a62] Threads::create_vm(JavaVMInitArgs*, bool*)+0x1e2
V [libjvm.so+0x55ff34] JNI_CreateJavaVM+0x74
C [libjli.so+0x2f8e] JavaMain+0x9e
--------------- P R O C E S S ---------------
Java Threads: ( => current thread )
Other Threads:
=>0x00007f8920007800 (exited) JavaThread "Unknown thread" [_thread_in_vm, id=2251, stack(0x00007f8928493000,0x00007f8928594000)]
VM state:not at safepoint (not fully initialized)
VM Mutex/Monitor currently owned by a thread: None
GC Heap History (0 events):
No events
Deoptimization events (0 events):
No events
Internal exceptions (0 events):
No events
Events (0 events):
No events
Dynamic libraries:
00400000-00401000 r-xp 00000000 ca:01 9384 /usr/java/jdk1.7.0_25/bin/java
00600000-00601000 rw-p 00000000 ca:01 9384 /usr/java/jdk1.7.0_25/bin/java
01f54000-01f75000 rw-p 00000000 00:00 0 [heap]
86600000-87ac0000 rw-p 00000000 00:00 0
87ac0000-8b800000 rw-p 00000000 00:00 0
8b800000-905b0000 rw-p 00000000 00:00 0
905b0000-d92b0000 rw-p 00000000 00:00 0
d92b0000-db980000 rw-p 00000000 00:00 0
db980000-100000000 rw-p 00000000 00:00 0
7f891d000000-7f891d270000 rwxp 00000000 00:00 0
7f891d270000-7f8920022000 rw-p 00000000 00:00 0
7f8920022000-7f8924000000 ---p 00000000 00:00 0
7f8925555000-7f892574d000 rw-p 00000000 00:00 0
7f892574d000-7f892574e000 ---p 00000000 00:00 0
7f892574e000-7f892584e000 rw-p 00000000 00:00 0
7f892584e000-7f892584f000 ---p 00000000 00:00 0
7f892584f000-7f892595a000 rw-p 00000000 00:00 0 [stack:2252]
7f892595a000-7f8925978000 rw-p 00000000 00:00 0
7f8925978000-7f892599f000 rw-p 00000000 00:00 0
7f892599f000-7f8925be6000 rw-p 00000000 00:00 0
7f8925be6000-7f8925bf1000 rw-p 00000000 00:00 0
7f8925bf1000-7f8925c0f000 rw-p 00000000 00:00 0
7f8925c0f000-7f8925c36000 rw-p 00000000 00:00 0
7f8925c36000-7f8925e7c000 rw-p 00000000 00:00 0
7f8925e7c000-7f8925e90000 rw-p 00000000 00:00 0
7f8925e90000-7f8925fb3000 rw-p 00000000 00:00 0
7f8925fb3000-7f8925fb4000 rw-p 00000000 00:00 0
7f8925fb4000-7f8925fcf000 r-xp 00000000 ca:01 17790 /usr/java/jdk1.7.0_25/jre/lib/amd64/libzip.so
7f8925fcf000-7f89261ce000 ---p 0001b000 ca:01 17790 /usr/java/jdk1.7.0_25/jre/lib/amd64/libzip.so
7f89261ce000-7f89261cf000 rw-p 0001a000 ca:01 17790 /usr/java/jdk1.7.0_25/jre/lib/amd64/libzip.so
7f89261cf000-7f89261db000 r-xp 00000000 ca:01 17544 /lib64/libnss_files-2.12.so
7f89261db000-7f89263da000 ---p 0000c000 ca:01 17544 /lib64/libnss_files-2.12.so
7f89263da000-7f89263db000 r--p 0000b000 ca:01 17544 /lib64/libnss_files-2.12.so
7f89263db000-7f89263dc000 rw-p 0000c000 ca:01 17544 /lib64/libnss_files-2.12.so
7f89263dc000-7f8926405000 r-xp 00000000 ca:01 17571 /usr/java/jdk1.7.0_25/jre/lib/amd64/libjava.so
7f8926405000-7f8926605000 ---p 00029000 ca:01 17571 /usr/java/jdk1.7.0_25/jre/lib/amd64/libjava.so
7f8926605000-7f8926607000 rw-p 00029000 ca:01 17571 /usr/java/jdk1.7.0_25/jre/lib/amd64/libjava.so
7f8926607000-7f8926614000 r-xp 00000000 ca:01 17789 /usr/java/jdk1.7.0_25/jre/lib/amd64/libverify.so
7f8926614000-7f8926813000 ---p 0000d000 ca:01 17789 /usr/java/jdk1.7.0_25/jre/lib/amd64/libverify.so
7f8926813000-7f8926815000 rw-p 0000c000 ca:01 17789 /usr/java/jdk1.7.0_25/jre/lib/amd64/libverify.so
7f8926815000-7f892681c000 r-xp 00000000 ca:01 17559 /lib64/librt-2.12.so
7f892681c000-7f8926a1b000 ---p 00007000 ca:01 17559 /lib64/librt-2.12.so
7f8926a1b000-7f8926a1c000 r--p 00006000 ca:01 17559 /lib64/librt-2.12.so
7f8926a1c000-7f8926a1d000 rw-p 00007000 ca:01 17559 /lib64/librt-2.12.so
7f8926a1d000-7f8926a9f000 r-xp 00000000 ca:01 17537 /lib64/libm-2.12.so
7f8926a9f000-7f8926c9e000 ---p 00082000 ca:01 17537 /lib64/libm-2.12.so
7f8926c9e000-7f8926c9f000 r--p 00081000 ca:01 17537 /lib64/libm-2.12.so
7f8926c9f000-7f8926ca0000 rw-p 00082000 ca:01 17537 /lib64/libm-2.12.so
7f8926ca0000-7f89276e1000 r-xp 00000000 ca:01 17795 /usr/java/jdk1.7.0_25/jre/lib/amd64/server/libjvm.so
7f89276e1000-7f89278e1000 ---p 00a41000 ca:01 17795 /usr/java/jdk1.7.0_25/jre/lib/amd64/server/libjvm.so
7f89278e1000-7f8927984000 rw-p 00a41000 ca:01 17795 /usr/java/jdk1.7.0_25/jre/lib/amd64/server/libjvm.so
7f8927984000-7f89279c0000 rw-p 00000000 00:00 0
7f89279c0000-7f8927b41000 r-xp 00000000 ca:01 7616 /lib64/libc-2.12.so
7f8927b41000-7f8927d41000 ---p 00181000 ca:01 7616 /lib64/libc-2.12.so
7f8927d41000-7f8927d45000 r--p 00181000 ca:01 7616 /lib64/libc-2.12.so
7f8927d45000-7f8927d46000 rw-p 00185000 ca:01 7616 /lib64/libc-2.12.so
7f8927d46000-7f8927d4c000 rw-p 00000000 00:00 0
7f8927d4c000-7f8927d4e000 r-xp 00000000 ca:01 17535 /lib64/libdl-2.12.so
7f8927d4e000-7f8927f4e000 ---p 00002000 ca:01 17535 /lib64/libdl-2.12.so
7f8927f4e000-7f8927f4f000 r--p 00002000 ca:01 17535 /lib64/libdl-2.12.so
7f8927f4f000-7f8927f50000 rw-p 00003000 ca:01 17535 /lib64/libdl-2.12.so
7f8927f50000-7f8927f66000 r-xp 00000000 ca:01 16704 /usr/java/jdk1.7.0_25/jre/lib/amd64/jli/libjli.so
7f8927f66000-7f8928166000 ---p 00016000 ca:01 16704 /usr/java/jdk1.7.0_25/jre/lib/amd64/jli/libjli.so
7f8928166000-7f8928167000 rw-p 00016000 ca:01 16704 /usr/java/jdk1.7.0_25/jre/lib/amd64/jli/libjli.so
7f8928167000-7f892817e000 r-xp 00000000 ca:01 7640 /lib64/libpthread-2.12.so
7f892817e000-7f892837d000 ---p 00017000 ca:01 7640 /lib64/libpthread-2.12.so
7f892837d000-7f892837e000 r--p 00016000 ca:01 7640 /lib64/libpthread-2.12.so
7f892837e000-7f892837f000 rw-p 00017000 ca:01 7640 /lib64/libpthread-2.12.so
7f892837f000-7f8928383000 rw-p 00000000 00:00 0
7f8928383000-7f89283a1000 r-xp 00000000 ca:01 9381 /lib64/ld-2.12.so
7f89283cb000-7f89283d5000 rw-p 00000000 00:00 0
7f89283d5000-7f892848b000 rw-p 00000000 00:00 0
7f892848b000-7f8928493000 rw-s 00000000 ca:01 8174 /tmp/hsperfdata_gurustrade/2250
7f8928493000-7f8928496000 ---p 00000000 00:00 0
7f8928496000-7f8928598000 rw-p 00000000 00:00 0 [stack:2251]
7f892859b000-7f892859e000 rw-p 00000000 00:00 0
7f892859e000-7f892859f000 r--p 00000000 00:00 0
7f892859f000-7f89285a0000 rw-p 00000000 00:00 0
7f89285a0000-7f89285a1000 r--p 0001d000 ca:01 9381 /lib64/ld-2.12.so
7f89285a1000-7f89285a2000 rw-p 0001e000 ca:01 9381 /lib64/ld-2.12.so
7f89285a2000-7f89285a3000 rw-p 00000000 00:00 0
7fff64527000-7fff64548000 rw-p 00000000 00:00 0 [stack]
7fff645ff000-7fff64600000 r-xp 00000000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
VM Arguments:
java_command: /home/gurustrade/proj/versions/trunk/gt_engine/gt_engine.jar
Launcher Type: SUN_STANDARD
Environment Variables:
PATH=/usr/bin:/bin
SHELL=/bin/sh
Signal Handlers:
SIGSEGV: [libjvm.so+0x8a26a0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGBUS: [libjvm.so+0x8a26a0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGFPE: [libjvm.so+0x73f3d0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGPIPE: [libjvm.so+0x73f3d0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGXFSZ: [libjvm.so+0x73f3d0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGILL: [libjvm.so+0x73f3d0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGUSR1: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGUSR2: [libjvm.so+0x73ed20], sa_mask[0]=0x00000000, sa_flags=0x10000004
SIGHUP: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGINT: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGTERM: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGQUIT: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000
--------------- S Y S T E M ---------------
OS:Linux
uname:Linux 3.4.48-45.46.amzn1.x86_64 #1 SMP Wed Jun 12 02:04:44 UTC 2013 x86_64
libc:glibc 2.12 NPTL 2.12
rlimit: STACK 8192k, CORE 0k, NPROC 1024, NOFILE 4096, AS infinity
load average:1.01 1.04 1.10
/proc/meminfo:
MemTotal: 7633612 kB
MemFree: 5784036 kB
Buffers: 186424 kB
Cached: 1025752 kB
SwapCached: 0 kB
Active: 980284 kB
Inactive: 706036 kB
Active(anon): 474096 kB
Inactive(anon): 100 kB
Active(file): 506188 kB
Inactive(file): 705936 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 88 kB
Writeback: 0 kB
AnonPages: 474084 kB
Mapped: 43752 kB
Shmem: 156 kB
Slab: 103892 kB
SReclaimable: 84548 kB
SUnreclaim: 19344 kB
KernelStack: 9032 kB
PageTables: 20180 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 3816804 kB
Committed_AS: 2130476 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 23908 kB
VmallocChunk: 34359714455 kB
AnonHugePages: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 7872512 kB
DirectMap2M: 0 kB
CPU:total 2 (4 cores per cpu, 1 threads per core) family 6 model 23 stepping 10, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, tsc
/proc/cpuinfo:
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 23
model name : Intel(R) Xeon(R) CPU E5430 @ 2.66GHz
stepping : 10
microcode : 0xa07
cpu MHz : 2660.000
cache size : 6144 KB
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu tsc msr pae cx8 cmov pat pse36 clflush dts mmx fxsr sse sse2 ss ht pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good nopl pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 lahf_lm dtherm tpr_shadow vnmi flexpriority
bogomips : 5320.00
clflush size : 64
cache_alignment : 64
address sizes : 38 bits physical, 48 bits virtual
power management:
processor : 1
vendor_id : GenuineIntel
cpu family : 6
model : 23
model name : Intel(R) Xeon(R) CPU E5430 @ 2.66GHz
stepping : 10
microcode : 0xa07
cpu MHz : 2660.000
cache size : 6144 KB
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu tsc msr pae cx8 cmov pat pse36 clflush dts mmx fxsr sse sse2 ss ht pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good nopl pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 lahf_lm dtherm tpr_shadow vnmi flexpriority
bogomips : 5320.00
clflush size : 64
cache_alignment : 64
address sizes : 38 bits physical, 48 bits virtual
power management:
Memory: 4k page, physical 7633612k(5784036k free), swap 0k(0k free)
vm_info: Java HotSpot(TM) 64-Bit Server VM (23.25-b01) for linux-amd64 JRE (1.7.0_25-b15), built on Jun 5 2013 20:48:36 by "java_re" with gcc 4.3.0 20080428 (Red Hat 4.3.0-8)
time: Mon Aug 5 16:11:16 2013
elapsed time: 0 seconds
Also, I use threadpoolexecutor with 30 core threads and 100 max (never user more than 30 although at some times I use all 30)
Finally, I user bonecp connection pool with 50 connections
Any suggestions? :)
After suggestion, I added a heap dump after running the application from my local pc.
How can I continue from here?
Are the critical orange jumps important?, or just the used important?
975
1) An easy way to solve OutOfMemoryError in java is to increase the maximum heap size by using JVM options "-Xmx512M", this will immediately solve your OutOfMemoryError.
If the Java heap is being exhausted, and increasing the Java heap size does not solve the problem, the next stage is to examine the objects that are on the heap, and look for suspect data structures that are referencing large numbers of Java objects that should have been released.
I was able to resolve an almost identical error by following the advice given by:
https://confluence.csiro.au/pages/viewpage.action?pageId=278167841
Essentially, they state:
When the Java Virtual Machine starts, by default it spawns a number of Garbage Collection (GC) threads, which are used for parallel GC operations … the number of such threads is calculated by this formula: (ncpus <= 8) ? ncpus : 3 + ((ncpus * 5) / 8) … Because of the creation of these many threads … the program will automatically reach the system limit …
I limited the number of garbage collecting threads using the environment variable, like so:
export _JAVA_OPTIONS="-XX:ParallelGCThreads=2"
Not sure about the overall performance hit regarding GC, but at least today's work is done.
Please check the original post for more options.
Good Luck, - Stu
99
Please try 64 bit version of Java if you are using 32 bit version,
Also add new JVM argument , "-XX:+CMSClassUnloadingEnabled \ " which clear some unused class objects
39
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
