Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Why the operating system says it can't allocate memory to jvm when it has enough memory

I'm trying to start a new jvm with the command: java -version, but it report a error:

# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 160088 bytes for AllocateHeap
# An error report file with more information is saved as:
# /users/xxx/xxx/hs_err_pid12365.log

Then I checked file /users/xxx/xxx/hs_err_pid12365.log, and I find out it say:

# Native memory allocation (malloc) failed to allocate 295856 bytes for Chunk::new

And the stack info also proves that the system can't allocate native memory:

V  [libjvm.so+0xad3525]  VMError::report_and_die()+0x2e5
V  [libjvm.so+0x4e0457]  report_vm_out_of_memory(char const*, int, unsigned long, VMErrorType, char const*)+0x67
V  [libjvm.so+0xa9775f]  universe_init()+0x35f
V  [libjvm.so+0x631fd0]  init_globals()+0x50
V  [libjvm.so+0xa79b69]  Threads::create_vm(JavaVMInitArgs*, bool*)+0x409
V  [libjvm.so+0x6d7b4f]  JNI_CreateJavaVM+0x4f
C  [libjli.so+0x7ee4]  JavaMain+0x84
C  [libpthread.so.0+0x68ca]  start_thread+0xba

But when I checked the available memory size using free -m. I find that the system still has enough available memory:

scada1:/users/xxx/xxx> free -m
             total       used       free     shared    buffers     cached
Mem:          7983       4867       3116          0        110       1031
-/+ buffers/cache:       3726       4257
Swap:        15257         71      15186

So why the system can't allocate memory when it has enough available memory

------- System Info ---------

The operating system info is:

scada1:/users/ems> uname -a
Linux scada1 2.6.32-5-linx-amd64 #1 SMP Mon May 18 04:02:09 UTC 2015 x86_64 GNU/Linux

The java version is:

jdk1.8.0_201

------- The hs_err_pidxxx.log Content ---------

#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 160088 bytes for AllocateHeap
# Possible reasons:
#   The system is out of physical RAM or swap space
#   The process is running with CompressedOops enabled, and the Java Heap may be blocking the growth of the native heap
# Possible solutions:
#   Reduce memory load on the system
#   Increase physical memory or swap space
#   Check if swap backing store is full
#   Decrease Java heap size (-Xmx/-Xms)
#   Decrease number of Java threads
#   Decrease Java thread stack sizes (-Xss)
#   Set larger code cache with -XX:ReservedCodeCacheSize=
#   JVM is running with Unscaled Compressed Oops mode in which the Java heap is
#     placed in the first 4GB address space. The Java Heap base address is the
#     maximum limit for the native heap growth. Please use -XX:HeapBaseMinAddress
#     to set the Java Heap base and to place the Java Heap above 4GB virtual address.
# This output file may be truncated or incomplete.
#
#  Out of Memory Error (allocation.inline.hpp:61), pid=1348, tid=0x00007f7f87350700
#
# JRE version:  (8.0_201-b09) (build )
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.201-b09 mixed mode linux-amd64 compressed oops)
# Core dump written. Default location: /users/ems/zoutairan/core or core.1348 (max size 521000 kB). To ensure a full core dump, try "ulimit -c unlimited" before starting Java again
#

---------------  T H R E A D  ---------------

Current thread (0x000000000107e000):  JavaThread "Unknown thread" [_thread_in_vm, id=1349, stack(0x00007f7f87250000,0x00007f7f87351000)]

Stack: [0x00007f7f87250000,0x00007f7f87351000],  sp=0x00007f7f8734f910,  free space=1022k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0xad3525]  VMError::report_and_die()+0x2e5
V  [libjvm.so+0x4e0457]  report_vm_out_of_memory(char const*, int, unsigned long, VMErrorType, char const*)+0x67
V  [libjvm.so+0xa9775f]  universe_init()+0x35f
V  [libjvm.so+0x631fd0]  init_globals()+0x50
V  [libjvm.so+0xa79b69]  Threads::create_vm(JavaVMInitArgs*, bool*)+0x409
V  [libjvm.so+0x6d7b4f]  JNI_CreateJavaVM+0x4f
C  [libjli.so+0x7ee4]  JavaMain+0x84
C  [libpthread.so.0+0x68ca]  start_thread+0xba


---------------  P R O C E S S  ---------------

Java Threads: ( => current thread )

Other Threads:

=>0x000000000107e000 (exited) JavaThread "Unknown thread" [_thread_in_vm, id=1349, stack(0x00007f7f87250000,0x00007f7f87351000)]

VM state:not at safepoint (not fully initialized)

VM Mutex/Monitor currently owned by a thread: None

heap address: 0x0000000083400000, size: 1996 MB, Compressed Oops mode: 32-bit
Narrow klass base: 0x0000000000000000, Narrow klass shift: 3
Compressed class space size: 1073741824 Address: 0x0000000100000000

GC Heap History (0 events):
No events

Deoptimization events (0 events):
No events

Classes redefined (0 events):
No events

Internal exceptions (0 events):
No events

Events (0 events):
No events


Dynamic libraries:
00400000-00401000 r-xp 00000000 ca:06 16598510                           /usr/java/jdk1.8.0_201/bin/java
00600000-00601000 r--p 00000000 ca:06 16598510                           /usr/java/jdk1.8.0_201/bin/java
00601000-00602000 rw-p 00001000 ca:06 16598510                           /usr/java/jdk1.8.0_201/bin/java
01060000-010b8000 rw-p 00000000 00:00 0                                  [heap]
83400000-88800000 rw-p 00000000 00:00 0 
88800000-d6700000 ---p 00000000 00:00 0 
d6700000-d9100000 rw-p 00000000 00:00 0 
d9100000-140000000 ---p 00000000 00:00 0 
7f7f6fc38000-7f7f70438000 ---p 00000000 00:00 0 
7f7f70438000-7f7f7467e000 rw-p 00000000 00:00 0 
7f7f7467e000-7f7f7467f000 ---p 00000000 00:00 0 
7f7f7467f000-7f7f7477f000 rw-p 00000000 00:00 0 
7f7f7477f000-7f7f74780000 ---p 00000000 00:00 0 
7f7f74780000-7f7f74880000 rw-p 00000000 00:00 0 
7f7f74880000-7f7f74881000 ---p 00000000 00:00 0 
7f7f74881000-7f7f74981000 rw-p 00000000 00:00 0 
7f7f74981000-7f7f74982000 ---p 00000000 00:00 0 
7f7f74982000-7f7f74aac000 rw-p 00000000 00:00 0 
7f7f74aac000-7f7f74d1c000 ---p 00000000 00:00 0 
7f7f74d1c000-7f7f74d46000 rw-p 00000000 00:00 0 
7f7f74d46000-7f7f74fb5000 ---p 00000000 00:00 0 
7f7f74fb5000-7f7f74fcb000 rw-p 00000000 00:00 0 
7f7f74fcb000-7f7f75102000 ---p 00000000 00:00 0 
7f7f75102000-7f7f7510d000 rw-p 00000000 00:00 0 
7f7f7510d000-7f7f754c3000 ---p 00000000 00:00 0 
7f7f754c3000-7f7f75733000 rwxp 00000000 00:00 0 
7f7f75733000-7f7f844c3000 ---p 00000000 00:00 0 
7f7f844c3000-7f7f844de000 r-xp 00000000 ca:06 18456592                   /usr/java/jdk1.8.0_201/jre/lib/amd64/libzip.so
7f7f844de000-7f7f846dd000 ---p 0001b000 ca:06 18456592                   /usr/java/jdk1.8.0_201/jre/lib/amd64/libzip.so
7f7f846dd000-7f7f846de000 r--p 0001a000 ca:06 18456592                   /usr/java/jdk1.8.0_201/jre/lib/amd64/libzip.so
7f7f846de000-7f7f846df000 rw-p 0001b000 ca:06 18456592                   /usr/java/jdk1.8.0_201/jre/lib/amd64/libzip.so
7f7f846df000-7f7f846eb000 r-xp 00000000 ca:06 14442521                   /lib/libnss_files-2.11.3.so
7f7f846eb000-7f7f848ea000 ---p 0000c000 ca:06 14442521                   /lib/libnss_files-2.11.3.so
7f7f848ea000-7f7f848eb000 r--p 0000b000 ca:06 14442521                   /lib/libnss_files-2.11.3.so
7f7f848eb000-7f7f848ec000 rw-p 0000c000 ca:06 14442521                   /lib/libnss_files-2.11.3.so
7f7f848ec000-7f7f848f6000 r-xp 00000000 ca:06 14442519                   /lib/libnss_nis-2.11.3.so
7f7f848f6000-7f7f84af5000 ---p 0000a000 ca:06 14442519                   /lib/libnss_nis-2.11.3.so
7f7f84af5000-7f7f84af6000 r--p 00009000 ca:06 14442519                   /lib/libnss_nis-2.11.3.so
7f7f84af6000-7f7f84af7000 rw-p 0000a000 ca:06 14442519                   /lib/libnss_nis-2.11.3.so
7f7f84af7000-7f7f84b0c000 r-xp 00000000 ca:06 14442515                   /lib/libnsl-2.11.3.so
7f7f84b0c000-7f7f84d0b000 ---p 00015000 ca:06 14442515                   /lib/libnsl-2.11.3.so
7f7f84d0b000-7f7f84d0c000 r--p 00014000 ca:06 14442515                   /lib/libnsl-2.11.3.so
7f7f84d0c000-7f7f84d0d000 rw-p 00015000 ca:06 14442515                   /lib/libnsl-2.11.3.so
7f7f84d0d000-7f7f84d0f000 rw-p 00000000 00:00 0 
7f7f84d0f000-7f7f84d16000 r-xp 00000000 ca:06 14442505                   /lib/libnss_compat-2.11.3.so
7f7f84d16000-7f7f84f15000 ---p 00007000 ca:06 14442505                   /lib/libnss_compat-2.11.3.so
7f7f84f15000-7f7f84f16000 r--p 00006000 ca:06 14442505                   /lib/libnss_compat-2.11.3.so
7f7f84f16000-7f7f84f17000 rw-p 00007000 ca:06 14442505                   /lib/libnss_compat-2.11.3.so
7f7f84f17000-7f7f84f43000 r-xp 00000000 ca:06 18456586                   /usr/java/jdk1.8.0_201/jre/lib/amd64/libjava.so
7f7f84f43000-7f7f85143000 ---p 0002c000 ca:06 18456586                   /usr/java/jdk1.8.0_201/jre/lib/amd64/libjava.so
7f7f85143000-7f7f85144000 r--p 0002c000 ca:06 18456586                   /usr/java/jdk1.8.0_201/jre/lib/amd64/libjava.so
7f7f85144000-7f7f85146000 rw-p 0002d000 ca:06 18456586                   /usr/java/jdk1.8.0_201/jre/lib/amd64/libjava.so
7f7f85146000-7f7f85153000 r-xp 00000000 ca:06 18456591                   /usr/java/jdk1.8.0_201/jre/lib/amd64/libverify.so
7f7f85153000-7f7f85352000 ---p 0000d000 ca:06 18456591                   /usr/java/jdk1.8.0_201/jre/lib/amd64/libverify.so
7f7f85352000-7f7f85354000 r--p 0000c000 ca:06 18456591                   /usr/java/jdk1.8.0_201/jre/lib/amd64/libverify.so
7f7f85354000-7f7f85355000 rw-p 0000e000 ca:06 18456591                   /usr/java/jdk1.8.0_201/jre/lib/amd64/libverify.so
7f7f85355000-7f7f8535c000 r-xp 00000000 ca:06 14442518                   /lib/librt-2.11.3.so
7f7f8535c000-7f7f8555b000 ---p 00007000 ca:06 14442518                   /lib/librt-2.11.3.so
7f7f8555b000-7f7f8555c000 r--p 00006000 ca:06 14442518                   /lib/librt-2.11.3.so
7f7f8555c000-7f7f8555d000 rw-p 00007000 ca:06 14442518                   /lib/librt-2.11.3.so
7f7f8555d000-7f7f855dd000 r-xp 00000000 ca:06 14442509                   /lib/libm-2.11.3.so
7f7f855dd000-7f7f857dd000 ---p 00080000 ca:06 14442509                   /lib/libm-2.11.3.so
7f7f857dd000-7f7f857de000 r--p 00080000 ca:06 14442509                   /lib/libm-2.11.3.so
7f7f857de000-7f7f857df000 rw-p 00081000 ca:06 14442509                   /lib/libm-2.11.3.so
7f7f857df000-7f7f864c7000 r-xp 00000000 ca:06 16728601                   /usr/java/jdk1.8.0_201/jre/lib/amd64/server/libjvm.so
7f7f864c7000-7f7f866c6000 ---p 00ce8000 ca:06 16728601                   /usr/java/jdk1.8.0_201/jre/lib/amd64/server/libjvm.so
7f7f866c6000-7f7f8675c000 r--p 00ce7000 ca:06 16728601                   /usr/java/jdk1.8.0_201/jre/lib/amd64/server/libjvm.so
7f7f8675c000-7f7f8678d000 rw-p 00d7d000 ca:06 16728601                   /usr/java/jdk1.8.0_201/jre/lib/amd64/server/libjvm.so
7f7f8678d000-7f7f867c8000 rw-p 00000000 00:00 0 
7f7f867c8000-7f7f86921000 r-xp 00000000 ca:06 14442502                   /lib/libc-2.11.3.so
7f7f86921000-7f7f86b20000 ---p 00159000 ca:06 14442502                   /lib/libc-2.11.3.so
7f7f86b20000-7f7f86b24000 r--p 00158000 ca:06 14442502                   /lib/libc-2.11.3.so
7f7f86b24000-7f7f86b25000 rw-p 0015c000 ca:06 14442502                   /lib/libc-2.11.3.so
7f7f86b25000-7f7f86b2a000 rw-p 00000000 00:00 0 
7f7f86b2a000-7f7f86b2c000 r-xp 00000000 ca:06 14442520                   /lib/libdl-2.11.3.so
7f7f86b2c000-7f7f86d2c000 ---p 00002000 ca:06 14442520                   /lib/libdl-2.11.3.so
7f7f86d2c000-7f7f86d2d000 r--p 00002000 ca:06 14442520                   /lib/libdl-2.11.3.so
7f7f86d2d000-7f7f86d2e000 rw-p 00003000 ca:06 14442520                   /lib/libdl-2.11.3.so
7f7f86d2e000-7f7f86d45000 r-xp 00000000 ca:06 16704901                   /usr/java/jdk1.8.0_201/lib/amd64/jli/libjli.so
7f7f86d45000-7f7f86f44000 ---p 00017000 ca:06 16704901                   /usr/java/jdk1.8.0_201/lib/amd64/jli/libjli.so
7f7f86f44000-7f7f86f45000 r--p 00016000 ca:06 16704901                   /usr/java/jdk1.8.0_201/lib/amd64/jli/libjli.so
7f7f86f45000-7f7f86f46000 rw-p 00017000 ca:06 16704901                   /usr/java/jdk1.8.0_201/lib/amd64/jli/libjli.so
7f7f86f46000-7f7f86f5d000 r-xp 00000000 ca:06 14442508                   /lib/libpthread-2.11.3.so
7f7f86f5d000-7f7f8715c000 ---p 00017000 ca:06 14442508                   /lib/libpthread-2.11.3.so
7f7f8715c000-7f7f8715d000 r--p 00016000 ca:06 14442508                   /lib/libpthread-2.11.3.so
7f7f8715d000-7f7f8715e000 rw-p 00017000 ca:06 14442508                   /lib/libpthread-2.11.3.so
7f7f8715e000-7f7f87162000 rw-p 00000000 00:00 0 
7f7f87162000-7f7f87180000 r-xp 00000000 ca:06 14442498                   /lib/ld-2.11.3.so
7f7f87229000-7f7f87250000 rw-p 00000000 00:00 0 
7f7f87250000-7f7f87253000 ---p 00000000 00:00 0 
7f7f87253000-7f7f87355000 rw-p 00000000 00:00 0 
7f7f87372000-7f7f8737a000 rw-s 00000000 ca:06 8848000                    /tmp/hsperfdata_ems/1348
7f7f8737a000-7f7f8737b000 rw-p 00000000 00:00 0 
7f7f8737b000-7f7f8737c000 r--p 00000000 00:00 0 
7f7f8737c000-7f7f8737f000 rw-p 00000000 00:00 0 
7f7f8737f000-7f7f87380000 r--p 0001d000 ca:06 14442498                   /lib/ld-2.11.3.so
7f7f87380000-7f7f87381000 rw-p 0001e000 ca:06 14442498                   /lib/ld-2.11.3.so
7f7f87381000-7f7f87382000 rw-p 00000000 00:00 0 
7fff8db9f000-7fff8dbb5000 rw-p 00000000 00:00 0                          [stack]
7fff8dbff000-7fff8dc00000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]

VM Arguments:
java_command: <unknown>
java_class_path (initial): .:/users/ems/websophic/tomcat/lib:/users/ems/websophic/zookeeper/lib:/users/oracle/ora11/jlib:/users/oracle/ora11/product/jlib:/users/oracle/jre/1.1.7:/users/mysql/bin:/users/mysql/lib
Launcher Type: SUN_STANDARD

Environment Variables:
JAVA_HOME=/usr/java/jdk1.8.0_201
CLASSPATH=.:/users/ems/websophic/tomcat/lib:/users/ems/websophic/zookeeper/lib:/users/oracle/ora11/jlib:/users/oracle/ora11/product/jlib:/users/oracle/jre/1.1.7:/users/mysql/bin:/users/mysql/lib
PATH=/usr/java/jdk1.8.0_201/bin:/users/ems/websophic/tomcat/bin:/users/ems/websophic/zookeeper/bin:/usr/local/ant/bin:/users/ems/pcs9700/deployment/bin/libs:/users/ems/pcs9700/deployment/bin/inta:/users/ems/pcs9700/deployment/bin/WaveEx:/users/ems/pcs9700/deployment/bin/base:/users/ems/pcs9700/deployment/bin/plat:/users/ems/pcs9700/deployment/bin/his/:/users/ems/pcs9700/deployment/bin/mmi:/users/ems/pcs9700/deployment/bin/scada:/users/ems/pcs9700/deployment/bin/fe:/users/ems/pcs9700/deployment/bin/fe/plugin:/users/ems/pcs9700/deployment/bin/fe/fe_61850:/users/ems/pcs9700/deployment/bin/pas:/users/ems/pcs9700/deployment/bin/alarm:/users/ems/pcs9700/deployment/bin/app:/users/ems/pcs9700/deployment/bin/pdr:/users/ems/pcs9700/deployment/bin/pdrmgr:/users/ems/pcs9700/deployment/bin/graph:/users/ems/pcs9700/deployment/bin/report:/users/ems/pcs9700/deployment/bin/rule:/users/ems/pcs9700/deployment/bin/dbtools:/users/ems/pcs9700/deployment/bin/relay:/users/ems/pcs9700/deployment/bin/optab:/users/ems/pcs9700/deployment/bin/scontrol:/users/ems/pcs9700/deployment/bin/grd:/users/ems/pcs9700/deployment/bin/web:/users/ems/pcs9700/deployment/bin/remote:/users/ems/pcs9700/deployment/bin/pst:/users/ems/pcs9700/deployment/bin/pst/plugins:/usr/local/Trolltech/Qt-4.8.7/bin:/usr/local/Trolltech/Qt-4.8.7/bin/designer:/usr/bin:/sbin:/opt/intel/cce/10.1.017/bin:/opt/intel/idbe/10.1.017/bin:/users/oracle/ora11/bin:/usr/sbin:/users/oracle/ora11/bin:/usr/sbin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
LD_LIBRARY_PATH=/users/ems/liuzp/license_qt4/bin:/users/ems/pcs9700/deployment/bin/relay/WaveEx:/users/ems/pcs9700/deployment/bin/libs:/users/ems/pcs9700/deployment/bin/inta:/users/ems/pcs9700/deployment/bin/WaveEx:/users/ems/pcs9700/deployment/bin/base:/users/ems/pcs9700/deployment/bin/plat:/users/ems/pcs9700/deployment/bin/mmi:/users/ems/pcs9700/deployment/bin/scada:/users/ems/pcs9700/deployment/bin/fe:/users/ems/pcs9700/deployment/bin/fe/plugin:/users/ems/pcs9700/deployment/bin/pas:/users/ems/pcs9700/deployment/bin/alarm:/users/ems/pcs9700/deployment/bin/app:/users/ems/pcs9700/deployment/bin/graph:/users/ems/pcs9700/deployment/bin/report:/users/ems/pcs9700/deployment/bin/pdr:/users/ems/pcs9700/deployment/bin/pdrmgr:/users/ems/pcs9700/deployment/bin/rule:/users/ems/pcs9700/deployment/bin/dbtools:/users/ems/pcs9700/deployment/bin/relay:/users/ems/pcs9700/deployment/bin/fe/fe_61850:/users/ems/pcs9700/deployment/bin/his:/users/ems/pcs9700/deployment/bin/optab:/users/ems/pcs9700/deployment/bin/scontrol:/users/ems/pcs9700/deployment/bin/grd:/users/ems/pcs9700/deployment/bin/web:/users/ems/pcs9700/deployment/bin/remote:/users/ems/pcs9700/deployment/bin/pst:/users/ems/pcs9700/deployment/bin/pst/plugins:/users/ems/ActiveMQ/APR/lib:/users/ems/ActiveMQ/ActiveMQ_CPP/lib:/usr/local/Trolltech/Qt-4.8.7/lib:/users/ems/ACE_wrappers/ace:/opt/intel/cce/10.1.017/lib:/users/mysql/lib:/users/oracle/ora11/lib
SHELL=/bin/tcsh
HOSTTYPE=x86_64-linux
OSTYPE=linux
MACHTYPE=x86_64

Signal Handlers:
SIGSEGV: [libjvm.so+0xad3e60], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGBUS: [libjvm.so+0xad3e60], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGFPE: [libjvm.so+0x90b880], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGPIPE: [libjvm.so+0x90b880], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGXFSZ: [libjvm.so+0x90b880], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGILL: [libjvm.so+0x90b880], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGUSR1: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
SIGUSR2: [libjvm.so+0x90b750], sa_mask[0]=00000000000000000000000000000000, sa_flags=SA_RESTART|SA_SIGINFO
SIGHUP: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
SIGINT: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
SIGTERM: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
SIGQUIT: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none


---------------  S Y S T E M  ---------------

OS:DISTRIB_ID=Linx
DISTRIB_RELEASE=6.0
DISTRIB_CODENAME=tang
DISTRIB_DESCRIPTION="Linx 6.0.60.4"

uname:Linux 2.6.32-5-linx-amd64 #1 SMP Mon May 18 04:02:09 UTC 2015 x86_64
libc:glibc 2.11.3 NPTL 2.11.3 
rlimit: STACK 8192k, CORE 521000k, NPROC 10000, NOFILE 65536, AS infinity
load average:0.10 0.13 0.20

/proc/meminfo:
MemTotal:        8175204 kB
MemFree:         3178756 kB
Buffers:          116704 kB
Cached:          1067580 kB
SwapCached:         2072 kB
Active:          2940716 kB
Inactive:        1551836 kB
Active(anon):    2381488 kB
Inactive(anon):   929380 kB
Active(file):     559228 kB
Inactive(file):   622456 kB
Unevictable:        2500 kB
Mlocked:            2500 kB
SwapTotal:      15624184 kB
SwapFree:       15551212 kB
Dirty:              4336 kB
Writeback:             0 kB
AnonPages:       3308716 kB
Mapped:           556328 kB
Shmem:               900 kB
Slab:             137660 kB
SReclaimable:     102892 kB
SUnreclaim:        34768 kB
KernelStack:       14040 kB
PageTables:       156040 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    15951192 kB
Committed_AS:   15837036 kB
VmallocTotal:   34359738367 kB
VmallocUsed:       49112 kB
VmallocChunk:   34359684168 kB
HardwareCorrupted:     0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:        8188 kB
DirectMap2M:     8372224 kB


CPU:total 4 (initial active 4) (32 cores per cpu, 2 threads per core) family 6 model 62 stepping 4, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, avx, aes, clmul, erms, ht, tsc

/proc/cpuinfo:
processor   : 0
vendor_id   : GenuineIntel
cpu family  : 6
model       : 62
model name  : Intel(R) Xeon(R) CPU E5-2658 v2 @ 2.40GHz
stepping    : 4
cpu MHz     : 2400.062
cache size  : 25600 KB
physical id : 0
siblings    : 4
core id     : 0
cpu cores   : 4
apicid      : 0
initial apicid  : 0
fpu     : yes
fpu_exception   : yes
cpuid level : 13
wp      : yes
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx rdtscp lm constant_tsc rep_good pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic popcnt aes xsave avx hypervisor lahf_lm
bogomips    : 4800.12
clflush size    : 64
cache_alignment : 64
address sizes   : 46 bits physical, 48 bits virtual
power management:

processor   : 1
vendor_id   : GenuineIntel
cpu family  : 6
model       : 62
model name  : Intel(R) Xeon(R) CPU E5-2658 v2 @ 2.40GHz
stepping    : 4
cpu MHz     : 2400.062
cache size  : 25600 KB
physical id : 0
siblings    : 4
core id     : 1
cpu cores   : 4
apicid      : 2
initial apicid  : 2
fpu     : yes
fpu_exception   : yes
cpuid level : 13
wp      : yes
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx rdtscp lm constant_tsc rep_good pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic popcnt aes xsave avx hypervisor lahf_lm
bogomips    : 731.73
clflush size    : 64
cache_alignment : 64
address sizes   : 46 bits physical, 48 bits virtual
power management:

processor   : 2
vendor_id   : GenuineIntel
cpu family  : 6
model       : 62
model name  : Intel(R) Xeon(R) CPU E5-2658 v2 @ 2.40GHz
stepping    : 4
cpu MHz     : 2400.062
cache size  : 25600 KB
physical id : 0
siblings    : 4
core id     : 2
cpu cores   : 4
apicid      : 4
initial apicid  : 4
fpu     : yes
fpu_exception   : yes
cpuid level : 13
wp      : yes
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx rdtscp lm constant_tsc rep_good pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic popcnt aes xsave avx hypervisor lahf_lm
bogomips    : 736.35
clflush size    : 64
cache_alignment : 64
address sizes   : 46 bits physical, 48 bits virtual
power management:

processor   : 3
vendor_id   : GenuineIntel
cpu family  : 6
model       : 62
model name  : Intel(R) Xeon(R) CPU E5-2658 v2 @ 2.40GHz
stepping    : 4
cpu MHz     : 2400.062
cache size  : 25600 KB
physical id : 0
siblings    : 4
core id     : 3
cpu cores   : 4
apicid      : 6
initial apicid  : 6
fpu     : yes
fpu_exception   : yes
cpuid level : 13
wp      : yes
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx rdtscp lm constant_tsc rep_good pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic popcnt aes xsave avx hypervisor lahf_lm
bogomips    : 728.50
clflush size    : 64
cache_alignment : 64
address sizes   : 46 bits physical, 48 bits virtual
power management:



Memory: 4k page, physical 8175204k(3178756k free), swap 15624184k(15551212k free)

vm_info: Java HotSpot(TM) 64-Bit Server VM (25.201-b09) for linux-amd64 JRE (1.8.0_201-b09), built on Dec 15 2018 11:35:59 by "java_re" with gcc 7.3.0

time: Fri Mar 29 08:19:32 2019
timezone: CST
elapsed time: 0 seconds (0d 0h 0m 0s)

------- The Output Of Strace ---------

The command strace -f java -version's output:

...
20359 clock_gettime(CLOCK_MONOTONIC, {4061888, 398184460}) = 0
20359 clock_gettime(CLOCK_MONOTONIC, {4061888, 398223171}) = 0
20359 open("/usr/java/jdk1.8.0_201/jre/lib/amd64/libzip.so", O_RDONLY) = 3
20359 fstat(3, {st_mode=S_IFREG|0755, st_size=127016, ...}) = 0
20359 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff2868e4000
20359 read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\20(\0\0\0\0\0\0"..., 4096) = 4096
20359 lseek(3, 122880, SEEK_SET)        = 122880
20359 read(3, "ffer\0Java_java_util_zip_ZipFile_"..., 4096) = 4096
20359 lseek(3, 0, SEEK_SET)             = 0
20359 read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\20(\0\0\0\0\0\0"..., 4096) = 4096
20359 read(3, "\2601\0\0\0\0\0\0000\0\0\0\0\0\0\0\222\7\0\0\22\0\r\0\220?\0\0\0\0\0\0"..., 4096) = 4096
20359 lseek(3, 122880, SEEK_SET)        = 122880
20359 read(3, "ffer\0Java_java_util_zip_ZipFile_"..., 4096) = 4096
20359 lseek(3, 4096, SEEK_SET)          = 4096
20359 read(3, "\2601\0\0\0\0\0\0000\0\0\0\0\0\0\0\222\7\0\0\22\0\r\0\220?\0\0\0\0\0\0"..., 4096) = 4096
20359 lseek(3, 122880, SEEK_SET)        = 122880
20359 read(3, "ffer\0Java_java_util_zip_ZipFile_"..., 4096) = 4096
20359 lseek(3, 110592, SEEK_SET)        = 110592
20359 read(3, "\220\255!\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\306$\0\0\0\0\0\0"..., 4096) = 4096
20359 read(3, " w\0\0\0\0\0\0 \0\0\0\0\0\0\0\341\6\0\0\2\0\r\0\3201\1\0\0\0\0\0"..., 4096) = 4096
20359 read(3, "\352\22\0\0\22\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\v\23\0\0\22\0\r\0"..., 4096) = 4096
20359 lseek(3, 122880, SEEK_SET)        = 122880
20359 read(3, "ffer\0Java_java_util_zip_ZipFile_"..., 4096) = 4096
20359 lseek(3, 118784, SEEK_SET)        = 118784
20359 read(3, "\352\22\0\0\22\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\v\23\0\0\22\0\r\0"..., 4096) = 4096
20359 read(3, "ffer\0Java_java_util_zip_ZipFile_"..., 4096) = 4096
20359 lseek(3, 126976, SEEK_SET)        = 126976
20359 read(3, "\252\347\1\0\0\0\0\0\372\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\1\0\0\0\0\0\0\0"..., 4096) = 40
20359 lseek(3, 122880, SEEK_SET)        = 122880
20359 read(3, "ffer\0Java_java_util_zip_ZipFile_"..., 4096) = 4096
20359 lseek(3, 126976, SEEK_SET)        = 126976
20359 read(3, "\252\347\1\0\0\0\0\0\372\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\1\0\0\0\0\0\0\0"..., 4096) = 40
20359 lseek(3, 0, SEEK_SET)             = 0
20359 read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\20(\0\0\0\0\0\0"..., 4096) = 4096
20359 close(3)                          = 0
20359 munmap(0x7ff2868e4000, 4096)      = 0
20359 open("/usr/java/jdk1.8.0_201/jre/lib/amd64/libzip.so", O_RDONLY) = 3
20359 read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\20(\0\0\0\0\0\0"..., 832) = 832
20359 fstat(3, {st_mode=S_IFREG|0755, st_size=127016, ...}) = 0
20359 mmap(NULL, 2208344, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7ff283a36000
20359 mprotect(0x7ff283a51000, 2093056, PROT_NONE) = 0
20359 mmap(0x7ff283c50000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1a000) = 0x7ff283c50000
20359 close(3)                          = 0
20359 mprotect(0x7ff283c50000, 4096, PROT_READ) = 0
20359 stat("/usr/java/jdk1.8.0_201/jre/lib/resources.jar", {st_mode=S_IFREG|0644, st_size=3505277, ...}) = 0
20359 stat("/usr/java/jdk1.8.0_201/jre/lib/rt.jar", {st_mode=S_IFREG|0644, st_size=66356161, ...}) = 0
20359 stat("/usr/java/jdk1.8.0_201/jre/lib/sunrsasign.jar", 0x7ff2868c2ab0) = -1 ENOENT (No such file or directory)
20359 stat("/usr/java/jdk1.8.0_201/jre/lib/jsse.jar", {st_mode=S_IFREG|0644, st_size=675599, ...}) = 0
20359 stat("/usr/java/jdk1.8.0_201/jre/lib/jce.jar", {st_mode=S_IFREG|0644, st_size=115646, ...}) = 0
20359 stat("/usr/java/jdk1.8.0_201/jre/lib/charsets.jar", {st_mode=S_IFREG|0644, st_size=3135615, ...}) = 0
20359 stat("/usr/java/jdk1.8.0_201/jre/lib/jfr.jar", {st_mode=S_IFREG|0644, st_size=560713, ...}) = 0
20359 stat("/usr/java/jdk1.8.0_201/jre/classes", 0x7ff2868c2ab0) = -1 ENOENT (No such file or directory)
20359 open("/usr/java/jdk1.8.0_201/jre/lib/meta-index", O_RDONLY) = 3
20359 fstat(3, {st_mode=S_IFREG|0644, st_size=2034, ...}) = 0
20359 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff2868e4000
20359 read(3, "% VERSION 2\n% WARNING: this file"..., 4096) = 2034
20359 read(3, "", 4096)                 = 0
20359 close(3)                          = 0
20359 munmap(0x7ff2868e4000, 4096)      = 0
20359 mmap(NULL, 251658240, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7ff274a36000
20359 mmap(0x7ff274a36000, 2555904, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7ff274a36000
20359 mmap(NULL, 3932160, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7ff274676000
20359 mmap(0x7ff274676000, 40960, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7ff274676000
20359 --- SIGSEGV (Segmentation fault) @ 0 (0) ---
20359 rt_sigreturn(0x7ff2868ed000)      = 6
20359 clock_gettime(CLOCK_MONOTONIC, {4061888, 402566821}) = 0
20359 clock_gettime(CLOCK_MONOTONIC, {4061888, 402611035}) = 0
20359 mmap(0x83400000, 2092957696, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x83400000
20359 mmap(NULL, 4091904, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7ff27428f000
20359 mmap(0x7ff274675000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7ff274675000
20359 mmap(0xd6700000, 44040192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Cannot allocate memory)
20359 write(2, "Java HotSpot(TM) 64-Bit Server V"..., 43) = 43
20359 write(2, "INFO: os::commit_memory(0x000000"..., 106) = 106
...
like image 721
1nfinity Avatar asked Mar 29 '19 08:03

1nfinity


People also ask

How much memory should I allocate to JVM?

When allocating memory to a JVM, 10% to 25% of the memory is for the JVM process itself and the remainder is for the code running in the JVM. This means that a JVM that has been allocated 4 GB may only have 3.5 GB available to the code running in the JVM. This resource memory used by the JVM is often called overhead.

How do I fix Java memory error?

Easy way to solve OutOfMemoryError in java is to increase the maximum heap size by using JVM options -Xmx512M , this will immediately solve your OutOfMemoryError.

What happens when JVM run out of memory?

OutOfMemoryError is a runtime error in Java which occurs when the Java Virtual Machine (JVM) is unable to allocate an object due to insufficient space in the Java heap. The Java Garbage Collector (GC) cannot free up the space required for a new object, which causes a java. lang.


1 Answers

It seems you have overcommit disabled:

vm.overcommit_memory = 2

And the amount of currently allocated memory on the system has almost reached the limit. You can see it in /proc/meminfo section of the crash log:

CommitLimit:    15951192 kB
Committed_AS:   15837036 kB

The solution is to enable heuristic overcommit handling:

sudo sysctl vm.overcommit_memory=0

or to increase overcommit ratio:

sudo sysctl vm.overcommit_ratio=100

For more information see overcommit-accounting and man 5 proc.

like image 85
apangin Avatar answered Oct 20 '22 07:10

apangin