Why the operating system says it can't allocate memory to jvm when it has enough memory

Question

I'm trying to start a new jvm with the command: java -version, but it report a error:

# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 160088 bytes for AllocateHeap
# An error report file with more information is saved as:
# /users/xxx/xxx/hs_err_pid12365.log

Then I checked file /users/xxx/xxx/hs_err_pid12365.log, and I find out it say:

# Native memory allocation (malloc) failed to allocate 295856 bytes for Chunk::new

And the stack info also proves that the system can't allocate native memory:

V  [libjvm.so+0xad3525]  VMError::report_and_die()+0x2e5
V  [libjvm.so+0x4e0457]  report_vm_out_of_memory(char const*, int, unsigned long, VMErrorType, char const*)+0x67
V  [libjvm.so+0xa9775f]  universe_init()+0x35f
V  [libjvm.so+0x631fd0]  init_globals()+0x50
V  [libjvm.so+0xa79b69]  Threads::create_vm(JavaVMInitArgs*, bool*)+0x409
V  [libjvm.so+0x6d7b4f]  JNI_CreateJavaVM+0x4f
C  [libjli.so+0x7ee4]  JavaMain+0x84
C  [libpthread.so.0+0x68ca]  start_thread+0xba

But when I checked the available memory size using free -m. I find that the system still has enough available memory:

scada1:/users/xxx/xxx> free -m
             total       used       free     shared    buffers     cached
Mem:          7983       4867       3116          0        110       1031
-/+ buffers/cache:       3726       4257
Swap:        15257         71      15186

So why the system can't allocate memory when it has enough available memory

------- System Info ---------

The operating system info is:

scada1:/users/ems> uname -a
Linux scada1 2.6.32-5-linx-amd64 #1 SMP Mon May 18 04:02:09 UTC 2015 x86_64 GNU/Linux

The java version is:

jdk1.8.0_201

------- The hs_err_pidxxx.log Content ---------

#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 160088 bytes for AllocateHeap
# Possible reasons:
#   The system is out of physical RAM or swap space
#   The process is running with CompressedOops enabled, and the Java Heap may be blocking the growth of the native heap
# Possible solutions:
#   Reduce memory load on the system
#   Increase physical memory or swap space
#   Check if swap backing store is full
#   Decrease Java heap size (-Xmx/-Xms)
#   Decrease number of Java threads
#   Decrease Java thread stack sizes (-Xss)
#   Set larger code cache with -XX:ReservedCodeCacheSize=
#   JVM is running with Unscaled Compressed Oops mode in which the Java heap is
#     placed in the first 4GB address space. The Java Heap base address is the
#     maximum limit for the native heap growth. Please use -XX:HeapBaseMinAddress
#     to set the Java Heap base and to place the Java Heap above 4GB virtual address.
# This output file may be truncated or incomplete.
#
#  Out of Memory Error (allocation.inline.hpp:61), pid=1348, tid=0x00007f7f87350700
#
# JRE version:  (8.0_201-b09) (build )
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.201-b09 mixed mode linux-amd64 compressed oops)
# Core dump written. Default location: /users/ems/zoutairan/core or core.1348 (max size 521000 kB). To ensure a full core dump, try "ulimit -c unlimited" before starting Java again
#

---------------  T H R E A D  ---------------

Current thread (0x000000000107e000):  JavaThread "Unknown thread" [_thread_in_vm, id=1349, stack(0x00007f7f87250000,0x00007f7f87351000)]

Stack: [0x00007f7f87250000,0x00007f7f87351000],  sp=0x00007f7f8734f910,  free space=1022k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0xad3525]  VMError::report_and_die()+0x2e5
V  [libjvm.so+0x4e0457]  report_vm_out_of_memory(char const*, int, unsigned long, VMErrorType, char const*)+0x67
V  [libjvm.so+0xa9775f]  universe_init()+0x35f
V  [libjvm.so+0x631fd0]  init_globals()+0x50
V  [libjvm.so+0xa79b69]  Threads::create_vm(JavaVMInitArgs*, bool*)+0x409
V  [libjvm.so+0x6d7b4f]  JNI_CreateJavaVM+0x4f
C  [libjli.so+0x7ee4]  JavaMain+0x84
C  [libpthread.so.0+0x68ca]  start_thread+0xba


---------------  P R O C E S S  ---------------

Java Threads: ( => current thread )

Other Threads:

=>0x000000000107e000 (exited) JavaThread "Unknown thread" [_thread_in_vm, id=1349, stack(0x00007f7f87250000,0x00007f7f87351000)]

VM state:not at safepoint (not fully initialized)

VM Mutex/Monitor currently owned by a thread: None

heap address: 0x0000000083400000, size: 1996 MB, Compressed Oops mode: 32-bit
Narrow klass base: 0x0000000000000000, Narrow klass shift: 3
Compressed class space size: 1073741824 Address: 0x0000000100000000

GC Heap History (0 events):
No events

Deoptimization events (0 events):
No events

Classes redefined (0 events):
No events

Internal exceptions (0 events):
No events

Events (0 events):
No events


Dynamic libraries:
00400000-00401000 r-xp 00000000 ca:06 16598510                           /usr/java/jdk1.8.0_201/bin/java
00600000-00601000 r--p 00000000 ca:06 16598510                           /usr/java/jdk1.8.0_201/bin/java
00601000-00602000 rw-p 00001000 ca:06 16598510                           /usr/java/jdk1.8.0_201/bin/java
01060000-010b8000 rw-p 00000000 00:00 0                                  [heap]
83400000-88800000 rw-p 00000000 00:00 0 
88800000-d6700000 ---p 00000000 00:00 0 
d6700000-d9100000 rw-p 00000000 00:00 0 
d9100000-140000000 ---p 00000000 00:00 0 
7f7f6fc38000-7f7f70438000 ---p 00000000 00:00 0 
7f7f70438000-7f7f7467e000 rw-p 00000000 00:00 0 
7f7f7467e000-7f7f7467f000 ---p 00000000 00:00 0 
7f7f7467f000-7f7f7477f000 rw-p 00000000 00:00 0 
7f7f7477f000-7f7f74780000 ---p 00000000 00:00 0 
7f7f74780000-7f7f74880000 rw-p 00000000 00:00 0 
7f7f74880000-7f7f74881000 ---p 00000000 00:00 0 
7f7f74881000-7f7f74981000 rw-p 00000000 00:00 0 
7f7f74981000-7f7f74982000 ---p 00000000 00:00 0 
7f7f74982000-7f7f74aac000 rw-p 00000000 00:00 0 
7f7f74aac000-7f7f74d1c000 ---p 00000000 00:00 0 
7f7f74d1c000-7f7f74d46000 rw-p 00000000 00:00 0 
7f7f74d46000-7f7f74fb5000 ---p 00000000 00:00 0 
7f7f74fb5000-7f7f74fcb000 rw-p 00000000 00:00 0 
7f7f74fcb000-7f7f75102000 ---p 00000000 00:00 0 
7f7f75102000-7f7f7510d000 rw-p 00000000 00:00 0 
7f7f7510d000-7f7f754c3000 ---p 00000000 00:00 0 
7f7f754c3000-7f7f75733000 rwxp 00000000 00:00 0 
7f7f75733000-7f7f844c3000 ---p 00000000 00:00 0 
7f7f844c3000-7f7f844de000 r-xp 00000000 ca:06 18456592                   /usr/java/jdk1.8.0_201/jre/lib/amd64/libzip.so
7f7f844de000-7f7f846dd000 ---p 0001b000 ca:06 18456592                   /usr/java/jdk1.8.0_201/jre/lib/amd64/libzip.so
7f7f846dd000-7f7f846de000 r--p 0001a000 ca:06 18456592                   /usr/java/jdk1.8.0_201/jre/lib/amd64/libzip.so
7f7f846de000-7f7f846df000 rw-p 0001b000 ca:06 18456592                   /usr/java/jdk1.8.0_201/jre/lib/amd64/libzip.so
7f7f846df000-7f7f846eb000 r-xp 00000000 ca:06 14442521                   /lib/libnss_files-2.11.3.so
7f7f846eb000-7f7f848ea000 ---p 0000c000 ca:06 14442521                   /lib/libnss_files-2.11.3.so
7f7f848ea000-7f7f848eb000 r--p 0000b000 ca:06 14442521                   /lib/libnss_files-2.11.3.so
7f7f848eb000-7f7f848ec000 rw-p 0000c000 ca:06 14442521                   /lib/libnss_files-2.11.3.so
7f7f848ec000-7f7f848f6000 r-xp 00000000 ca:06 14442519                   /lib/libnss_nis-2.11.3.so
7f7f848f6000-7f7f84af5000 ---p 0000a000 ca:06 14442519                   /lib/libnss_nis-2.11.3.so
7f7f84af5000-7f7f84af6000 r--p 00009000 ca:06 14442519                   /lib/libnss_nis-2.11.3.so
7f7f84af6000-7f7f84af7000 rw-p 0000a000 ca:06 14442519                   /lib/libnss_nis-2.11.3.so
7f7f84af7000-7f7f84b0c000 r-xp 00000000 ca:06 14442515                   /lib/libnsl-2.11.3.so
7f7f84b0c000-7f7f84d0b000 ---p 00015000 ca:06 14442515                   /lib/libnsl-2.11.3.so
7f7f84d0b000-7f7f84d0c000 r--p 00014000 ca:06 14442515                   /lib/libnsl-2.11.3.so
7f7f84d0c000-7f7f84d0d000 rw-p 00015000 ca:06 14442515                   /lib/libnsl-2.11.3.so
7f7f84d0d000-7f7f84d0f000 rw-p 00000000 00:00 0 
7f7f84d0f000-7f7f84d16000 r-xp 00000000 ca:06 14442505                   /lib/libnss_compat-2.11.3.so
7f7f84d16000-7f7f84f15000 ---p 00007000 ca:06 14442505                   /lib/libnss_compat-2.11.3.so
7f7f84f15000-7f7f84f16000 r--p 00006000 ca:06 14442505                   /lib/libnss_compat-2.11.3.so
7f7f84f16000-7f7f84f17000 rw-p 00007000 ca:06 14442505                   /lib/libnss_compat-2.11.3.so
7f7f84f17000-7f7f84f43000 r-xp 00000000 ca:06 18456586                   /usr/java/jdk1.8.0_201/jre/lib/amd64/libjava.so
7f7f84f43000-7f7f85143000 ---p 0002c000 ca:06 18456586                   /usr/java/jdk1.8.0_201/jre/lib/amd64/libjava.so
7f7f85143000-7f7f85144000 r--p 0002c000 ca:06 18456586                   /usr/java/jdk1.8.0_201/jre/lib/amd64/libjava.so
7f7f85144000-7f7f85146000 rw-p 0002d000 ca:06 18456586                   /usr/java/jdk1.8.0_201/jre/lib/amd64/libjava.so
7f7f85146000-7f7f85153000 r-xp 00000000 ca:06 18456591                   /usr/java/jdk1.8.0_201/jre/lib/amd64/libverify.so
7f7f85153000-7f7f85352000 ---p 0000d000 ca:06 18456591                   /usr/java/jdk1.8.0_201/jre/lib/amd64/libverify.so
7f7f85352000-7f7f85354000 r--p 0000c000 ca:06 18456591                   /usr/java/jdk1.8.0_201/jre/lib/amd64/libverify.so
7f7f85354000-7f7f85355000 rw-p 0000e000 ca:06 18456591                   /usr/java/jdk1.8.0_201/jre/lib/amd64/libverify.so
7f7f85355000-7f7f8535c000 r-xp 00000000 ca:06 14442518                   /lib/librt-2.11.3.so
7f7f8535c000-7f7f8555b000 ---p 00007000 ca:06 14442518                   /lib/librt-2.11.3.so
7f7f8555b000-7f7f8555c000 r--p 00006000 ca:06 14442518                   /lib/librt-2.11.3.so
7f7f8555c000-7f7f8555d000 rw-p 00007000 ca:06 14442518                   /lib/librt-2.11.3.so
7f7f8555d000-7f7f855dd000 r-xp 00000000 ca:06 14442509                   /lib/libm-2.11.3.so
7f7f855dd000-7f7f857dd000 ---p 00080000 ca:06 14442509                   /lib/libm-2.11.3.so
7f7f857dd000-7f7f857de000 r--p 00080000 ca:06 14442509                   /lib/libm-2.11.3.so
7f7f857de000-7f7f857df000 rw-p 00081000 ca:06 14442509                   /lib/libm-2.11.3.so
7f7f857df000-7f7f864c7000 r-xp 00000000 ca:06 16728601                   /usr/java/jdk1.8.0_201/jre/lib/amd64/server/libjvm.so
7f7f864c7000-7f7f866c6000 ---p 00ce8000 ca:06 16728601                   /usr/java/jdk1.8.0_201/jre/lib/amd64/server/libjvm.so
7f7f866c6000-7f7f8675c000 r--p 00ce7000 ca:06 16728601                   /usr/java/jdk1.8.0_201/jre/lib/amd64/server/libjvm.so
7f7f8675c000-7f7f8678d000 rw-p 00d7d000 ca:06 16728601                   /usr/java/jdk1.8.0_201/jre/lib/amd64/server/libjvm.so
7f7f8678d000-7f7f867c8000 rw-p 00000000 00:00 0 
7f7f867c8000-7f7f86921000 r-xp 00000000 ca:06 14442502                   /lib/libc-2.11.3.so
7f7f86921000-7f7f86b20000 ---p 00159000 ca:06 14442502                   /lib/libc-2.11.3.so
7f7f86b20000-7f7f86b24000 r--p 00158000 ca:06 14442502                   /lib/libc-2.11.3.so
7f7f86b24000-7f7f86b25000 rw-p 0015c000 ca:06 14442502                   /lib/libc-2.11.3.so
7f7f86b25000-7f7f86b2a000 rw-p 00000000 00:00 0 
7f7f86b2a000-7f7f86b2c000 r-xp 00000000 ca:06 14442520                   /lib/libdl-2.11.3.so
7f7f86b2c000-7f7f86d2c000 ---p 00002000 ca:06 14442520                   /lib/libdl-2.11.3.so
7f7f86d2c000-7f7f86d2d000 r--p 00002000 ca:06 14442520                   /lib/libdl-2.11.3.so
7f7f86d2d000-7f7f86d2e000 rw-p 00003000 ca:06 14442520                   /lib/libdl-2.11.3.so
7f7f86d2e000-7f7f86d45000 r-xp 00000000 ca:06 16704901                   /usr/java/jdk1.8.0_201/lib/amd64/jli/libjli.so
7f7f86d45000-7f7f86f44000 ---p 00017000 ca:06 16704901                   /usr/java/jdk1.8.0_201/lib/amd64/jli/libjli.so
7f7f86f44000-7f7f86f45000 r--p 00016000 ca:06 16704901                   /usr/java/jdk1.8.0_201/lib/amd64/jli/libjli.so
7f7f86f45000-7f7f86f46000 rw-p 00017000 ca:06 16704901                   /usr/java/jdk1.8.0_201/lib/amd64/jli/libjli.so
7f7f86f46000-7f7f86f5d000 r-xp 00000000 ca:06 14442508                   /lib/libpthread-2.11.3.so
7f7f86f5d000-7f7f8715c000 ---p 00017000 ca:06 14442508                   /lib/libpthread-2.11.3.so
7f7f8715c000-7f7f8715d000 r--p 00016000 ca:06 14442508                   /lib/libpthread-2.11.3.so
7f7f8715d000-7f7f8715e000 rw-p 00017000 ca:06 14442508                   /lib/libpthread-2.11.3.so
7f7f8715e000-7f7f87162000 rw-p 00000000 00:00 0 
7f7f87162000-7f7f87180000 r-xp 00000000 ca:06 14442498                   /lib/ld-2.11.3.so
7f7f87229000-7f7f87250000 rw-p 00000000 00:00 0 
7f7f87250000-7f7f87253000 ---p 00000000 00:00 0 
7f7f87253000-7f7f87355000 rw-p 00000000 00:00 0 
7f7f87372000-7f7f8737a000 rw-s 00000000 ca:06 8848000                    /tmp/hsperfdata_ems/1348
7f7f8737a000-7f7f8737b000 rw-p 00000000 00:00 0 
7f7f8737b000-7f7f8737c000 r--p 00000000 00:00 0 
7f7f8737c000-7f7f8737f000 rw-p 00000000 00:00 0 
7f7f8737f000-7f7f87380000 r--p 0001d000 ca:06 14442498                   /lib/ld-2.11.3.so
7f7f87380000-7f7f87381000 rw-p 0001e000 ca:06 14442498                   /lib/ld-2.11.3.so
7f7f87381000-7f7f87382000 rw-p 00000000 00:00 0 
7fff8db9f000-7fff8dbb5000 rw-p 00000000 00:00 0                          [stack]
7fff8dbff000-7fff8dc00000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]

VM Arguments:
java_command: <unknown>
java_class_path (initial): .:/users/ems/websophic/tomcat/lib:/users/ems/websophic/zookeeper/lib:/users/oracle/ora11/jlib:/users/oracle/ora11/product/jlib:/users/oracle/jre/1.1.7:/users/mysql/bin:/users/mysql/lib
Launcher Type: SUN_STANDARD

Environment Variables:
JAVA_HOME=/usr/java/jdk1.8.0_201
CLASSPATH=.:/users/ems/websophic/tomcat/lib:/users/ems/websophic/zookeeper/lib:/users/oracle/ora11/jlib:/users/oracle/ora11/product/jlib:/users/oracle/jre/1.1.7:/users/mysql/bin:/users/mysql/lib
PATH=/usr/java/jdk1.8.0_201/bin:/users/ems/websophic/tomcat/bin:/users/ems/websophic/zookeeper/bin:/usr/local/ant/bin:/users/ems/pcs9700/deployment/bin/libs:/users/ems/pcs9700/deployment/bin/inta:/users/ems/pcs9700/deployment/bin/WaveEx:/users/ems/pcs9700/deployment/bin/base:/users/ems/pcs9700/deployment/bin/plat:/users/ems/pcs9700/deployment/bin/his/:/users/ems/pcs9700/deployment/bin/mmi:/users/ems/pcs9700/deployment/bin/scada:/users/ems/pcs9700/deployment/bin/fe:/users/ems/pcs9700/deployment/bin/fe/plugin:/users/ems/pcs9700/deployment/bin/fe/fe_61850:/users/ems/pcs9700/deployment/bin/pas:/users/ems/pcs9700/deployment/bin/alarm:/users/ems/pcs9700/deployment/bin/app:/users/ems/pcs9700/deployment/bin/pdr:/users/ems/pcs9700/deployment/bin/pdrmgr:/users/ems/pcs9700/deployment/bin/graph:/users/ems/pcs9700/deployment/bin/report:/users/ems/pcs9700/deployment/bin/rule:/users/ems/pcs9700/deployment/bin/dbtools:/users/ems/pcs9700/deployment/bin/relay:/users/ems/pcs9700/deployment/bin/optab:/users/ems/pcs9700/deployment/bin/scontrol:/users/ems/pcs9700/deployment/bin/grd:/users/ems/pcs9700/deployment/bin/web:/users/ems/pcs9700/deployment/bin/remote:/users/ems/pcs9700/deployment/bin/pst:/users/ems/pcs9700/deployment/bin/pst/plugins:/usr/local/Trolltech/Qt-4.8.7/bin:/usr/local/Trolltech/Qt-4.8.7/bin/designer:/usr/bin:/sbin:/opt/intel/cce/10.1.017/bin:/opt/intel/idbe/10.1.017/bin:/users/oracle/ora11/bin:/usr/sbin:/users/oracle/ora11/bin:/usr/sbin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
LD_LIBRARY_PATH=/users/ems/liuzp/license_qt4/bin:/users/ems/pcs9700/deployment/bin/relay/WaveEx:/users/ems/pcs9700/deployment/bin/libs:/users/ems/pcs9700/deployment/bin/inta:/users/ems/pcs9700/deployment/bin/WaveEx:/users/ems/pcs9700/deployment/bin/base:/users/ems/pcs9700/deployment/bin/plat:/users/ems/pcs9700/deployment/bin/mmi:/users/ems/pcs9700/deployment/bin/scada:/users/ems/pcs9700/deployment/bin/fe:/users/ems/pcs9700/deployment/bin/fe/plugin:/users/ems/pcs9700/deployment/bin/pas:/users/ems/pcs9700/deployment/bin/alarm:/users/ems/pcs9700/deployment/bin/app:/users/ems/pcs9700/deployment/bin/graph:/users/ems/pcs9700/deployment/bin/report:/users/ems/pcs9700/deployment/bin/pdr:/users/ems/pcs9700/deployment/bin/pdrmgr:/users/ems/pcs9700/deployment/bin/rule:/users/ems/pcs9700/deployment/bin/dbtools:/users/ems/pcs9700/deployment/bin/relay:/users/ems/pcs9700/deployment/bin/fe/fe_61850:/users/ems/pcs9700/deployment/bin/his:/users/ems/pcs9700/deployment/bin/optab:/users/ems/pcs9700/deployment/bin/scontrol:/users/ems/pcs9700/deployment/bin/grd:/users/ems/pcs9700/deployment/bin/web:/users/ems/pcs9700/deployment/bin/remote:/users/ems/pcs9700/deployment/bin/pst:/users/ems/pcs9700/deployment/bin/pst/plugins:/users/ems/ActiveMQ/APR/lib:/users/ems/ActiveMQ/ActiveMQ_CPP/lib:/usr/local/Trolltech/Qt-4.8.7/lib:/users/ems/ACE_wrappers/ace:/opt/intel/cce/10.1.017/lib:/users/mysql/lib:/users/oracle/ora11/lib
SHELL=/bin/tcsh
HOSTTYPE=x86_64-linux
OSTYPE=linux
MACHTYPE=x86_64

Signal Handlers:
SIGSEGV: [libjvm.so+0xad3e60], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGBUS: [libjvm.so+0xad3e60], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGFPE: [libjvm.so+0x90b880], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGPIPE: [libjvm.so+0x90b880], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGXFSZ: [libjvm.so+0x90b880], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGILL: [libjvm.so+0x90b880], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGUSR1: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
SIGUSR2: [libjvm.so+0x90b750], sa_mask[0]=00000000000000000000000000000000, sa_flags=SA_RESTART|SA_SIGINFO
SIGHUP: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
SIGINT: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
SIGTERM: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
SIGQUIT: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none


---------------  S Y S T E M  ---------------

OS:DISTRIB_ID=Linx
DISTRIB_RELEASE=6.0
DISTRIB_CODENAME=tang
DISTRIB_DESCRIPTION="Linx 6.0.60.4"

uname:Linux 2.6.32-5-linx-amd64 #1 SMP Mon May 18 04:02:09 UTC 2015 x86_64
libc:glibc 2.11.3 NPTL 2.11.3 
rlimit: STACK 8192k, CORE 521000k, NPROC 10000, NOFILE 65536, AS infinity
load average:0.10 0.13 0.20

/proc/meminfo:
MemTotal:        8175204 kB
MemFree:         3178756 kB
Buffers:          116704 kB
Cached:          1067580 kB
SwapCached:         2072 kB
Active:          2940716 kB
Inactive:        1551836 kB
Active(anon):    2381488 kB
Inactive(anon):   929380 kB
Active(file):     559228 kB
Inactive(file):   622456 kB
Unevictable:        2500 kB
Mlocked:            2500 kB
SwapTotal:      15624184 kB
SwapFree:       15551212 kB
Dirty:              4336 kB
Writeback:             0 kB
AnonPages:       3308716 kB
Mapped:           556328 kB
Shmem:               900 kB
Slab:             137660 kB
SReclaimable:     102892 kB
SUnreclaim:        34768 kB
KernelStack:       14040 kB
PageTables:       156040 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    15951192 kB
Committed_AS:   15837036 kB
VmallocTotal:   34359738367 kB
VmallocUsed:       49112 kB
VmallocChunk:   34359684168 kB
HardwareCorrupted:     0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:        8188 kB
DirectMap2M:     8372224 kB


CPU:total 4 (initial active 4) (32 cores per cpu, 2 threads per core) family 6 model 62 stepping 4, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, avx, aes, clmul, erms, ht, tsc

/proc/cpuinfo:
processor   : 0
vendor_id   : GenuineIntel
cpu family  : 6
model       : 62
model name  : Intel(R) Xeon(R) CPU E5-2658 v2 @ 2.40GHz
stepping    : 4
cpu MHz     : 2400.062
cache size  : 25600 KB
physical id : 0
siblings    : 4
core id     : 0
cpu cores   : 4
apicid      : 0
initial apicid  : 0
fpu     : yes
fpu_exception   : yes
cpuid level : 13
wp      : yes
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx rdtscp lm constant_tsc rep_good pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic popcnt aes xsave avx hypervisor lahf_lm
bogomips    : 4800.12
clflush size    : 64
cache_alignment : 64
address sizes   : 46 bits physical, 48 bits virtual
power management:

processor   : 1
vendor_id   : GenuineIntel
cpu family  : 6
model       : 62
model name  : Intel(R) Xeon(R) CPU E5-2658 v2 @ 2.40GHz
stepping    : 4
cpu MHz     : 2400.062
cache size  : 25600 KB
physical id : 0
siblings    : 4
core id     : 1
cpu cores   : 4
apicid      : 2
initial apicid  : 2
fpu     : yes
fpu_exception   : yes
cpuid level : 13
wp      : yes
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx rdtscp lm constant_tsc rep_good pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic popcnt aes xsave avx hypervisor lahf_lm
bogomips    : 731.73
clflush size    : 64
cache_alignment : 64
address sizes   : 46 bits physical, 48 bits virtual
power management:

processor   : 2
vendor_id   : GenuineIntel
cpu family  : 6
model       : 62
model name  : Intel(R) Xeon(R) CPU E5-2658 v2 @ 2.40GHz
stepping    : 4
cpu MHz     : 2400.062
cache size  : 25600 KB
physical id : 0
siblings    : 4
core id     : 2
cpu cores   : 4
apicid      : 4
initial apicid  : 4
fpu     : yes
fpu_exception   : yes
cpuid level : 13
wp      : yes
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx rdtscp lm constant_tsc rep_good pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic popcnt aes xsave avx hypervisor lahf_lm
bogomips    : 736.35
clflush size    : 64
cache_alignment : 64
address sizes   : 46 bits physical, 48 bits virtual
power management:

processor   : 3
vendor_id   : GenuineIntel
cpu family  : 6
model       : 62
model name  : Intel(R) Xeon(R) CPU E5-2658 v2 @ 2.40GHz
stepping    : 4
cpu MHz     : 2400.062
cache size  : 25600 KB
physical id : 0
siblings    : 4
core id     : 3
cpu cores   : 4
apicid      : 6
initial apicid  : 6
fpu     : yes
fpu_exception   : yes
cpuid level : 13
wp      : yes
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx rdtscp lm constant_tsc rep_good pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic popcnt aes xsave avx hypervisor lahf_lm
bogomips    : 728.50
clflush size    : 64
cache_alignment : 64
address sizes   : 46 bits physical, 48 bits virtual
power management:



Memory: 4k page, physical 8175204k(3178756k free), swap 15624184k(15551212k free)

vm_info: Java HotSpot(TM) 64-Bit Server VM (25.201-b09) for linux-amd64 JRE (1.8.0_201-b09), built on Dec 15 2018 11:35:59 by "java_re" with gcc 7.3.0

time: Fri Mar 29 08:19:32 2019
timezone: CST
elapsed time: 0 seconds (0d 0h 0m 0s)

------- The Output Of Strace ---------

The command strace -f java -version's output:

...
20359 clock_gettime(CLOCK_MONOTONIC, {4061888, 398184460}) = 0
20359 clock_gettime(CLOCK_MONOTONIC, {4061888, 398223171}) = 0
20359 open("/usr/java/jdk1.8.0_201/jre/lib/amd64/libzip.so", O_RDONLY) = 3
20359 fstat(3, {st_mode=S_IFREG|0755, st_size=127016, ...}) = 0
20359 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff2868e4000
20359 read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\20(\0\0\0\0\0\0"..., 4096) = 4096
20359 lseek(3, 122880, SEEK_SET)        = 122880
20359 read(3, "ffer\0Java_java_util_zip_ZipFile_"..., 4096) = 4096
20359 lseek(3, 0, SEEK_SET)             = 0
20359 read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\20(\0\0\0\0\0\0"..., 4096) = 4096
20359 read(3, "\2601\0\0\0\0\0\0000\0\0\0\0\0\0\0\222\7\0\0\22\0\r\0\220?\0\0\0\0\0\0"..., 4096) = 4096
20359 lseek(3, 122880, SEEK_SET)        = 122880
20359 read(3, "ffer\0Java_java_util_zip_ZipFile_"..., 4096) = 4096
20359 lseek(3, 4096, SEEK_SET)          = 4096
20359 read(3, "\2601\0\0\0\0\0\0000\0\0\0\0\0\0\0\222\7\0\0\22\0\r\0\220?\0\0\0\0\0\0"..., 4096) = 4096
20359 lseek(3, 122880, SEEK_SET)        = 122880
20359 read(3, "ffer\0Java_java_util_zip_ZipFile_"..., 4096) = 4096
20359 lseek(3, 110592, SEEK_SET)        = 110592
20359 read(3, "\220\255!\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\306$\0\0\0\0\0\0"..., 4096) = 4096
20359 read(3, " w\0\0\0\0\0\0 \0\0\0\0\0\0\0\341\6\0\0\2\0\r\0\3201\1\0\0\0\0\0"..., 4096) = 4096
20359 read(3, "\352\22\0\0\22\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\v\23\0\0\22\0\r\0"..., 4096) = 4096
20359 lseek(3, 122880, SEEK_SET)        = 122880
20359 read(3, "ffer\0Java_java_util_zip_ZipFile_"..., 4096) = 4096
20359 lseek(3, 118784, SEEK_SET)        = 118784
20359 read(3, "\352\22\0\0\22\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\v\23\0\0\22\0\r\0"..., 4096) = 4096
20359 read(3, "ffer\0Java_java_util_zip_ZipFile_"..., 4096) = 4096
20359 lseek(3, 126976, SEEK_SET)        = 126976
20359 read(3, "\252\347\1\0\0\0\0\0\372\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\1\0\0\0\0\0\0\0"..., 4096) = 40
20359 lseek(3, 122880, SEEK_SET)        = 122880
20359 read(3, "ffer\0Java_java_util_zip_ZipFile_"..., 4096) = 4096
20359 lseek(3, 126976, SEEK_SET)        = 126976
20359 read(3, "\252\347\1\0\0\0\0\0\372\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\1\0\0\0\0\0\0\0"..., 4096) = 40
20359 lseek(3, 0, SEEK_SET)             = 0
20359 read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\20(\0\0\0\0\0\0"..., 4096) = 4096
20359 close(3)                          = 0
20359 munmap(0x7ff2868e4000, 4096)      = 0
20359 open("/usr/java/jdk1.8.0_201/jre/lib/amd64/libzip.so", O_RDONLY) = 3
20359 read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\20(\0\0\0\0\0\0"..., 832) = 832
20359 fstat(3, {st_mode=S_IFREG|0755, st_size=127016, ...}) = 0
20359 mmap(NULL, 2208344, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7ff283a36000
20359 mprotect(0x7ff283a51000, 2093056, PROT_NONE) = 0
20359 mmap(0x7ff283c50000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1a000) = 0x7ff283c50000
20359 close(3)                          = 0
20359 mprotect(0x7ff283c50000, 4096, PROT_READ) = 0
20359 stat("/usr/java/jdk1.8.0_201/jre/lib/resources.jar", {st_mode=S_IFREG|0644, st_size=3505277, ...}) = 0
20359 stat("/usr/java/jdk1.8.0_201/jre/lib/rt.jar", {st_mode=S_IFREG|0644, st_size=66356161, ...}) = 0
20359 stat("/usr/java/jdk1.8.0_201/jre/lib/sunrsasign.jar", 0x7ff2868c2ab0) = -1 ENOENT (No such file or directory)
20359 stat("/usr/java/jdk1.8.0_201/jre/lib/jsse.jar", {st_mode=S_IFREG|0644, st_size=675599, ...}) = 0
20359 stat("/usr/java/jdk1.8.0_201/jre/lib/jce.jar", {st_mode=S_IFREG|0644, st_size=115646, ...}) = 0
20359 stat("/usr/java/jdk1.8.0_201/jre/lib/charsets.jar", {st_mode=S_IFREG|0644, st_size=3135615, ...}) = 0
20359 stat("/usr/java/jdk1.8.0_201/jre/lib/jfr.jar", {st_mode=S_IFREG|0644, st_size=560713, ...}) = 0
20359 stat("/usr/java/jdk1.8.0_201/jre/classes", 0x7ff2868c2ab0) = -1 ENOENT (No such file or directory)
20359 open("/usr/java/jdk1.8.0_201/jre/lib/meta-index", O_RDONLY) = 3
20359 fstat(3, {st_mode=S_IFREG|0644, st_size=2034, ...}) = 0
20359 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff2868e4000
20359 read(3, "% VERSION 2\n% WARNING: this file"..., 4096) = 2034
20359 read(3, "", 4096)                 = 0
20359 close(3)                          = 0
20359 munmap(0x7ff2868e4000, 4096)      = 0
20359 mmap(NULL, 251658240, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7ff274a36000
20359 mmap(0x7ff274a36000, 2555904, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7ff274a36000
20359 mmap(NULL, 3932160, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7ff274676000
20359 mmap(0x7ff274676000, 40960, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7ff274676000
20359 --- SIGSEGV (Segmentation fault) @ 0 (0) ---
20359 rt_sigreturn(0x7ff2868ed000)      = 6
20359 clock_gettime(CLOCK_MONOTONIC, {4061888, 402566821}) = 0
20359 clock_gettime(CLOCK_MONOTONIC, {4061888, 402611035}) = 0
20359 mmap(0x83400000, 2092957696, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x83400000
20359 mmap(NULL, 4091904, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7ff27428f000
20359 mmap(0x7ff274675000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7ff274675000
20359 mmap(0xd6700000, 44040192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Cannot allocate memory)
20359 write(2, "Java HotSpot(TM) 64-Bit Server V"..., 43) = 43
20359 write(2, "INFO: os::commit_memory(0x000000"..., 106) = 106
...

Answer 1

It seems you have overcommit disabled:

vm.overcommit_memory = 2

And the amount of currently allocated memory on the system has almost reached the limit. You can see it in /proc/meminfo section of the crash log:

CommitLimit:    15951192 kB
Committed_AS:   15837036 kB

The solution is to enable heuristic overcommit handling:

sudo sysctl vm.overcommit_memory=0

or to increase overcommit ratio:

sudo sysctl vm.overcommit_ratio=100

For more information see overcommit-accounting and man 5 proc.