JMeter - out of memory on linux

Question

I am trying to run a load test for a application. For this i am using JMeter (v.2.13) on an Ubuntu Vm with 60GB Ram and more than enough CPU power. Goal is to reach 10k Users connected via WebSocket.

However during the test runs i get the following errors on the ssh-console (at approx. 1.5k to 2.5k simulated users)

OpenJDK 64-Bit Server VM warning: Attempt to protect stack guard pages failed.
OpenJDK 64-Bit Server VM warning: Attempt to deallocate stack guard pages failed.
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f20ee653000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 12288 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /jmetertests/jm/bin/hs_err_pid1833.log
OpenJDK 64-Bit Server VM warning: Attempt to deallocate stack guard pages failed.
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f2218de8000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)

The mentioned error report file looks like this

# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 12288 bytes for committing reserved memory.
# Possible reasons:
#   The system is out of physical RAM or swap space
#   In 32 bit mode, the process size limit was hit
# Possible solutions:
#   Reduce memory load on the system
#   Increase physical memory or swap space
#   Check if swap backing store is full
#   Use 64 bit Java on a 64 bit OS
#   Decrease Java heap size (-Xmx/-Xms)
#   Decrease number of Java threads
#   Decrease Java thread stack sizes (-Xss)
#   Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
#  Out of Memory Error (os_linux.cpp:2798), pid=1833, tid=140472285792000
#
# JRE version: OpenJDK Runtime Environment (7.0_75-b13) (build 1.7.0_75-b13)
# Java VM: OpenJDK 64-Bit Server VM (24.75-b04 mixed mode linux-amd64 )
# Derivative: IcedTea 2.5.4
# Distribution: Ubuntu 14.04 LTS, package 7u75-2.5.4-1~trusty1
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again

I tried to allocate more memory by modifying the jmeter.sh through adding the following line right before the java-command is issued in the .sh-file:

JVM_ARGS="-Xms5g -Xmx20g -Xss300k"

I also tried setting the _JAVA_OPTIONS environment variable with the following command

export _JAVA_OPTIONS="-Xms5g -Xmx20g"

And finally I found this command im some SO-thread

sysctl -w vm.max_map_count=500000

The top-Command gives me the following information on memory

KiB Mem:  61836576 total, 15163400 used, 46673176 free,    10636 buffers
KiB Swap:        0 total,        0 used,        0 free.    94492 cached Mem

UPDATE

FYI: I do not use any listeners in JMeter except the simple data writer. However errors are thrown even if I disable that last listener.

java -Xms40g -version

is succesful - so i can really allocate so much memory

I reduced the stack size by using

-Xss300k

which helped ot at least changed something as I now get a

Uncaught Exception java.lang.OutOfMemoryError:
unable to create new native thread. See log file for details.

error. It seems that the number of threads on the system are exhausted?

Update2

as requested by some user the results uf ulimit -a

core file size          (blocks, -c) 0
scheduling priority             (-e) 0
pending signals                 (-i) 491456
max locked memory       (kbytes, -l) 64
open files                      (-n) 500000
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
max user processes              (-u) 491456
... all other entries are set to 'unlimited'

and the limits.conf contains the following entries

*         hard    nofile      900000
*         soft    nofile      900000
root      hard    nofile      900000
root      soft    nofile      900000

Additonally i have set the threads-max under proc/sys/kernel to something ridiculous high and also increased the value of proc/sys/vm/max_map_count

Did I miss something or did i do something wrong? Thank you for your help.

Answer 1

Okay meanwhile i figured it out:

1. make sure to read Dmriti T's post
2. set the java's stacksize to something small - usually the smallest value will be enough. In my case 200k (see also point 4.)
3. increase the maximum number of open files in the sysctl.conf (see point 4)
4. increase heap size - but as jmeter will create a lot of threads also leave enough space for the memory the threads will need on OS level (so a new thread needs memory on OS and java heap - therefore leave the os enough memory too) (see below)
5. update the limits.conf (see below)
6. update the sysctl.conf-file (see below)

limits.conf

/etc/security/limits.conf

*         hard    nofile      900000
*         soft    nofile      900000
root      hard    nofile      900000
root      soft    nofile      900000

sysctl.conf

/etc/sysctl.conf

kernel.pid_max=999999
kernel.thread-max=999999
vm.max_map_count=999999
fs.file-max=999999

attention: is pid_max and thread-max

jmeter.sh / jmeter.bat

you will find those in the /bin/ folder of your jmeter installation.

under linux add the line

JVM_ARGS="-Xmx15g -Xss250k"

somewhere before the actual java call on the last line. This will decrease the stacksize allocated for each thread.

under windows you have to edit the jmeter.bat. I have no configured it under windows but at least regarding the heap there should be a line starting with set HEAP. I dont know where to put the -Xss parameter under windows.

Answer 2

First of all check if you can allocate as much as 50G heap by the following command:

java -Xms50G -version

If it doesn't succeed - try reducing minimum heap size unless you see your Java version information without errors.

As long as you're NOT getting java.lang.OutOfMemoryError: Java heap space error I believe that your heap size tunings are successfully applied.

Perhaps you have a memory-intensive listener enabled, like View Results Tree or View Results in Table? If so - disable them (as well as any other listeners)

If possible try switching to Oracle JDK as at least for Java 6 there were immense performance differences.

Also make sure that you following recommendations from JMeter Performance and Tuning Tips guide.