Yarn container is running out of memory

Question

My yarn container is running out of memory: This specific container runs an Apache-Spark driver node.

The part I don't understand: I am limiting my driver's heap size to 512MB (you can see this in the error message below). But the yarn container is complaining about memory>1GB (Also see message below). You can validate that yarn is launching java is run with Xmx512m. My containers are setup for 1GB memory with .5GB increments. Also my physical machines hosting the yarn containers have 32GB each. I SSH'ed to one of the physical machines and saw that it had alot of free memory...

Another strange thing, is that java is not throwing OutOfMemory exceptions. When I look at the driver logs, I see that eventually it gets a SIGTERM from yarn, and shuts down nicely. If the java process inside Yarn was going over 512MB shouldn't I have gotten an OutOfMemory exception in Java before it ever tried to allocate 1GB from yarn?

I also tried running with a 1024m heap. That time, container crashed with a usage of 1.5GB. This happened consistantly. So clearly the container had the capacity to allocate another 0.5GB beyond the 1GB limit. (quite Logical since the physical machine has 30GB of free memory)

Is there something else inside the YARN container beside java which could be taking up the extra 512MB?

Im running CDH 5.4.1 with Apache spark on Yarn. The java version on the cluster was also upgraded to oracle Java8. I saw some people claiming that the default maxPermSize in java8 has been changed, but I hardly believe that it could take up 512MB...

Yarn error message:

Diagnostics: Container [pid=23335,containerID=container_1453125563779_0160_02_000001] is running beyond physical memory limits. Current usage: 1.0 GB of 1 GB physical memory used; 2.6 GB of 2.1 GB virtual memory used. Killing container.
Dump of the process-tree for container_1453125563779_0160_02_000001 :
    |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
    |- 23335 23333 23335 23335 (bash) 1 0 11767808 432 /bin/bash -c LD_LIBRARY_PATH=/opt/cloudera/parcels/CDH-5.4.1-1.cdh5.4.1.p0.6/lib/hadoop/lib/native::/opt/cloudera/parcels/CDH-5.4.1-1.cdh5.4.1.p0.6/lib/hadoop/lib/native /usr/lib/jvm/java-8-oracle/bin/java -server -Xmx512m -Djava.io.tmpdir=/var/yarn/nm/usercache/hdfs/appcache/application_1453125563779_0160/container_1453125563779_0160_02_000001/tmp '-Dspark.eventLog.enabled=true' '-Dspark.executor.memory=512m' '-Dspark.executor.extraClassPath=/opt/cloudera/parcels/CDH/lib/hbase/lib/htrace-core-3.1.0-incubating.jar' '-Dspark.yarn.am.extraLibraryPath=/opt/cloudera/parcels/CDH-5.4.1-1.cdh5.4.1.p0.6/lib/hadoop/lib/native' '-Dspark.executor.extraLibraryPath=/opt/cloudera/parcels/CDH-5.4.1-1.cdh5.4.1.p0.6/lib/hadoop/lib/native' '-Dspark.shuffle.service.enabled=true' '-Dspark.yarn.jar=local:/opt/cloudera/parcels/CDH-5.4.1-1.cdh5.4.1.p0.6/lib/spark/assembly/lib/spark-assembly-1.3.0-cdh5.4.1-hadoop2.6.0-cdh5.4.1.jar' '-Dspark.app.name=not_telling-1453479057517' '-Dspark.shuffle.service.port=7337' '-Dspark.driver.extraClassPath=/etc/hbase/conf:/opt/cloudera/parcels/CDH/lib/hbase/lib/htrace-core-3.1.0-incubating.jar' '-Dspark.serializer=org.apache.spark.serializer.KryoSerializer' '-Dspark.yarn.historyServer.address=http://XXXX-cdh-dev-cdh-node2:18088' '-Dspark.driver.extraLibraryPath=/opt/cloudera/parcels/CDH-5.4.1-1.cdh5.4.1.p0.6/lib/hadoop/lib/native' '-Dspark.eventLog.dir=hdfs://XXXX-cdh-dev-cdh-node1:8020/user/spark/applicationHistory' '-Dspark.master=yarn-cluster' -Dspark.yarn.app.container.log.dir=/var/log/hadoop-yarn/container/application_1453125563779_0160/container_1453125563779_0160_02_000001 org.apache.spark.deploy.yarn.ApplicationMaster --class 'not_telling' --jar file:/home/cloud-user/temp/not_telling.jar --arg '--conf' --arg 'spark.executor.extraClasspath=/opt/cloudera/parcels/CDH/jars/htrace-core-3.0.4.jar' --executor-memory 512m --executor-cores 4 --num-executors  10 1> /var/log/hadoop-yarn/container/application_1453125563779_0160/container_1453125563779_0160_02_000001/stdout 2> /var/log/hadoop-yarn/container/application_1453125563779_0160/container_1453125563779_0160_02_000001/stderr 
    |- 23338 23335 23335 23335 (java) 95290 10928 2786668544 261830 /usr/lib/jvm/java-8-oracle/bin/java -server -Xmx512m -Djava.io.tmpdir=/var/yarn/nm/usercache/hdfs/appcache/application_1453125563779_0160/container_1453125563779_0160_02_000001/tmp -Dspark.eventLog.enabled=true -Dspark.executor.memory=512m -Dspark.executor.extraClassPath=/opt/cloudera/parcels/CDH/lib/hbase/lib/htrace-core-3.1.0-incubating.jar -Dspark.yarn.am.extraLibraryPath=/opt/cloudera/parcels/CDH-5.4.1-1.cdh5.4.1.p0.6/lib/hadoop/lib/native -Dspark.executor.extraLibraryPath=/opt/cloudera/parcels/CDH-5.4.1-1.cdh5.4.1.p0.6/lib/hadoop/lib/native -Dspark.shuffle.service.enabled=true -Dspark.yarn.jar=local:/opt/cloudera/parcels/CDH-5.4.1-1.cdh5.4.1.p0.6/lib/spark/assembly/lib/spark-assembly-1.3.0-cdh5.4.1-hadoop2.6.0-cdh5.4.1.jar -Dspark.app.name=not_tellin-1453479057517 -Dspark.shuffle.service.port=7337 -Dspark.driver.extraClassPath=/etc/hbase/conf:/opt/cloudera/parcels/CDH/lib/hbase/lib/htrace-core-3.1.0-incubating.jar -Dspark.serializer=org.apache.spark.serializer.KryoSerializer -Dspark.yarn.historyServer.address=http://XXXX-cdh-dev-cdh-node2:18088 -Dspark.driver.extraLibraryPath=/opt/cloudera/parcels/CDH-5.4.1-1.cdh5.4.1.p0.6/lib/hadoop/lib/native -Dspark.eventLog.dir=hdfs://XXXX-cdh-dev-cdh-node1:8020/user/spark/applicationHistory -Dspark.master=yarn-cluster -Dspark.yarn.app.container.log.dir=/var/log/hadoop-yarn/container/application_1453125563779_0160/container_1453125563779_0160_02_000001 org.apache.spark.deploy.yarn.ApplicationMaster --class not_telling --jar file:not_telling.jar --arg --conf --arg spark.executor.extraClasspath=/opt/cloudera/parcels/CDH/jars/htrace-core-3.0.4.jar --executor-memory 512m --executor-cores 4 --num-executors 10 

Answer 1

Your application is being killed for virtual memory usage (notice, the 2.6 out of 2.1GB used message).

A couple options that could help:

1. Disable virtual memory checks in yarn-site.xml by changing "yarn.nodemanager.vmem-check-enabled" to false. This is pretty frequently done, this is usually what I do to be honest.
2. Increase "spark.yarn.executor.memoryOverhead" and "spark.yarn.driver.memoryOverhead" until your job stops getting killed.

The reasoning for this is because YARN places a limit on the amount of off heap memory your process is allowed to use. If your application has a ton of executable code (large perm gen in java 7 or earlier) you will hit this limit pretty quickly. You're also pretty likely to hit it if you use pyspark where off-heap memory is pretty frequently used.

Answer 2

Check out this article, it has a great description. You might want to note where they're saying "Be aware of the max (7%, 384m) overhead off-heap memory when calculating the memory for executors."

Edit (by Eshalev): I'm Accepting this answer, and elaborating on what was found. Java8 uses a different memory scheme. Specifically CompressedClasses reserve 1024MB in the "Metaspace". This is much larger than what previous versions of java would allocate in "perm-gen" memory. You can use "jmap -heap [pid]" to examine this. We currently keep the app from crashing by over-allocating 1024MB beyond our heap requirements. This is wastefull, but it keeps the app from crashing.

Answer 3

unless you're dealing with very few lines of data you won't go far with 1GB memory per executor.

the best way to calculate the correct ressources you can use is this way: take the nb of cpu and memory you have on 1 node, leave from 1 to 4 cpu cores for the system hdfs (1 core in case of 4core node and 4 cores if you have 32 cores node) divide by 2 to 5 (at least 2 to have multitask with broadcasted data and don't go over 5 as you will face bad hdfs IO bandwidth) and you will get the number of executor you can have on one node. now take the amount of ram for this node, look the maximum taht yarn allows you for all containers in one node (that should be near 26 GB for your case) and divide it by the number of executor calculated before. remove 10% and you got the amount or memory for one executor.

Set manually the spark.yarn.executor.memoryOverhead to 10% of the executor memory as HDP or CDH might force it to 384MB wich is the minimum value.

now for the number of instances, multiply the number of executor X number of nodes and remove 1 for the driver (and yes you should raise the amount of memory and cpu for the driver the same way)

so for example i have 3 nodes on aws R4.8xlarge each with 32 cpu and 244GB memory and that allows me to have 20 executors each with 4 cpu and 26 GB memory

spark.executor.memory=26g
spark.yarn.executor.memoryOverhead=2600
spark.driver.memory=26g
spark.yarn.driver.memoryOverhead=2600
spark.executor.cores=4
spark.executor.instances=20
spark.driver.cores=4

after that you may have to tune according to your configuration, for example you may reduce the number of executors to allow them to have more memory.