SparkConf settings not used when running Spark app in cluster mode on YARN

Question

I wrote a Spark application, which sets sets some configuration stuff via SparkConf instance, like this:

SparkConf conf = new SparkConf().setAppName("Test App Name");

conf.set("spark.driver.cores", "1");
conf.set("spark.driver.memory", "1800m");

conf.set("spark.yarn.am.cores", "1");
conf.set("spark.yarn.am.memory", "1800m");

conf.set("spark.executor.instances", "30");
conf.set("spark.executor.cores", "3");
conf.set("spark.executor.memory", "2048m");

JavaSparkContext sc = new JavaSparkContext(conf);

JavaRDD<String> inputRDD = sc.textFile(...);
...

When I run this application with the command (master=yarn & deploy-mode=client)

spark-submit --class spark.MyApp --master yarn --deploy-mode client /home/myuser/application.jar

everything seems to work fine, the Spark History UI shows correct executor information:

But when running it with (master=yarn & deploy-mode=cluster)

my Spark UI shows wrong executor information (~512 MB instead of ~1400 MB):

Also my App name equals Test App Name when running in client mode, but is spark.MyApp when running in cluster mode. It seems that however some default settings are taken when running in Cluster mode. What am I doing wrong here? How can I make these settings for the Cluster mode?

I'm using Spark 1.6.2 on a HDP 2.5 cluster, managed by YARN.

Answer 1

OK, I think I found out the problem! In short form: There's a difference between running Spark settings in Standalone and in YARN-managed mode!

---

So when you run Spark applications in the Standalone mode, you can focus on the Configuration documentation of Spark, see http://spark.apache.org/docs/1.6.2/configuration.html

You can use the following settings for Driver & Executor CPU/RAM (just as explained in the documentation):

• spark.executor.cores
• spark.executor.memory
• spark.driver.cores
• spark.driver.memory

---

BUT: When running Spark inside a YARN-managed Hadoop environment, you have to be careful with the following settings and consider the following points:

• orientate on the "Spark on YARN" documentation rather then on the Configuration documentation linked above: http://spark.apache.org/docs/1.6.2/running-on-yarn.html (the properties explained here have a higher priority then the ones explained in the Configuration docu (this seems to describe only the Standalone cluster vs. client mode, not the YARN cluster vs. client mode!!))

• you can't use SparkConf to set properties in yarn-cluster mode! Instead use the corresponding spark-submit parameters:

  • --executor-cores 5
  • --executor-memory 5g
  • --driver-cores 3
  • --driver-memory 3g

• In yarn-client mode you can't use the spark.driver.cores and spark.driver.memory properties! You have to use the corresponding AM properties in a SparkConf instance:

  • spark.yarn.am.cores
  • spark.yarn.am.memory
  • You can't set these AM properties via spark-submit parameters!
• To set executor resources in yarn-client mode you can use
  • spark.executor.cores and spark.executor.memory in SparkConf
  • --executor-cores and executor-memory parameters in spark-submit
  • if you set both, the SparkConf settings overwrite the spark-submit parameter values!

---

This is the textual form of my notes:

Hope I can help anybody else with this findings...

Answer 2

Just to add on to D. Müller's answer:

Same issue happened to me and I tried the settings with some different combination. I am running Pypark 2.0.0 on YARN cluster.

I found that driver-memory must be written during spark submit but executor-memory can be written in script (i.e. SparkConf) and the application will still work.

My application will die if driver-memory is less than 2g. The error is:

ERROR yarn.ApplicationMaster: RECEIVED SIGNAL TERM

ERROR yarn.ApplicationMaster: User application exited with status 143

---

CASE 1: driver & executor both written in SparkConf

spark = (SparkSession
    .builder
    .appName("driver_executor_inside")
    .enableHiveSupport()
    .config("spark.executor.memory","4g")
    .config("spark.executor.cores","2")
    .config("spark.yarn.executor.memoryOverhead","1024")
    .config("spark.driver.memory","2g")
    .getOrCreate())

spark-submit --master yarn --deploy-mode cluster myscript.py

---

CASE 2: - driver in spark submit - executor in SparkConf in script

spark = (SparkSession
    .builder
    .appName("executor_inside")
    .enableHiveSupport()
    .config("spark.executor.memory","4g")
    .config("spark.executor.cores","2")
    .config("spark.yarn.executor.memoryOverhead","1024")
    .getOrCreate())

spark-submit --master yarn --deploy-mode cluster --conf spark.driver.memory=2g myscript.py

The job Finished with succeed status. Executor memory correct.

---

CASE 3: - driver in spark submit - executor not written

spark = (SparkSession
    .builder
    .appName("executor_not_written")
    .enableHiveSupport()
    .config("spark.executor.cores","2")
    .config("spark.yarn.executor.memoryOverhead","1024")
    .getOrCreate())

spark-submit --master yarn --deploy-mode cluster --conf spark.driver.memory=2g myscript.py

Apparently the executor memory is not set. Meaning CASE 2 actually captured executor memory settings despite writing it inside sparkConf.