How to pass environment variables to spark driver in cluster mode with spark-submit

Question

spark-submit allows to configure the executor environment variables with --conf spark.executorEnv.FOO=bar, and the Spark REST API allows to pass some environment variables with the environmentVariables field. Unfortunately I've found nothing similar to configure the environment variable of the driver when submitting the driver with spark-submit in cluster mode:

spark-submit --deploy-mode cluster myapp.jar

Is it possible to set the environment variables of the driver with spark-submit in cluster mode?

Answer 1

On YARN at least, this works:

spark-submit --deploy-mode cluster --conf spark.yarn.appMasterEnv.FOO=bar myapp.jar

---

It's mentioned in http://spark.apache.org/docs/latest/configuration.html#environment-variables that:

Note: When running Spark on YARN in cluster mode, environment variables need to be set using the spark.yarn.appMasterEnv.[EnvironmentVariableName] property in your conf/spark-defaults.conf file.

I have tested that it can be passed with --conf flag for spark-submit, so that you don't have to edit global conf files.

Answer 2

On Yarn in cluster mode, it worked by adding the environment variables in the spark-submit command using --conf as below-

spark-submit --master yarn-cluster --num-executors 15 --executor-memory 52g --executor-cores 7 --driver-memory 52g --conf "spark.yarn.appMasterEnv.FOO=/Path/foo" --conf "spark.executorEnv.FOO2=/path/foo2" app.jar

Also, you can do it by adding them in conf/spark-defaults.conf file.