Spark-submit master url and SparkSession master url in the main class, what is difference?

Question

When submitting a job with spark-submit I set the master URL and give him a main class, ex:

spark-submit --class WordCount --master spark://spark:7077 my.jar

But inside this main class my spark context define another master url :

SparkSession.builder().appName("Word2vec").master("local").

This get me confused, what happens if I send a job with spark-submit to the master of a standalone cluster (spark://spark:7077) that start a SparkSession with a local master ?

Should the SparkSession master url always be the same as the spark-submit url when executed on a cluster ?

Answer 1

There is no difference between these properties. If set both, properties set directly in application take precedence. To quote documentation:

Any values specified as flags or in the properties file will be passed on to the application and merged with those specified through SparkConf. Properties set directly on the SparkConf take highest precedence, then flags passed to spark-submit or spark-shell, then options in the spark-defaults.conf file. A few configuration keys have been renamed since earlier versions of Spark; in such cases, the older key names are still accepted, but take lower precedence than any instance of the newer key.