How to set spark.local.dir property from spark shell?

Question

I'm trying to set spark.local.dir from spark-shell using sc.getconf.set("spark.local.dir","/temp/spark"), But it is not working. Is there any other way to set this property from sparkshell.

Answer 1

You can't do it from inside the shell - since the Spark context was already created, so the local dir was already set (and used). You should pass it as parameter when starting the shell:

./spark-shell --conf spark.local.dir=/temp/spark

Answer 2

@Tzach Zohar solution seems to be the right answer.

However, if you insist to set spark.local.dir from spark-shell you can do it:

1) close the current spark context

    sc.stop()

2) updated the sc configuration, and restart it.

The updated code was kindly provided by @Tzach-Zohar:

SparkSession.builder.config(sc.getConf).config("spark.local.‌​dir","/temp/spark").‌​getOrCreate())

@Tzach Zohar note: "but you get a WARN SparkContext: Use an existing SparkContext, some configuration may not take effect, which suggests this isn't the recommended path to take.