What is the right way to edit spark-env.sh before running spark-shell?

Question

I am running spark on my local windows machine. I am able to start spark shell successfully.

I want to edit the spark-env.sh file residing in conf/ folder. What is the right way to add values to the spark-env.sh file.

E.g If I want to add value to SPARK_EXECUTOR_MEMORY variable how to do it? Am getting confused between different answers that are available 1. SPARK_EXECUTOR_MEMORY="2G" 2. export

Answer 1

The spark-env.sh is a regular bash script intended for Unix, so on a Windows installation it will never get picked up.

On Windows, you'll need to have a spark-env.cmd file in the conf directory and instead use the following syntax :

set SPARK_EXECUTOR_MEMORY=2G

On Unix, the file will be called spark-env.sh and you will need to preprend each of your properties with export (e.g. : export SPARK_EXECUTOR_MEMORY=2G)