Change Executor Memory (and other configs) for Spark Shell

Question

How to change executor memory (and other configs) for Apache Spark Shell?

In particular I would like to give flats to spark-shell, like -Dspark-cores-max=12 when I start it so that my jobs in the spark shell will use those configuration settings.

Answer 1

As of spark 1.2.0 you can set memory and cores by giving following arguments to spark-shell.

spark-shell --driver-memory 10G --executor-memory 15G --executor-cores 8

to see other options you can give following commands to spark shell

spark-shell --help

Answer 2

If you are running the spark-shell on spark installed on standalone mode (1 node), use

./bin/spark-shell --driver-memory 4g

If you are running the spark-shell on spark installed on cluster (2+ nodes), use

./bin/spark-shell --executor-memory 4g

4g is 4GB.

Answer 3

DEPRECATED USE ACCEPTED ANSWER

Write a script like this:

#!/bin/bash
export SPARK_JAVA_OPTS="$*"
MASTER=spark://ec2-99-99-99-99:7077 /usr/share/spark/bin/spark-shell

/usr/share/spark/bin/spark-shell should be the path to where the long spark-shell starting script is. On my cluster there was another script in /usr/local/bin/ but this one was just a few lines similar to above and had SPARK_JAVA_OPTS hardcoded.

Anyway, example use:

my-spark-starter-script -Dspark-cores-max=12 -Dspark.executor.memory=26000m