Spark Configuration: SPARK_MEM vs. SPARK_WORKER_MEMORY

Question

In spark-env.sh, it's possible to configure the following environment variables:

# - SPARK_WORKER_MEMORY, to set how much memory to use (e.g. 1000m, 2g)
export SPARK_WORKER_MEMORY=22g
[...]
# - SPARK_MEM, to change the amount of memory used per node (this should
#   be in the same format as the JVM's -Xmx option, e.g. 300m or 1g)
export SPARK_MEM=3g

If I start a standalone cluster with this:

$SPARK_HOME/bin/start-all.sh

I can see at the Spark Master UI webpage that all the workers start with only 3GB RAM:

-- Workers Memory Column --
22.0 GB (3.0 GB Used)
22.0 GB (3.0 GB Used)
22.0 GB (3.0 GB Used)
[...]

However, I specified 22g as SPARK_WORKER_MEMORY in spark-env.sh

I'm somewhat confused by this. Probably I don't understand the difference between "node" and "worker".

Can someone explain the difference between the two memory settings and what I might have done wrong?

I'm using spark-0.7.0. See also here for more configuration info.

Answer 1

A standalone cluster can host multiple Spark clusters (each "cluster" is tied to a particular SparkContext). i.e. you can have one cluster running kmeans, one cluster running Shark, and another one running some interactive data mining.

In this case, the 22GB is the total amount of memory you allocated to the Spark standalone cluster, and your particular instance of SparkContext is using 3GB per node. So you can create 6 more SparkContext's using up to 21GB.