What controls how much of a Spark Cluster is given to an application?

Question

In this page of the docs https://spark.apache.org/docs/latest/job-scheduling.html for static partitioning it says "With this approach, each application is given a maximum amount of resources it can use".

I was just wondering, what are these maximum resources? I found the memory per executor setting (mentioned just below in dynamic partitioning) this I assume limits the memory resource an application gets. But what decides how many executors are started / how many nodes from the cluster are used e.g. the total cluster memory and the cores that get "taken"?

On another similar note is there a way to change the memory asked for on a per job or task level?

Answer 1

The amount of resources depends on the cluster manager being used as different cluster managers will provide different allocation.

Eg In standalone mode, Spark will try to use all nodes. spark.max.cores will control how many cores in total the job will take across nodes. If not set, Spark will use spark.deploy.defaultCores. The documentation from spark.deploy.defaultCores further clarifies its use:

Default number of cores to give to applications in Spark's standalone mode if they don't set spark.cores.max. If not set, applications always get all available cores unless they configure spark.cores.max themselves. Set this lower on a shared cluster to prevent users from grabbing the whole cluster by default.

In Mesos coarse grained mode, Spark will allocate by default all available cores. Use spark.max.cores to limit that per job.

In Mesos fine-grained mode, Spark will allocate a core per task as needed by the job and release them afterwards. This ensures fair usage at the cost of higher task allocation overhead.

In YARN, per documentation:

The --num-executors option to the Spark YARN client controls how many executors it will allocate on the cluster, while --executor-memory and --executor-cores control the resources per executor.

Regarding memory, there's no way to set the total memory per job or task, only per executor, using spark.executor.memory. The memory assigned to your job will be spark.executor.memory x #executors.