how to limit the number of concurrent map tasks per executor?

Question

A map operation in my Spark APP takes an RDD[A] as input and map each element in RDD[A] using a custom mapping function func(x:A):B to another object of type B. Because func() requires significant amount of memory when computing each input x, I want to limit the number of concurrent map tasks per executor such that the total amount of memory required by all tasks on the same executor does not exceeds the amount of physical memory available on the node.

I checked out available spark configurations, but not sure which one to use. Does using coalesce(numPartitions) to set the number of partitions for RDD[A] fulfil the purpose?

Answer 1

The number of concurrent tasks per executor is related to the available number of cores, not the number of tasks, so changing the parallelism level using coalesce or repartition will not help in constraining the used memory for each task, only the amount of data on each partition that needs to be processed by a given task (*).

As far as I know, there's no way to constrain the memory used by a single task, because it's sharing the resources of the worker JVM, and hence sharing memory with the other tasks on the same executor.

Assuming a fair share per task, a guideline for the amount of memory available per task (core) will be:

spark.executor.memory * spark.storage.memoryFraction / #cores-per-executor

Probably, a way to force less tasks per executor, and hence more memory available per task, would be to assign more cores per task, using spark.task.cpus (default = 1)

(*) Given that the concern here is at the level of each element x of an RDD, the only possible setting that could affect memory usage is to set a parallelism level less than the number of CPUs of a single executor, but that would result in severe under-utilization of the cluster resources as all workers but one will be idle.