Can num-executors override dynamic allocation in spark-submit

Question

Can specifying num-executors in spark-submit command override alreay enabled dynamic allocation (spark.dynamicAllocation.enable true) ?

Answer 1

You can see from log:

INFO util.Utils: Using initial executors = 60, 
max of spark.dynamicAllocation.initialExecutors, spark.dynamicAllocation.minExecutors and spark.executor.instances

That means spark will take the max(spark.dynamicAllocation.initialExecutors, spark.dynamicAllocation.minExecutors, spark.executor.instances)

spark.executor.instances is --num-executor.

Answer 2

In your spark-defaults.conf file you can set the following to control the behaviour of dynamic allocation on Spark2

spark.dynamicAllocation.enabled=true
spark.dynamicAllocation.initialExecutors=1
spark.dynamicAllocation.minExecutors=1
spark.dynamicAllocation.maxExecutors=5

If your spark2-submit command does not specify anything then your job starts with 1 executor and increases to 5 if required.

If your spark2-submit command specifies the following

--num-executors=3

then your job will start with 3 executors and still grow to 5 executors if required.

Check your log messages for

Using initial executors = [initialExecutors], max of spark.dynamicAllocation.initialExecutors, spark.dynamicAllocation.minExecutors and spark.executor.instances

Additionally if do not specify spark.dynamicAllocation.maxExecutors at all then, given a resource hungry job, it will continue to use as many executors as it can (in the case of Yarn this could be restricted by a limit defined on the Queue you submitted your job to). I have seen "rogue" spark jobs on Yarn hog huge amounts of resource on shared clusters starving other jobs. Your Yarn administrators should prevent resource starvation, etc by configuring sensible defaults and splitting different types of work loads across different queues.

I would advise performance testing any changes you intend to make in overriding the defaults, particularly trying to simulate busy periods of your system.