how does YARN "Fair Scheduler" work with spark-submit configuration parameter

Question

I have a basic question about YARN "Fair Scheduler". According to the definition of "Fair Scheduler- Fair scheduling is a method of assigning resources to applications such that all apps get, on average, an equal share of resources over time".

Following is my understanding and question.

(1) If multiple applications are running on YARN then it will make sure that all the applications will get more or less equal share of resources over a period of time.

(2) My question is if in YARN this property has set to true then does it make any difference if we use following configurations while submitting spark-submit?

   (i)   driver-memory
   (ii)  executor-memory
   (iii) num-executors
   (iv)  executor-cores

What will happen if I mention these conf parameters while using spark-submit? Will these parameters be accepted and resource will be allocated as per the request or these conf parameters will simply be ignored and the spark application will be allocated some default amount of resource by YARN based on fair scheduling.

Kindly let me know if any other clarification is needed for this question. Thanks

Answer 1

Actually Fair Scheduler is way more sophisticated than this. At the top level resources are organized into pools / queues where each can have its own weight and internal scheduling policy which is not necessarily fair (you can use FIFO scheduling if you want).

Furthermore Fair Scheduling doesn't mean that submitted application will get required shared of resources right away. If application is submitted to a busy cluster and requested resources cannot be assigned it will have to wait until other applications finish, or resources are freed using preemption mechanism (if enabled).

• Parameters used with spark-submit declare amount of resources required to run the application. This "what" part of the problem
• Job of the Fair Scheduler is to assign these resources if possible. Its configuration determines amount of resources that can be assigned to a queue or an application. This "how" part of the problem.

As you can see these two things are not mutually exclusive and submit parameters are meaningful and accepted. As usual amount of requested resources must not exceed amount of resources available on the cluster, otherwise job will fail. You should also keep it below the resource share for a particular queue.