Switching from Capacity Scheduler to Fair Scheduler on Hortonworks Data Platform

Question

My organization is currently using Hortonworks HDP to manage our Hadoop Cluster. The default YARN scheduler is the Capacity Scheduler. I would like to switch to a Fair Scheduler. I am completely new to HDP.

In the absence of a cluster management suite, this would be done by editing the yarn-site.xml and changing the yarn.resourcemanager.scheduler.class property to org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler and creating an extra fair-scheduler.xml file to specify the queue configurations as mentioned here and then referring YARN to that configuration by setting the yarn.scheduler.fair.allocation.file property in yarn-site.xml.

Now in Ambari, while it is possible to change the yarn.resourcemanager.scheduler.class property via the UI, and add a new custom property yarn.scheduler.fair.allocation.file, I cannot (for the love of god) find a way to have ambari read fair-scheduler.xml instead of capacity-scheduler.xml.

So my question is; How do I go about switching to the fair scheduler via Ambari? There's got to be an easy way, right?

Properties in capacity-scheduler.xml

Answer 1

On your RM node, set yarn.scheduler.fair.allocation.file with the full path of your fair-scheduler.xml (or in custom yarn-site under ambari)

tail -n 1000 /var/log/hadoop-yarn/yarn/hadoop-yarn-resourcemanager-master.log | grep "fair-scheduler.xml"

After a restart of your ResourceManager, you should see that it is loading your file:

2019-02-19 15:49:26,358 INFO  fair.AllocationFileLoaderService (AllocationFileLoaderService.java:reloadAllocations(230)) - Loading allocation file file:/usr/hdp/current/hadoop-client/conf/fair-scheduler.xml

Works on Hdp 3.1.1, and probably 3.0.0 too