Usecases for mapred.job.queue.name

Question

What are the real world use cases on using map reduce job queues i.e. the value of mapred.job.queue.name property. I see default being used as the value always.

Answer 1

mapred.job.queue.name is what you use to assign a job to a particular queue. By default all jobs go to "default" queue. However, it is possible to create hierarchical queue. Like root, root.q1, root.q1.q1a and so on. Each of these queues can have its own set of attributes to ensure certain priority.

A real world scenario will be when you have multiple stakeholder asking for reports on same set of infrastructure. For example, at my work place , we have the data scientist trying to run various kind of research job, the customer support team looking for various daily and weekly figures and then there are real jobs that supports the day to day business. It is at the heart that the infrastructure should be able to support the best it can. Having various queues with different priorities just makes it easy for Hadoop to decide what to do next when a processor is available, or how much it can use.

So data scientist will assign to "Data Analyst" queue, marketing team will assign to "Marketing" queue. It is possible to change priority of a queue depending on time of the day.