I have 4 queues under the root queue with the following configuration.
|-------------|-----------------|---------------------|-------------------|
| Queue Name | Capacity (in %) | Max Capacity (in %) | User Limit Factor |
|-------------|-----------------|---------------------|-------------------|
| default | 10 | 30 | 10 |
|-------------|-----------------|---------------------|-------------------|
| thriftsvr | 5 | 30 | 10 |
|-------------|-----------------|---------------------|-------------------|
| stream | 70 | 70 | 10 |
|-------------|-----------------|---------------------|-------------------|
| batch | 15 | 30 | 10 |
|-------------|-----------------|---------------------|-------------------|
I have set up capacity by yarn.scheduler.capacity.root.<queue-name>.capacity
and max capacity by yarn.scheduler.capacity.root.<queue-name>.maximum-capacity
property.
My understanding is, above 2 properties set ABSOLUTE capacity and ABSOLUTE maximum capacity respectively. That means queue stream's 100% is equal to the 70% of cluster's total capacity and it can fill up to 100% of queue's capacity that is also the 70% of cluster's total capacity.
Now, the problem is when queue 'stream' is filled with 66.4% (i.e. when Used Capacity: 66.4% & Absolute Used Capacity: 46.5%) then new jobs are getting in the pending state which is submitted in queue 'stream' by saying "waiting for AM container to be allocated, launched and register with RM".
When I checked queue configuration on yarn UI it shows Configured Max Capacity: 70.0% & Absolute Configured Max Capacity: 70.0% but according to the configuration, queue 'stream' can be filled till Used Capacity: 100% & Absolute Used Capacity: 70%
Any idea, why new jobs are unable to utilize the queue stream's capacity till 100%?
I suspect the confusing thing here is that .capacity
and .maximum-capacity
properties can both be defined as either
root
's percentage (as float, e.g. 12.5
)[memory=204800,vcores=122]
)If you have any further questions, please do ask.
For full reference, just read the doc: https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html#Queue_Properties
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With