hadoop yarn: show the pending resoure request of an application

Question

How could I get the number of pending requests sent to the resource manager by an application? As I know, the resource manager may not allocate the requested resources immediately, so requests will be pended, right?

Answer 1

I am using Hadoop 2.7.1. From the Resource Manager UI, you can see the "Outstanding Resource Requests".

In the RM UI:

1. Click on the running application (for e.g. application_1447644421851_0004)
2. In the application page, click on the attempt ID (for e.g. appattempt_1447644421851_0004_000001)
3. In the Application Attempt ID page, you'll see the details of "Total Allocated Containers" and "Total Outstanding Resource Requests".

For e.g., I ran a DistCp job, which needed 24 mappers. So, total containers allocated were 24 + 1 (1 container for Application Master) = 25.

As per my settings, each container needed 1,228 MB of memory and 1 vCore.

I have explained the progress of this job with the pictures below:

Stage 1:

To begin with 7 mappers started running and totally 8 containers were allotted (7 for mappers and 1 for Application Master).

So,

Total Allocated Containers: 8
Total Outstanding Requests: <memory: 20876, vCores:17>

You can observe that, there are outstanding requests for 17 containers (25 minus 8). So total outstanding request for memory is = 17 * 1228 MB = 20876 MB and for vCores it is = 17 * 1 = 17

Check the picture below:

Stage 2:

7 mappers completed and next 7 were scheduled.

Now:

Total Allocated Containers: 15
Total Outstanding Requests: <memory: 12280, vCores:10>

You can observe that, outstanding requests has now come down to 10 containers (25 minus 15). So total outstanding requests for memory is = 10 * 1228 MB = 12280 MB and for vCores it is = 10 * 1 = 10

Check the picture below:

Stage 3:

14 mappers completed and next 7 were scheduled.

Now:

Total Allocated Containers: 22
Total Outstanding Requests: <memory: 3684, vCores:3>

You can observe that, outstanding requests has come down to 3 containers (25 minus 22). So total outstanding request for memory is = 3 * 1228 MB = 3684 MB and for vCores it is = 3 * 1 = 3

Check the picture below:

Final Stage:

Finally, when all the containers were allocated, you will just see "Total Allocated Containers".

Check the picture below:

Hope this clarified your doubt.