Hadoop headerroom miscalculation

Question

I tried to run a hadoop application. It is running when only on single node, but can't run on multi-node cluster. The log file shows this error everytime before the application stops

2015-11-27 14:50:19,513 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating schedule, headroom=<memory:19456, vCores:-3>
2015-11-27 14:50:19,513 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Reduce slow start threshold not met. completedMapsForReduceSlowstart 1

I searched for some solutions, they suggested to set virtual memory limits in mapred-site.xml and set the virtual memory ration in yarn-site.xml So, I now my mapred-site.xml file currently is

<configuration>
<property>
    <name>mapreduce.job.tracker</name>
    <value>HadoopMaster:5431</value>
</property>
<property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
</property>
<property>
    <name>mapreduce.task.timeout</name>
    <value>1800000</value>
</property>
<property>
    <name>mapreduce.map.memory.mb</name>
    <value>4096</value>
</property>
<property>
    <name>mapreduce.reduce.memory.mb</name>
    <value>4096</value>
</property>
<property>
    <name>mapreduce.map.java.opts</name>
    <value>-Xmx3072m</value>
</property>
<property>
    <name>mapreduce.reduce.java.opts</name>
    <value>-Xmx3072m</value>
</property>
</configuration>

And, my yarn-site.xml file is:

<configuration>

<!-- Site specific YARN configuration properties -->
<property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
</property>
<property>
    <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
    <value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
    <name>yarn.resourcemanager.resource-tracker.address</name>
    <value>HadoopMaster:8025</value>
</property>
<property>
    <name>yarn.resourcemanager.scheduler.address</name>
    <value>HadoopMaster:8035</value>
</property>
<property>
    <name>yarn.resourcemanager.address</name>
    <value>HadoopMaster:8050</value>
</property>
<property>
    <name>yarn.nodemanager.vmem-pmem-ratio</name>
    <value>4</value>
</property>
<property>
    <name>yarn.nodemanager.vmem-check-enabled</name>
    <value>false</value>
</property>
</configuration>

Answer 1

Following error occurs, because available resources for this job is "null" (Means no resources are available).

org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating schedule, headroom=

And following error occurs, when the number of completed mappers is less than the threshold needed for starting the reducers. This value is determined by the configuration parameter: mapred.reduce.slowstart.completed.maps (in mapred-site.xml), which is by default set to .05 (means, reducers are not started till at least 5% of mappers are completed).

org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Reduce slow start threshold not met. completedMapsForReduceSlowstart 1

In your case, the completedMapsForReduceSlowStart is set to 1.

Following piece of code determines this value:

//check for slow start
if (!getIsReduceStarted()) {//not set yet
  int completedMapsForReduceSlowstart = (int)Math.ceil(reduceSlowStart * 
                  totalMaps);
  if(completedMaps < completedMapsForReduceSlowstart) {
    LOG.info("Reduce slow start threshold not met. " +
          "completedMapsForReduceSlowstart " + 
        completedMapsForReduceSlowstart);
    return;

It means, at least 1 mapper should have been completed (because, I guess your threshold is set to 5% and you have 20 mappers, hence 5% of 20 = 1), before reducers can start.

So, in your case, not even 1 mapper has completed execution.

This indicates, there is a serious problem with your memory settings in yarn-site.xml and mapred-site.xml. The YARN RM is unable to allocate any containers for your mappers, hence not a single mapper is running.

I have pasted my memory settings below (which work perfectly for me). Compare your settings with the settings below and apply the settings proportionally. For e.g. in my case, (yarn.scheduler.maximum-allocation-mb) / (yarn.scheduler.minimum-allocation-mb) = (9830) / (1228) = 8.

yarn-site.xml

<property> <name>yarn.app.mapreduce.am.resource.mb</name> <value>1228</value> </property>
<property> <name>yarn.app.mapreduce.am.command-opts</name> <value>-Xmx983m</value> </property>
<property> <name>yarn.scheduler.minimum-allocation-mb</name> <value>1228</value> </property>
<property> <name>yarn.scheduler.maximum-allocation-mb</name> <value>9830</value> </property>
<property> <name>yarn.nodemanager.resource.memory-mb</name> <value>9830</value> </property>

mapred-site.xml

<property> <name>mapreduce.map.memory.mb</name> <value>1228</value> </property>
<property> <name>mapreduce.reduce.memory.mb</name> <value>1228</value> </property>
<property> <name>mapreduce.map.java.opts</name> <value>-Xmx983m</value> </property>
<property> <name>mapreduce.reduce.java.opts</name> <value>-Xmx983m</value> </property>

Also, refer to the following blogs on tuning the YARN configuration:

1. Tuning the Cluster for MapReduce v2 (YARN)
2. How to Plan and Configure YARN and MapReduce 2 in HDP 2.0

After setting your memory configurations properly, you should be able to run your job.