MapReduce jobs get stuck in Accepted state

Question

I have my own MapReduce code that I'm trying to run, but it just stays at Accepted state. I tried running another sample MR job that I'd run previously and which was successful. But now, both the jobs stay in Accepted state. I tried changing various properties in the mapred-site.xml and yarn-site.xml as mentioned here and here but that didn't help either. Can someone please point out what could possibly be going wrong. I'm using hadoop-2.2.0

I've tried many values for the various properties, here is one set of values- In mapred-site.xml

<property>
<name>mapreduce.job.tracker</name>
<value>localhost:54311</value>
</property> 

<property>
<name>mapreduce.job.tracker.reserved.physicalmemory.mb</name>
<value></value>
</property>

<property>
<name>mapreduce.map.memory.mb</name>
<value>256</value>
</property>

<property>
<name>mapreduce.reduce.memory.mb</name>
<value>256</value>
</property>


<property>
<name>yarn.app.mapreduce.am.resource.mb</name>
<value>400</value>
<source>mapred-site.xml</source>
</property>

In yarn-site.xml

<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>400</value>
<source>yarn-site.xml</source>
</property>
<property>
<name>yarn.scheduler.capacity.maximum-am-resource-percent</name>
<value>.3</value>
</property>

Answer 1

I've had the same effect and found that making the system have more memory available per worker node and reduce the memory required for an application helped.

The settings I have (on my very small experimental boxes) in my yarn-site.xml:

<property>
  <name>yarn.nodemanager.resource.memory-mb</name>
  <value>2200</value>
  <description>Amount of physical memory, in MB, that can be allocated for containers.</description>
</property>

<property>
  <name>yarn.scheduler.minimum-allocation-mb</name>
  <value>500</value>
</property>

Answer 2

Had the same issue, and for me it was a full hard drive (>90% full) which was the issue. Cleaning space saved me.

Answer 3

A job stuck in accepted state on YARN is usually because of free resources are not enough. You can check it at http://resourcemanager:port/cluster/scheduler:

1. if Memory Used + Memory Reserved >= Memory Total, memory is not enough
2. if VCores Used + VCores Reserved >= VCores Total, VCores is not enough

It may also be limited by parameters such as maxAMShare.