Hadoop Mapper is failing because of "Container killed by the ApplicationMaster"

Question

I am trying to execute a map reduce program on Hadoop.

When i submit my job to the hadoop single node cluster. The job is getting created but failing with the message

"Container killed by the ApplicationMaster"

The input used is of the size 10 MB.

When i used the same script of input file 400 KB, it got succeded. But failing for the the input file of size 10 MB.

The complete log that is displayed in my terminal is as follows.

    15/05/29 09:52:16 WARN util.NativeCodeLoader: Unable to `load native-  hadoop library for your platform... using builtin-java classes      where applicable
Submitting job on the cluster...
15/05/29 09:52:17 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
15/05/29 09:52:18 INFO input.FileInputFormat: Total input paths to process : 1
15/05/29 09:52:18 INFO mapreduce.JobSubmitter: number of splits:1
15/05/29 09:52:19 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1432910768528_0001
15/05/29 09:52:19 INFO impl.YarnClientImpl: Submitted application application_1432910768528_0001
15/05/29 09:52:19 INFO mapreduce.Job: The url to track the job: http://localhost:8088/proxy/application_1432910768528_0001/
15/05/29 09:52:19 INFO mapreduce.Job: Running job: job_1432910768528_0001
15/05/29 09:52:29 INFO mapreduce.Job: Job job_1432910768528_0001 running in uber mode : false
15/05/29 09:52:29 INFO mapreduce.Job:  map 0% reduce 0%
15/05/29 09:52:41 INFO mapreduce.Job:  map 100% reduce 0%
15/05/29 10:03:01 INFO mapreduce.Job:  map 0% reduce 0%
15/05/29 10:03:01 INFO mapreduce.Job: Task Id : attempt_1432910768528_0001_m_000000_0, Status : FAILED
AttemptID:attempt_1432910768528_0001_m_000000_0 Timed out after 600 secs
Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143

My mapper here is triggering the other program which is going to process my input file here. The program which is getting trigger by mapper usually consumes lots of memory.

So please help me in this regard.

Answer 1

Include below properties in yarn-site.xml and restart VM,

<property>
   <name>yarn.nodemanager.vmem-check-enabled</name>
   <value>false</value>
   <description>Whether virtual memory limits will be enforced for containers</description>
</property>

<property>
   <name>yarn.nodemanager.vmem-pmem-ratio</name>
   <value>4</value>
   <description>Ratio between virtual memory to physical memory when setting memory limits for containers</description>
</property>

Answer 2

A container is a yarn JVM process. In Mapreduce the application master service, mapper and reducer tasks are all containers that execute inside the yarn framework.

You can fix this issue by either increase the number of reducers ( say mapreduce.job.reduces=10 ) or by increasing the reduce heap size ( mapreduce.reduce.java.opts=-Xmx2014m )

If you would want to have fixed number of reducer at run time, you can do it while passing the Map/Reduce job at command line. Using -D mapreduce.job.reduces=10 with desired number will spawn that many reducers at runtime.

In the code,you can configure JobConf variable to set number of mappers and reducers. Lets say we have JobConf variable as job.

Configuration conf = new Configuration();
Job job = new Job(conf);
job.setNumReduceTasks(10); // 10 reducers

You can also split the file into smaller size for this particular job to avoid memory issue.

If you are still getting issue, please check yarn log and post the log.