Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Mapreduce job fail when submitted from windows machine

I am trying to submit a M/R job from windows machine to hadoop cluster on linux. I am using hadoop 2.2.0 (HDP 2.0).

I am getting following error:

2014-06-06 08:32:37,684 [main] INFO  Job.monitorAndPrintJob  - Job job_1399458460502_0053 running in uber mode : false
2014-06-06 08:32:37,704 [main] INFO  Job.monitorAndPrintJob  -  map 0% reduce 0%
2014-06-06 08:32:37,717 [main] INFO  Job.monitorAndPrintJob  - Job job_1399458460502_0053 failed with state FAILED due to: Application application_1399458460502_0053 failed 2 times due to AM Container for appattempt_1399458460502_0053_000002 exited with  exitCode: 1 due to: Exception from container-launch: 
org.apache.hadoop.util.Shell$ExitCodeException: /bin/bash: line 0: fg: no job control

    at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
    at org.apache.hadoop.util.Shell.run(Shell.java:379)
    at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
    at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)


.Failing this attempt.. Failing the application.

java.io.IOException: Job failed!

I using configuration taken from the cluster so there should be exact match.

Any hints or clue, what might be wrong?

Thx

like image 761
jaksky Avatar asked Jun 06 '14 06:06

jaksky


Video Answer


2 Answers

conf.set("mapreduce.app-submission.cross-platform", "true");
like image 77
Dyin Avatar answered Oct 22 '22 11:10

Dyin


I found solution. It is officially reported bug

https://issues.apache.org/jira/browse/MAPREDUCE-4052

with workaround mentioned down in discussion. Workaround works fine for me. Seems that those bugs with something resolved on a client side are ever green.

like image 38
jaksky Avatar answered Oct 22 '22 10:10

jaksky