MapReduce job in headless environment fails N times due to AM Container exception from container-launch

Question

When running a map reduce job in a headless environment in MacOSX (e.g., when running jobs when ssh'ed as a particular user), I get the following exception or something like it…

2013-12-04 15:08:28,513 WARN org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=hadoop   OPERATION=Application Finished - Failed TARGET=RMAppManager     RESULT=FAILURE  DESCRIPTION=App failed with state: FAILED       PERMISSIONS=Application application_1386194876944_0001 failed 2 times due to AM Container for appattempt_1386194876944_0001_000002 exited with  exitCode: 1 due to: Exception from container-launch:
org.apache.hadoop.util.Shell$ExitCodeException:
        at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
        at org.apache.hadoop.util.Shell.run(Shell.java:379)
        at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
        at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)

If instead, I log in as that user, the error does not occur and the MR job runs to conclusion, while a Java icon labeled "MRAppMaster" pops up in the dock.

I have narrowed this down to the ResourceManager starting a Java process without passing along the -Djava.awt.headless=true. When this occurs in a headless environment, the JVM does not have permission to display in the root window. This has showed up in a number of other circumstances, and I've corrected each.

This is not a question of permissions (something suggested elsewhere) or missing directories.

But I'm at a loss as to how to affect the last of the offending attempts to access the root window without permission.

I have added the -Djava.awt.headless=true option to the following:

• HADOOP_OPTS in hadoop-env.sh
• HADOOP_JOB_HISTORYSERVER_OPTS in mapred-env.sh YARN_OPTS in yarn-env.sh
• YARN_RESOURCEMANAGER_OPTS in yarn-env.sh (although that probably duplicates the YARN_OPTS
• mapred.{map|reduce}.child.java.opts and mapred.child.java.opts in mapred-site.xml

What am I missing? Might I be better off adding this to my Java options globally?

FYI, this is merely a pseudo cluster setup on a Mac, OS X 10.8.5, running Hadoop 2.2.0 downloaded from Apache with Java 1.6.0_65-b14. I did not use Homebrew or any other distribution. I am testing the pseudo-cluster with the WordCount example.

Thanks.

---

Ok. Mea culpa. I finally found all the settings to add… Search for all of the "opt" entries in the mapred-default.xml configuration instructions.

Here they are in-situ…

<property>
    <name>mapred.child.java.opts</name>
    <value>-Djava.awt.headless=true</value>
</property>
<!-- add headless to default -Xmx1024m -->
<property>
    <name>yarn.app.mapreduce.am.command-opts</name>
    <value>-Djava.awt.headless=true -Xmx1024m</value>
</property>
<property>
    <name>yarn.app.mapreduce.am.admin-command-opts</name>
    <value>-Djava.awt.headless=true</value>
</property>

I had also tried to accomplish the same thing by adding the parameter to _JAVA_OPTIONS in /etc/profile. Java picked it up, except when running the MRAppMaster!!!

Hopefully this helps someone else.

Answer 1

The problem is caused because YARN is using different path for JAVA executable different then you have in your OS.

The hardcoded path to check for java is /bin/java however if you don't have /bin/java as your Java executable the YARN job will fail. Like in OSX I have Java 1.7 running at /usr/bin/java as below:

$java -version  
 java  version "1.7.0_45" 
 Java(TM) SE Runtime Environment (build 1.7.0_45-b18) 
 Java HotSpot(TM) 64-Bit Server VM (build 24.45-b08, mixed mode)

To solve this problem in OSX I created a link from /bin/java to /usr/bin/java as below:

$ sudo ln -s /usr/bin/java /bin/java                                                                        
  Password: *****

After then the job ran successfully.