spark on yarn, Connecting to ResourceManager at /0.0.0.0:8032

Question

I was writing a spark program in my developing machine, which is a mac. The version of hadoop is 2.6, the version of spark is 1.6.2. The hadoop cluster have 3 nodes, of course all in linux machine. I run the spark program in idea IDE in spark standalone mode, it works successfully. But now, I change it to yarn-client mode, it doesn't work successfully, and gives the message as follows:

...
2017-02-23 11:01:33,725-[HL] INFO main org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at /0.0.0.0:8032
2017-02-23 11:01:34,839-[HL] INFO main org.apache.hadoop.ipc.Client - Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2017-02-23 11:01:35,842-[HL] INFO main org.apache.hadoop.ipc.Client - Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2017-02-23 11:01:36,847-[HL] INFO main org.apache.hadoop.ipc.Client - Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2017-02-23 11:01:37,854-[HL] INFO main org.apache.hadoop.ipc.Client - Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
...

I have already added corresponding configuration files to the resources directory of the project. If I make it a jar package and use spark-submit to run this program, it will be ok. Now, I want to run this program in IDE as yarn-client mode, just like spark standalone mode. How can I fix this problem? Thanks.

Answer 1

Ensure the YARN configurations are available for Spark to use when running in yarn mode. Add these files core-site.xml, hdfs-site.xml and yarn-site.xml files to the conf directory of spark.
Also make sure, the yarn-site.xml contains the address of the resource manager

<property>
   <name>yarn.resourcemanager.address</name>
   <value>resource_manager_ip:8032</value>
</property>

Answer 2

Set your conf object like this, its work for me:

conf = new SparkConf().setAppName(setup.getAppname).setMaster("yarn")
            .set("spark.hadoop.yarn.resourcemanager.hostname", "resourcemanager.fqdn")
            .set("spark.hadoop.yarn.resourcemanager.address", "resourcemanager.fqdn:8032")`

Font: hortonworks.com