I setup a hadoop cluster with two nodes hadoop01(master- 10.0.0.151) and hadoop02(slaves- 10.0.0.152)
when a type start-dfs.sh and then visit website
my_ip(just 10.0.0.151 above):50070. It's successful.
But when i type start-yarn.sh then visit website my_ip:8088. It's failed.
my yarn-site.xml:
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>hadoop01:8032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>hadoop01:8030</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>hadoop01:8031</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>hadoop01:8033</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>hadoop01:8088</value>
</property>
core-site.xml:
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://hadoop01:8020</value>
</property>
</configuration>
the same set as hadoop02(slave)
my hadoop01 hadoop-2.2.0/etc/hadoop/slaves set
hadoop01
hadoop02
After typing start-dfs.sh & start-yarn.sh then i type jps
hadoop01:
21594 NameNode
22345 NodeManager
22007 SecondaryNameNode
22171 ResourceManager
23147 Jps
21762 DataNode
hadoop02:
29861 NodeManager
30358 Jps
29665 DataNode
my /etc/hosts in hadoop01:
localhost hadoop01
10.0.0.151 hadoop01
10.0.0.152 hadoop02
my /etc/hosts in hadoop02:
localhost hadoop02
10.0.0.151 hadoop01
10.0.0.152 hadoop02
This bellow link is my yarn-nodemanager.log I upload to google dirve https://drive.google.com/file/d/0B7nCJ_XJWSrQN1BZVTVyOEgxd1E/edit?usp=sharing but yarn-nodemanager.log doesn't appear ERROR if i didn't miss some information..
Please help me improve the problem why I can't visit the website http://10.0.0.151:8088
**if need other information (such as hdfs-site.xml...etc) just tell me. I'll update..
netstat -tunalp | grep LISTEN
tcp 0 0 0.0.0.0:50010 0.0.0.0:* LISTEN 17442/java
tcp 0 0 0.0.0.0:50075 0.0.0.0:* LISTEN 17442/java
tcp 0 0 0.0.0.0:50020 0.0.0.0:* LISTEN 17442/java
tcp 0 0 0.0.0.0:50090 0.0.0.0:* LISTEN 17693/java
tcp 0 0 10.0.0.151:8020 0.0.0.0:* LISTEN 17267/java
tcp 0 0 0.0.0.0:50070 0.0.0.0:* LISTEN 17267/java
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN -
tcp6 0 0 :::13562 :::* LISTEN 21061/java
tcp6 0 0 10.0.0.151:8030 :::* LISTEN 20881/java
tcp6 0 0 10.0.0.151:8031 :::* LISTEN 20881/java
tcp6 0 0 10.0.0.151:8032 :::* LISTEN 20881/java
tcp6 0 0 10.0.0.151:8033 :::* LISTEN 20881/java
tcp6 0 0 :::33762 :::* LISTEN 21061/java
tcp6 0 0 :::8040 :::* LISTEN 21061/java
tcp6 0 0 :::8042 :::* LISTEN 21061/java
tcp6 0 0 :::22 :::* LISTEN -
tcp6 0 0 10.0.0.151:8088 :::* LISTEN 20881/java
After disable ipv6 , then i type netstat -tunalp | grep LISTEN
tcp 0 0 0.0.0.0:13562 0.0.0.0:* LISTEN 30608/java
tcp 0 0 0.0.0.0:50010 0.0.0.0:* LISTEN 29967/java
tcp 0 0 0.0.0.0:50075 0.0.0.0:* LISTEN 29967/java
tcp 0 0 10.0.0.151:8030 0.0.0.0:* LISTEN 30424/java
tcp 0 0 10.0.0.151:8031 0.0.0.0:* LISTEN 30424/java
tcp 0 0 0.0.0.0:52992 0.0.0.0:* LISTEN 30608/java
tcp 0 0 10.0.0.151:8032 0.0.0.0:* LISTEN 30424/java
tcp 0 0 10.0.0.151:8033 0.0.0.0:* LISTEN 30424/java
tcp 0 0 0.0.0.0:50020 0.0.0.0:* LISTEN 29967/java
tcp 0 0 0.0.0.0:8040 0.0.0.0:* LISTEN 30608/java
tcp 0 0 0.0.0.0:8042 0.0.0.0:* LISTEN 30608/java
tcp 0 0 0.0.0.0:50090 0.0.0.0:* LISTEN 30222/java
tcp 0 0 10.0.0.151:8020 0.0.0.0:* LISTEN 29790/java
tcp 0 0 0.0.0.0:50070 0.0.0.0:* LISTEN 29790/java
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN -
tcp 0 0 10.0.0.151:8088 0.0.0.0:* LISTEN 30424/java
tcp6 0 0 :::22 :::* LISTEN -
the reason is you should change 10.0.0.151:8088 to 0.0.0.0:8088. the method is open the yarn-site.xml, add:
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>0.0.0.0:8088</value>
</property>
Here the problem is resouce manager is running, but the port(8030,8031,8032,8033,8088) occupied by resource manager uses tcp6 instead of tcp(see the left portion). You have two options either you can disable ipv6 in the linux system then restart yarn services.
or
Try modify your yarn-site.xml only on master node as follows. Don't modify the yarn-site.xml in slave nodes:
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<configuration>
Specifying hostname causes the ports started as tcp6, even with out specifying those ports it would take the default value. Have a look at the following default ports
http://hadoop.apache.org/docs/r2.2.0/hadoop-yarn/hadoop-yarn-common/yarn-default.xml
After modiying yarn-site.xml restart your yarn-service
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With