Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Hadoop2.2.0 can't visit the web http://<ip>:8088

Tags:

hadoop

I setup a hadoop cluster with two nodes hadoop01(master- 10.0.0.151) and hadoop02(slaves- 10.0.0.152) when a type start-dfs.sh and then visit website my_ip(just 10.0.0.151 above):50070. It's successful.
But when i type start-yarn.sh then visit website my_ip:8088. It's failed.

my yarn-site.xml:

<configuration>
<property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
    <value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
    <name>yarn.resourcemanager.address</name>
    <value>hadoop01:8032</value>
</property>        
<property>
    <name>yarn.resourcemanager.scheduler.address</name>
    <value>hadoop01:8030</value>
</property>

<property>
    <name>yarn.resourcemanager.resource-tracker.address</name>
    <value>hadoop01:8031</value>
</property>

<property>
    <name>yarn.resourcemanager.admin.address</name>
    <value>hadoop01:8033</value>
</property>
<property>
    <name>yarn.resourcemanager.webapp.address</name>
    <value>hadoop01:8088</value>
</property>

core-site.xml:

<configuration>
<property>
     <name>fs.defaultFS</name>
     <value>hdfs://hadoop01:8020</value>
 </property>
 </configuration>

the same set as hadoop02(slave)

my hadoop01 hadoop-2.2.0/etc/hadoop/slaves set

hadoop01
hadoop02

After typing start-dfs.sh & start-yarn.sh then i type jps

hadoop01:
21594 NameNode
22345 NodeManager
22007 SecondaryNameNode
22171 ResourceManager
23147 Jps
21762 DataNode
hadoop02:
29861 NodeManager
30358 Jps
29665 DataNode

my /etc/hosts in hadoop01:

localhost       hadoop01
10.0.0.151      hadoop01
10.0.0.152      hadoop02

my /etc/hosts in hadoop02:

localhost       hadoop02
10.0.0.151      hadoop01
10.0.0.152      hadoop02

This bellow link is my yarn-nodemanager.log I upload to google dirve https://drive.google.com/file/d/0B7nCJ_XJWSrQN1BZVTVyOEgxd1E/edit?usp=sharing but yarn-nodemanager.log doesn't appear ERROR if i didn't miss some information..

Please help me improve the problem why I can't visit the website http://10.0.0.151:8088

**if need other information (such as hdfs-site.xml...etc) just tell me. I'll update..

netstat -tunalp | grep LISTEN
tcp        0      0 0.0.0.0:50010           0.0.0.0:*               LISTEN      17442/java
tcp        0      0 0.0.0.0:50075           0.0.0.0:*               LISTEN      17442/java
tcp        0      0 0.0.0.0:50020           0.0.0.0:*               LISTEN      17442/java
tcp        0      0 0.0.0.0:50090           0.0.0.0:*               LISTEN      17693/java
tcp        0      0 10.0.0.151:8020         0.0.0.0:*               LISTEN      17267/java
tcp        0      0 0.0.0.0:50070           0.0.0.0:*               LISTEN      17267/java
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      -
tcp6       0      0 :::13562                :::*                    LISTEN      21061/java
tcp6       0      0 10.0.0.151:8030         :::*                    LISTEN      20881/java
tcp6       0      0 10.0.0.151:8031         :::*                    LISTEN      20881/java
tcp6       0      0 10.0.0.151:8032         :::*                    LISTEN      20881/java
tcp6       0      0 10.0.0.151:8033         :::*                    LISTEN      20881/java
tcp6       0      0 :::33762                :::*                    LISTEN      21061/java
tcp6       0      0 :::8040                 :::*                    LISTEN      21061/java
tcp6       0      0 :::8042                 :::*                    LISTEN      21061/java
tcp6       0      0 :::22                   :::*                    LISTEN      -
tcp6       0      0 10.0.0.151:8088         :::*                    LISTEN      20881/java

After disable ipv6 , then i type netstat -tunalp | grep LISTEN

tcp        0      0 0.0.0.0:13562           0.0.0.0:*               LISTEN      30608/java
tcp        0      0 0.0.0.0:50010           0.0.0.0:*               LISTEN      29967/java
tcp        0      0 0.0.0.0:50075           0.0.0.0:*               LISTEN      29967/java
tcp        0      0 10.0.0.151:8030         0.0.0.0:*               LISTEN      30424/java
tcp        0      0 10.0.0.151:8031         0.0.0.0:*               LISTEN      30424/java
tcp        0      0 0.0.0.0:52992           0.0.0.0:*               LISTEN      30608/java
tcp        0      0 10.0.0.151:8032         0.0.0.0:*               LISTEN      30424/java
tcp        0      0 10.0.0.151:8033         0.0.0.0:*               LISTEN      30424/java
tcp        0      0 0.0.0.0:50020           0.0.0.0:*               LISTEN      29967/java
tcp        0      0 0.0.0.0:8040            0.0.0.0:*               LISTEN      30608/java
tcp        0      0 0.0.0.0:8042            0.0.0.0:*               LISTEN      30608/java
tcp        0      0 0.0.0.0:50090           0.0.0.0:*               LISTEN      30222/java
tcp        0      0 10.0.0.151:8020         0.0.0.0:*               LISTEN      29790/java
tcp        0      0 0.0.0.0:50070           0.0.0.0:*               LISTEN      29790/java
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      -  
tcp        0      0 10.0.0.151:8088         0.0.0.0:*               LISTEN      30424/java
tcp6       0      0 :::22                   :::*                    LISTEN      -  
like image 660
steven Avatar asked Apr 17 '14 08:04

steven


2 Answers

the reason is you should change 10.0.0.151:8088 to 0.0.0.0:8088. the method is open the yarn-site.xml, add:

<property> 
  <name>yarn.resourcemanager.webapp.address</name>
  <value>0.0.0.0:8088</value>
</property>
like image 96
user5319773 Avatar answered Sep 22 '22 07:09

user5319773


Here the problem is resouce manager is running, but the port(8030,8031,8032,8033,8088) occupied by resource manager uses tcp6 instead of tcp(see the left portion). You have two options either you can disable ipv6 in the linux system then restart yarn services.

or

Try modify your yarn-site.xml only on master node as follows. Don't modify the yarn-site.xml in slave nodes:

<configuration>
<property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
    <value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<configuration>

Specifying hostname causes the ports started as tcp6, even with out specifying those ports it would take the default value. Have a look at the following default ports

http://hadoop.apache.org/docs/r2.2.0/hadoop-yarn/hadoop-yarn-common/yarn-default.xml

After modiying yarn-site.xml restart your yarn-service

like image 22
SachinJ Avatar answered Sep 19 '22 07:09

SachinJ