Getting Master not running exception

Question

I have been struggling to get Hbase working in my machine. I believe there is something wrong with my setup and I am not able to fix it. I get the master not running error but Jsp clearly shows it is running. After starting Hadoop and Hbase. I did JPS

yeshwanthvenkatesh@mymachineip /usr/local/Cellar/hbase/0.94.4/bin (master)$ jps
1609 Main
715 DataNode
985 TaskTracker
614 NameNode
886 JobTracker
1463 HRegionServer
1263 HQuorumPeer
814 SecondaryNameNode
1695
1349 HMaster
1842 Jps

When I try to invoke commands in Hbase shell I get the following errors.

hbase(main):001:0> list
TABLE
2013-04-25 16:20:28.933 java[1609:1703] Unable to load realm info from SCDynamicStore

ERROR: org.apache.hadoop.hbase.MasterNotRunningException: Retried 7 times

Here is some help for this command:
List all tables in hbase. Optional regular expression parameter could
be used to filter the output. Examples:

  hbase> list
  hbase> list 'abc.*'

hbase(main):002:0> scan 'ResultStore'
ROW                                                  COLUMN+CELL
13/04/25 16:26:13 INFO ipc.HBaseRPC: Problem connecting to server: mymachineip/129.10.3.181:60020

Here is my site.xml

<configuration>
    <property>
        <name>hbase.rootdir</name>
        <value>hdfs://129.10.3.181:8020/hbase</value>
    </property>
    <property>
        <name>hbase.zookeeper.property.dataDir</name>
        <value>/Users/yeshwanthvenkatesh/Documents/Hadoop/zookeeper</value>
    </property>
    <property>
        <name>hbase.master</name>
        <value>129.10.3.181:60000</value>
        <description>The host and port that the HBase master runs at.</description>
    </property>
    <property>
        <name>hbase.zookeeper.quorum</name>
        <value>129.10.3.181</value>
    </property>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
    <property>
        <name>hbase.cluster.distributed</name>
        <value>true</value>
    </property>
    <property>
        <name>hbase.regionserver.port</name>
        <value>60020</value>
        <description>The host and port that the HBase master runs at.</description>
    </property>
    <property>
        <name>hbase.zookeeper.property.clientPort</name>
        <value>2181</value>
    </property>
    <property>
        <name>hbase.zookeeper.property.dataDir</name>
        <value>/Users/yeshwanthvenkatesh/Documents/Hadoop/Daemons/zookeeper</value>
    </property>
</configuration>

Here is my etc/hosts file

127.0.0.1       localhost
129.10.3.181  mymachineip
255.255.255.255 broadcasthost
# For Hadoop
# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

I am not managing Zookeeper instance , so I have set that to true in my hbase-env.sh. What is that I am doing wrong ? The logs are not much of a help either.

13/04/25 16:08:52 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=129.10.3.181:2181 sessionTimeout=180000 watcher=hconnection
13/04/25 16:08:52 INFO zookeeper.ClientCnxn: Opening socket connection to server mymachineip/129.10.3.181:2181. Will not attempt to authenticate using SASL (unknown error)
13/04/25 16:08:52 INFO zookeeper.RecoverableZooKeeper: The identifier of this process is 2310@mymachineip
13/04/25 16:08:52 INFO zookeeper.ClientCnxn: Socket connection established to mymachineip/129.10.3.181:2181, initiating session
13/04/25 16:08:53 INFO zookeeper.ClientCnxn: Session establishment complete on server mymachineip/129.10.3.181:2181, sessionid = 0x13e42c9a109000a, negotiated timeout = 180000
13/04/25 16:08:53 INFO client.HConnectionManager$HConnectionImplementation: getMaster attempt 0 of 1 failed; no more retrying.
org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet
    at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1415)

    at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:1021)
    at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:150)
    at sun.proxy.$Proxy5.getProtocolVersion(Unknown Source)
    at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:183)
    at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:335)
    at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:312)
    at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:364)
    at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:682)
    at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:113)
    at org.apache.hadoop.hbase.master.HMasterCommandLine.stopMaster(HMasterCommandLine.java:175)
    at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:106)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
    at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:76)
    at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:1928)
13/04/25 16:08:53 INFO client.HConnectionManager$HConnectionImplementation: Closed zookeeper sessionid=0x13e42c9a109000a
13/04/25 16:08:53 INFO zookeeper.ZooKeeper: Session: 0x13e42c9a109000a closed
13/04/25 16:08:53 INFO zookeeper.ClientCnxn: EventThread shut down
13/04/25 16:08:53 ERROR master.HMasterCommandLine: Master not running

Answer 1

It seems your namenode or datanode is in safemode. So we need to leave the safe mode to insert data in hdfs or hbase table in hdfs. We can use following command to leave the safe mode of the namenode or datanode:

hdfs dfsadmin -safemode leave

On successful execution this message is shown:

Safe mode is OFF

And after this above error does not occur and the hbase shell runs properly.