Hbase master not starting correctly

Question

I'm using Hadoop2.4.0 / Hbase 0.98.0 / Hive 0.14.0

Hadoop and HBase were running fine until I restarted my HMaster. The following error appears in hbase-hduser-master-master.log file :

 2015-02-17 05:46:15,157 INFO  [master:master:60000] master.TableNamespaceManager: Namespace table not found. Creating...
    2015-02-17 05:46:15,193 DEBUG [master:master:60000] lock.ZKInterProcessLockBase: Acquired a lock for /hbase/table-lock/hbase:namespace/write-master:600000000000004
    2015-02-17 05:46:15,212 DEBUG [master:master:60000] lock.ZKInterProcessLockBase: Released /hbase/table-lock/hbase:namespace/write-master:600000000000004
    2015-02-17 05:46:15,212 FATAL [master:master:60000] master.HMaster: Master server abort: loaded coprocessors are: []
    2015-02-17 05:46:15,213 FATAL [master:master:60000] master.HMaster: Unhandled exception. Starting shutdown.
    org.apache.hadoop.hbase.TableExistsException: hbase:namespace
        at org.apache.hadoop.hbase.master.handler.CreateTableHandler.prepare(CreateTableHandler.java:120)
        at org.apache.hadoop.hbase.master.TableNamespaceManager.createNamespaceTable(TableNamespaceManager.java:232)
        at org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:86)
        at org.apache.hadoop.hbase.master.HMaster.initNamespace(HMaster.java:1049)
        at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:913)
        at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:606)
        at java.lang.Thread.run(Unknown Source)
    2015-02-17 05:46:15,214 INFO  [master:master:60000] master.HMaster: Aborting
    2015-02-17 05:46:15,214 INFO  [master,60000,1424180766819-BalancerChore] balancer.BalancerChore: master,60000,1424180766819-BalancerChore exiting
    2015-02-17 05:46:15,215 INFO  [master,60000,1424180766819-ClusterStatusChore] balancer.ClusterStatusChore: master,60000,1424180766819-ClusterStatusChore exiting
    2015-02-17 05:46:15,215 INFO  [CatalogJanitor-master:60000] master.CatalogJanitor: CatalogJanitor-master:60000 exiting
    2015-02-17 05:46:15,216 DEBUG [master:master:60000] master.HMaster: Stopping service threads
    2015-02-17 05:46:15,216 INFO  [master:master:60000] ipc.RpcServer: Stopping server on 60000
    2015-02-17 05:46:15,216 INFO  [RpcServer.listener,port=60000] ipc.RpcServer: RpcServer.listener,port=60000: stopping
    2015-02-17 05:46:15,218 INFO  [RpcServer.responder] ipc.RpcServer: RpcServer.responder: stopped
    2015-02-17 05:46:15,218 INFO  [RpcServer.responder] ipc.RpcServer: RpcServer.responder: stopping
    2015-02-17 05:46:15,218 INFO  [master:master:60000.oldLogCleaner] cleaner.LogCleaner: master:master:60000.oldLogCleaner exiting
    2015-02-17 05:46:15,218 INFO  [master:master:60000.oldLogCleaner] master.ReplicationLogCleaner: Stopping replicationLogCleaner-0x14b97c83f580008, quorum=slave:2181,master:2181, baseZNode=/hbase
    2015-02-17 05:46:15,219 INFO  [master:master:60000.archivedHFileCleaner] cleaner.HFileCleaner: master:master:60000.archivedHFileCleaner exiting
    2015-02-17 05:46:15,219 INFO  [master:master:60000] master.HMaster: Stopping infoServer
    2015-02-17 05:46:15,223 INFO  [master:master:60000.oldLogCleaner] zookeeper.ZooKeeper: Session: 0x14b97c83f580008 closed
    2015-02-17 05:46:15,223 INFO  [master:master:60000-EventThread] zookeeper.ClientCnxn: EventThread shut down
    2015-02-17 05:46:15,229 INFO  [master:master:60000] mortbay.log: Stopped [email protected]:60010
    2015-02-17 05:46:15,236 DEBUG [master:master:60000] catalog.CatalogTracker: Stopping catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@19f9598
    2015-02-17 05:46:15,236 INFO  [master:master:60000] client.HConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x14b97c83f580007
    2015-02-17 05:46:15,237 INFO  [master:master:60000-EventThread] zookeeper.ClientCnxn: EventThread shut down
    2015-02-17 05:46:15,238 INFO  [master:master:60000] zookeeper.ZooKeeper: Session: 0x14b97c83f580007 closed
    2015-02-17 05:46:15,238 INFO  [master,60000,1424180766819.splitLogManagerTimeoutMonitor] master.SplitLogManager$TimeoutMonitor: master,60000,1424180766819.splitLogManagerTimeoutMonitor exiting
    2015-02-17 05:46:15,243 INFO  [main-EventThread] zookeeper.ClientCnxn: EventThread shut down
    2015-02-17 05:46:15,243 INFO  [master:master:60000] zookeeper.ZooKeeper: Session: 0x14b97c83f580006 closed
    2015-02-17 05:46:15,243 INFO  [master:master:60000] master.HMaster: HMaster main thread exiting
    2015-02-17 05:46:15,243 ERROR [main] master.HMasterCommandLine: Master exiting
    java.lang.RuntimeException: HMaster Aborted
        at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:192)
        at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:134)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
        at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2785)

what's wrong here and what HMaster Aborted means ?

for more information this is what my hbase-site.xml looks like:

<property>
<name>hbase.rootdir</name>
<value>hdfs://master:54310/hbase</value>
</property>
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
</property>
<property>
<name>hbase.zookeeper.quorum</name>
<value>master,slave</value>
</property>
<property>
<name>hbase.zookeeper.property.clientPort</name>
<value>2181</value>
</property>
<property>
<name>hbase.zookeeper.property.dataDir</name>
<value>/usr/local/hbase/zookeeper</value>
</property>

Answer 1

I ran into this problem today! My solution is as follows:

Step 1:stop Hbase.

Step 2:run the follow command

hbase org.apache.hadoop.hbase.util.hbck.OfflineMetaRepair

This command is used to repair MetaData of Hbase

Step 3:delete the data in zookeeper (WARNING It will make you lost you old data)

./opt/cloudera/parcels/CDH-5.1.0-1.cdh5.1.0.p0.53/lib/zookeeper/bin/zkCli.sh

you can use ls / to scan the data in zookeeper

use rmr /hbase to delete the hbase's data in zookeeper

Step 4：Start hbase