Hbase error zookeeper exists failed after 3 retiries

Question

I am using HBASE 0.94.8 standalone mode in Ubuntu. Its working fine i am able to do every operations in Hbase-shell. But after i logged of my system its giving following error

15/07/28 15:10:30 ERROR zookeeper.RecoverableZooKeeper: ZooKeeper exists failed after 3 retries
15/07/28 15:10:30 WARN zookeeper.ZKUtil: hconnection-0x14ed40513350009 Unable to set watcher on znode (/hbase)
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
    at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1041)
    at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:172)
    at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:450)
    at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.checkIfBaseNodeAvailable(ZooKeeperNodeTracker.java:208)
    at org.apache.hadoop.hbase.zookeeper.RootRegionTracker.waitRootRegionLocation(RootRegionTracker.java:77)
    at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:885)
    at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:998)
    at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:896)
    at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:998)
    at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:900)
    at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:857)

Yes sure i have searched a lot. I have found some information ZooKeeper exists failed after 3 retries. May be this error is because zookeeper is stopped. But i don't know to to restart it again. I tried to start Hbase and thrift again but still this is issue.

This command ps axww | grep QuorumPeerMain gives me following output:

 6162 pts/2    S+     0:00 grep --color=auto QuorumPeerMain

Hbase starts working if i restart my system. But i want proper solution.

---

Temporary solution

with following command i grep this process of HBASE:

ps -fe grep | hbase

and then kill all process of HBASE :

kill -9 4555//assuming 4555 is process id of hbase

Then restarted hbase with sudo and thrift and it start working but i want permanent solution. Because if i am using HBASE in server (means not local machine) i can't restart HBASE everytime.

Answer 1

Issue:

Hbase error zookeeper exists failed after 3 retiries clearly indicates that zookeeper quorum is not running - most probable cause can be some inconsistency with your zookeeper.quorum setting in conf/hbase-site.xml, the minimal has to be:

<configuration>
  <property>
    <name>hbase.rootdir</name>
    <value>file:///home/testuser/hbase</value>
  </property>
  <property>
    <name>hbase.zookeeper.property.dataDir</name>
    <value>/home/testuser/zookeeper</value>
  </property>
</configuration>

In the next section it is succinctly mentioned why zookeeper is required and how can one verify if it's running.

---

An overview:

Pre-assuming from your text (standalone setup) - you're mixing things up. Zookeeper in simple words manages HBase, and is a must requirement.

By default HBase itself handles zookeeper setup, start-stop (though one can change) - to verify look into the file conf/hbase-evn.sh (in your hbase directory) there must be a line:

export HBASE_MANAGES_ZK=true

Basically tells HBase whether it should manage its own instance of Zookeeper or not. In case it is set to false, edit to true.

Now for verification there's a helpful command (forget about the ps and then grep):

$ jps

the command will list all the java processes (HBase is itself a Java application) on the machine i.e. the probable output has to be (for a minimal standalone HBase setup):

62019 Jps
61098 HMaster        
61233 HRegionServer     
61003 HQuorumPeer

Don't just kill the HBase process, instead use the start-stop utility:

$ ./bin/stop-hbase.sh

make the neccessary changes and start it again:

$ ./bin/start-hbase.sh

P.S. I could have misinterpreted your question (completely), do let me know in the comments I'll get back to you again and get the solution right - for the upcoming SO visitors.