Hbase Region server shutdown

Question

I am Mohanselvakumar and am new to the Hbase system, requesting to help me on the following with one of my hbase server.

The Scenario is I have sett-ed up 10 Node Cluster with 2 Name Node (one Primary and one Secondary).

And on the 10 Nodes, am running Datanode as well as Regionserver service on all of the nodes.

Now the problem is out of the 10 Node, one of the node is continuously going down because of the following exception;

Frequent Error No:01

2013-01-25 12:10:51,918 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server serverName=DC1AuthDFSC1D1.cidr.gov.in,6020,1359093222948, load=(requests=10, regions=876, usedHeap=1925, maxHeap=11897): regionserver:6020-0x3c13d364d92db2-0x3c13d364d92db2 regionserver:6020-0x3c13d364d92db2-0x3c13d364d92db2 received expired from ZooKeeper, aborting
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired
    at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:328)
    at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:246)
    at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
    at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506)
2013-01-25 12:10:51,931 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics: request=0.0, regions=876, stores=992, storefiles=2380, storefileIndexSize=769, memstoreSize=19, compactionQueueSize=0, flushQueueSize=0, usedHeap=2043, maxHeap=11897, blockCacheSize=808359784, blockCacheFree=1686753176, blockCacheCount=11005, blockCacheHitCount=80692, blockCacheMissCount=149039, blockCacheEvictedCount=0, blockCacheHitRatio=35, blockCacheHitCachingRatio=87
2013-01-25 12:10:51,931 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: regionserver:6020-0x3c13d364d92db2-0x3c13d364d92db2 regionserver:6020-0x3c13d364d92db2-0x3c13d364d92db2 received expired from ZooKeeper, aborting
2013-01-25 12:10:51,932 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down
2013-01-25 12:10:54,703 INFO org.apache.hadoop.hbase.regionserver.LogRoller: LogRoller exiting.
2013-01-25 12:10:58,660 INFO org.apache.hadoop.hbase.regionserver.CompactSplitThread: regionserver6020.compactor exiting
2013-01-25 12:11:00,866 INFO org.apache.hadoop.hbase.regionserver.MemStoreFlusher: regionserver6020.cacheFlusher exiting
2013-01-25 12:11:02,647 INFO org.apache.hadoop.ipc.HBaseServer: Stopping server on 6020
2013-01-25 12:11:02,648 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 0 on 6020: exiting
2013-01-25 12:11:02,648 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 5 on 6020: exiting
2013-01-25 12:11:02,648 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 3 on 6020: exiting
2013-01-25 12:11:02,648 INFO org.apache.hadoop.ipc.HBaseServer: Stopping IPC Server listener on 6020
2013-01-25 12:11:02,648 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 7 on 6020: exiting
2013-01-25 12:11:02,648 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 6 on 6020: exiting
2013-01-25 12:11:02,648 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 5 on 6020: exiting
2013-01-25 12:11:02,648 INFO org.apache.hadoop.ipc.HBaseServer: Stopping IPC Server Responder
2013-01-25 12:11:02,649 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 3 on 6020: exiting
2013-01-25 12:11:02,649 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 0 on 6020: exiting

---

Frequent Error No:02

    2013-01-25 12:11:19,106 INFO org.apache.hadoop.hbase.regionserver.Leases: regionserver6020 closing leases
    2013-01-25 12:11:19,106 INFO org.apache.hadoop.hbase.regionserver.Leases: regionserver6020 closed leases
    2013-01-25 12:11:19,136 INFO org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: Closed zookeeper sessionid=0x13c13d323792d99
    2013-01-25 12:11:19,138 INFO org.apache.zookeeper.ZooKeeper: Session: 0x13c13d323792d99 closed
    2013-01-25 12:11:19,138 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down
    2013-01-25 12:11:19,143 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver6020 exiting
    2013-01-25 12:11:19,152 INFO org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=Thread[Thread-14,5,main]
    2013-01-25 12:11:19,152 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: Shutdown hook
    2013-01-25 12:11:19,152 INFO org.apache.hadoop.hbase.regionserver.ShutdownHook: Starting fs shutdown hook thread.
    2013-01-25 12:11:19,153 ERROR org.apache.hadoop.hdfs.DFSClient: Exception closing file 

Answer 1

Zookeeper expiration can happen due to low memory on the node, GCs that are too long etc. see section 12.9.2.7 here.

Answer 2

You can add session timeout in your hbase-site.xml like this

      <property>
        <name>zookeeper.session.timeout</name>
        <value>60000</value>
      </property>

Hopping this help