HBase oldWALs: what it is and how can I clean it?

Question

We are running out of space in our small hadoop cluster so I was checking disk usage on HDFS and I saw that most of the space was occupied by the /hbase/oldWALs folder.

I have checked in the "HBase Definitive Book" and others books, web-site and I have also search my issue on google but I didn't find a proper response...

So I would like to know what does this folder, what is use for and also how can I free space from this folder without breaking everything...

If it's related to a specific version... our cluster is under 5.3.0-1.cdh5.3.0.p0.30 from cloudera (hbase 0.98.6).

Thx for your help!

Answer 1

FYI

I have post this issue in the hbase user list. Here is the answer from Enis Söztutar (an hbase committer) and how I solved it:

The folder gets cleaned regularly by a chore in master. When a WAL file is not needed any more for recovery purposes (when HBase can guaratee HBase has flushed all the data in the WAL file), it is moved to the oldWALs folder for archival. The log stays there until all other references to the WAL file are finished. There is currently two services which may keep the files in the archive dir. First is a TTL process, which ensures that the WAL files are kept at least for 10 min. This is mainly for debugging. You can reduce this time by setting hbase.master.logcleaner.ttl configuration property in master. It is by default 600000. The other one is replication. If you have replication setup, the replication processes will hang on to the WAL files until they are replicated. Even if you disabled the replication, the files are still referenced.

You can look at the logs from master from classes (LogCleaner, TimeToLiveLogCleaner, ReplicationLogCleaner) to see whether the master is actually running this chore and whether it is getting any exceptions.

The replication was indeed unable on the all cluster but in the past it was enable because we used the hbase-indexer to copy data from HBase to Solr and this mecanism is based on replication.

I have run this commands on hbase shell:

hbase(main):005:0> list_peers
PEER_ID CLUSTER_KEY STATE TABLE_CFS
Indexer_profilesIndexer m1.prod.ps,m2.prod.ps,m3.prod.ps:2181:/ngdata/sep/hbase-slave/Indexer_profilesIndexer DISABLED nil
1 row(s) in 0.0070 seconds

hbase(main):006:0> remove_peer 'Indexer_profilesIndexer'
0 row(s) in 0.0050 seconds


hbase(main):007:0> list_peers
PEER_ID CLUSTER_KEY STATE TABLE_CFS
0 row(s) in 0.0020 seconds

And finally I have deleted the oldsWALs folder on hdfs!

The folder no longer grows!