HBase - What's the difference between WAL and MemStore?

Question

I am trying to understand the HBase architecture. I can see two different terms are used for same purpose.

Write Ahead Logs and Memstore, both are used to store new data that hasn't yet been persisted to permanent storage.

What's the difference between WAL and MemStore?

Update:

WAL - is used to recover not-yet-persisted data in case a server crashes. MemStore - stores updates in memory as Sorted Keyvalue.

It seems lot of duplication of data before writing the data to Disk.

Answer 1

WAL is for recovery NOT for data duplication.(further see my answer here)

Pls go through below to understand more...

• A Hbase Store hosts a MemStore and 0 or more StoreFiles (HFiles). A Store corresponds to a column family for a table for a given region.

• The Write Ahead Log (WAL) records all changes to data in HBase, to file-based storage. if a RegionServer crashes or becomes unavailable before the MemStore is flushed, the WAL ensures that the changes to the data can be replayed.

• With a single WAL per RegionServer, the RegionServer must write to the WAL serially, because HDFS files must be sequential. This causes the WAL to be a performance bottleneck.

• WAL can be disabled to improve performance bottleneck. This is done by calling the Hbase client field

Mutation.writeToWAL(false)

General Note : Its general practice that while doing bulkloading data, WAL is disabled to get speed. But side effect is if you disable WAL you cant get back data to replay if in case any memory crashes.

More over if you use solr+ HBASE + LILY, i.e LILY Morphiline NRT indexes with hbase then it will work on WAL if you disable WAL for performance reasons, then Solr NRT indexing wont work. since Lily works on WAL.

please have a look at Hbase architecture section