Best practice for integrating Kafka and HBase

Question

what are best practices for "importing" streamed data from Kafka into HBase?

The usecase is as follows: Vehicle sensor data are streamed to Kafka. Afterwards, these sensordata must be transformed (i.e., deserialized from protobuf in humanreadable data) and stored within HBase.

1) Which toolset do you recommend (e.g., Kafka --> Flume --> HBase, Kafka --> Storm --> HBase, Kafka --> Spark Streaming --> HBase, Kafka --> HBase)

2) What is the best place for doing the protobuf deseralization (e.g., within Flume using interceptors)?

Thank you for your support.

Best, Thomas

Answer 1

I think you just need to do Kafka -> Storm -> HBase.

Storm: Storm spout will subscribe to Kafka topic.
Then Storm bolts can transform the data and write it into HBase.
You can use HBase client api in java to write data to HBase from Storm.

I suggested Storm because it actually processes one tuple at a time. In Spark streaming, a micro-batch is processed. However, if you would like to use common infrastructure for Batch and Stream processing then Spark might be a good choice.

If you end up using Spark then also your flow will be Kafka -> Spark -> HBase.