kafka node js client compression issue with snappy

Question

I am using kafka-node (https://github.com/SOHU-Co/kafka-node) consumer to retrieve data. I think the data which I get is compressed with SNAPPY. How do I decompress the data after I get it. I tried using node-snappy (https://github.com/kesla/node-snappy) to decompress the data, but it didn't work.

Is there any option in the library to set the compression to none?

Anyone used kafka-node library to get data from kafka..??

Thanks, chandu

Answer 1

I also encountered these exact problems. I found a solution, at last! You can use kafkacat ('like netcat for kafka') download here, which requires librdkafka. This enables you to interact with Kafka from the command line using the librdkafka C/C++ library. (no JVM required =D )

Now that we have those dependencies taken care of we can use this fun node.js repo: node-kafkacat

You'll find that there's likely enough documentation between those three libraries to get you started and unlike some of the other kafka-node modules on github, they seem to have been updated fairly recently.

I have only successfully installed on Linux and Mac so far but things are working great with our Apache/Java environment. I'm not the author of any of these packages, btw - just some guy who kept hoping your question would be answered over the last couple weeks.

Answer 2

By default, compression is determined by the producer through the configuration property 'compression.type'. Currently gzip, snappy and lz4 are supported. kafka-node should automatically uncompress both gzip and snappy. I just tried it with snappy and this works out of the box. lz4 appears not to be implemented by kafka-node at this point.

Compression can be configured using the configuration property 'compression.type':

• Per broker: https://kafka.apache.org/documentation/#brokerconfigs. Default: producer determines compression
• Per topic: https://kafka.apache.org/documentation/#brokerconfigs (scroll further down). Default: producer determines compression
• Per producer: https://kafka.apache.org/documentation/#producerconfigs. Default: none

If you can't influence the producer, you could therefore consider overriding the topic configuration.

Also, note that kafka-node may not directly return the data as you expect. For instance, my (key, value) pair is of type (integer, integer), and I have to call both message.key.readIntBE() and message.value.readIntBE() to extract the integer values.

Hope this helps!