How to get message by key from kafka topic

Question

I try to get message by key from kafka. I found the only one solution is to use StateStore but I think it might be not a good idea. How to get message by key from kafka topic? Is it a good idea to use StateStore for this operation?

Answer 1

You can't "get messages by key from Kafka".

One solution, if practical, would be to have as many partitions as keys and always route messages for a key to the same partition.

Answer 2

Every record written to Kafka can optionally have a key (but it doesn't have to!), the key can be accessed a number of ways:

Console consumer:

$ kafka-console-consumer --bootstrap-server <servername>:9092 --topic topicname --from-beginning --property print.key=true --property key.separator=:

kafkacat:

$ kafkacat -b <servername>:9092 -C -t topicname -o beginning -K :

Java Consumer API:

ConsumerRecord#key()

Kafka isn't indexed by key though, it's indexed by offset, and optionally by timestamp. If you need to lookup a key then you'll need to materialize the data to a system that's been designed to lookup by key: relational database, key value store, or some index. You can do this pretty easily with Kafka Connect, or if you'd like to build it into your service you could use the interactive queries feature of Kafka Streams.