How to fetch recent messages from Kafka topic

Question

Do we have any option like fetching recent 10/20/ etc., messages from Kafka topic. I can see --from-beginning option to fetch all messages from the topic but if I want to fetch only few messages first, last, middle or latest 10. do we have some options?

Answer 1

First N messages

You can use --max-messages N in order to fetch the first N messages of a topic.

For example, to get the first 10 messages, run

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning  --max-messages 10

Next N messages

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --max-messages 10

Last N messages

To get the last N messages, you need to define a specific partition and the offset:

bin/kafka-simple-consumer-shell.sh --bootstrap-server localhost:9092 --topic test--partition testPartition --offset yourOffset

M to N messages

Again, for this case you'd have to define both the partition and the offset. For example, you can run the following in order to get N messages starting from an offset of your choice:

bin/kafka-simple-consumer-shell.sh --bootstrap-server localhost:9092 --topic test--partition testPartition --offset yourOffset --max-messages 10

If you don't want to stick to the binaries, I would suggest you to use kt which is a Kafka command line tool with more options and functionality.

---

For more details refer to the article How to fetch specific messages in Apache Kafka