We have an issue where it appears that a Kafka consumer is not receiving messages published to a topic. (I say appears as I have not yet got to the bottom of this, and I could be wrong.)
I am using Spring for Apache Kafka and my consumer is actually a method annotated with @KafkaListener
.
This issue is intermittent and I am having trouble recreating it.
Is there a way for me to look at the Kafka broker's logs, or any other tool to help me find out the offset for my consumer? I want concrete evidence that my consumer is receiving the message or not.
Consumer offset is recorded in Kafka so if the consumer processing the partition in the consumer group goes down and when the consumer comes back, the consumer will read the offset to start reading the messages from the topic from where it is left off. This avoids duplication in message consumption.
Use the kafka-consumer-groups along with the consumer group id followed by a describe. You will see 2 entries related to offsets – CURRENT-OFFSET and LOG-END-OFFSET for the partitions in the topic for that consumer group. CURRENT-OFFSET is the current offset for the partition in the consumer group.
If your Kafka topic is in Confluent Cloud, use the kafka-console-consumer command with the --partition and --offset flags to read from a specific partition and offset. You can also read messages from a specified partition and offset using the Confluent Cloud Console: Run it.
OFFSET IN KAFKA In other words, it is a position within a partition for the next message to be sent to a consumer. A simple integer number which is actually used by Kafka to maintain the current position of a consumer.
Take a look at the kafka-consumer-groups tool, which can be used to check offsets and lag of consumers (consumer has to be active at the time you run this command).
./kafka-consumer-groups --bootstrap-server 127.0.0.1:9092 --new-consumer --describe --group console-consumer-55936
GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG OWNER
console-consumer-55936 test 0 6 6 0 consumer-1_/192.168.0.83
console-consumer-55936 test 1 1 1 0 consumer-1_/192.168.0.83
console-consumer-55936 test 2 1 1 0 consumer-1_/192.168.0.83
console-consumer-55936 test 3 1 1 0 consumer-1_/192.168.0.83
console-consumer-55936 test 4 2 2 0 consumer-1_/192.168.0.83
console-consumer-55936 test 5 1 1 0 consumer-1_/192.168.0.83
console-consumer-55936 test 6 1 1 0 consumer-1_/192.168.0.83
console-consumer-55936 test 7 2 2 0 consumer-1_/192.168.0.83
console-consumer-55936 test 8 1 1 0 consumer-1_/192.168.0.83
This should allow you to track whether anything is actually being consumed or not.
Besides using ./kafka-consumer-groups as described in the other answer which is correct and perfectly valid you can also use a Windows GUI application that shows you the same information about Consumer_groups and their offset/lag:
http://www.kafkatool.com/download.html
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With