Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to determine a Kafka consumer's offset

We have an issue where it appears that a Kafka consumer is not receiving messages published to a topic. (I say appears as I have not yet got to the bottom of this, and I could be wrong.)

I am using Spring for Apache Kafka and my consumer is actually a method annotated with @KafkaListener.

This issue is intermittent and I am having trouble recreating it.

Is there a way for me to look at the Kafka broker's logs, or any other tool to help me find out the offset for my consumer? I want concrete evidence that my consumer is receiving the message or not.

like image 390
vegemite4me Avatar asked Mar 02 '17 10:03

vegemite4me


People also ask

What determines Kafka consumer offset?

Consumer offset is recorded in Kafka so if the consumer processing the partition in the consumer group goes down and when the consumer comes back, the consumer will read the offset to start reading the messages from the topic from where it is left off. This avoids duplication in message consumption.

How do I find my consumer group offset?

Use the kafka-consumer-groups along with the consumer group id followed by a describe. You will see 2 entries related to offsets – CURRENT-OFFSET and LOG-END-OFFSET for the partitions in the topic for that consumer group. CURRENT-OFFSET is the current offset for the partition in the consumer group.

How do you check the offset of a topic in Kafka?

If your Kafka topic is in Confluent Cloud, use the kafka-console-consumer command with the --partition and --offset flags to read from a specific partition and offset. You can also read messages from a specified partition and offset using the Confluent Cloud Console: Run it.

How does Kafka define offset?

OFFSET IN KAFKA In other words, it is a position within a partition for the next message to be sent to a consumer. A simple integer number which is actually used by Kafka to maintain the current position of a consumer.


2 Answers

Take a look at the kafka-consumer-groups tool, which can be used to check offsets and lag of consumers (consumer has to be active at the time you run this command).

./kafka-consumer-groups --bootstrap-server 127.0.0.1:9092 --new-consumer --describe --group console-consumer-55936

GROUP                          TOPIC                          PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             OWNER
console-consumer-55936         test                           0          6               6               0               consumer-1_/192.168.0.83
console-consumer-55936         test                           1          1               1               0               consumer-1_/192.168.0.83
console-consumer-55936         test                           2          1               1               0               consumer-1_/192.168.0.83
console-consumer-55936         test                           3          1               1               0               consumer-1_/192.168.0.83
console-consumer-55936         test                           4          2               2               0               consumer-1_/192.168.0.83
console-consumer-55936         test                           5          1               1               0               consumer-1_/192.168.0.83
console-consumer-55936         test                           6          1               1               0               consumer-1_/192.168.0.83
console-consumer-55936         test                           7          2               2               0               consumer-1_/192.168.0.83
console-consumer-55936         test                           8          1               1               0               consumer-1_/192.168.0.83

This should allow you to track whether anything is actually being consumed or not.

like image 184
Sönke Liebau Avatar answered Oct 11 '22 15:10

Sönke Liebau


Besides using ./kafka-consumer-groups as described in the other answer which is correct and perfectly valid you can also use a Windows GUI application that shows you the same information about Consumer_groups and their offset/lag:

http://www.kafkatool.com/download.html

like image 38
Aleja_Vigo Avatar answered Oct 11 '22 13:10

Aleja_Vigo