Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to check consumer offsets when the offset store is Kafka?

Tags:

apache-kafka

From 0.8.1.1 release, Kafka provides the provision for storage of offsets in Kafka, instead of Zookeeper (see this). I'm not able to figure out how to check the details of offsets consumed, as the current tools only provide consumer offset count checks for zookeeper only.(I'm referring to this)

If there are any tools available to check consumer offset, please let me know.

like image 514
Abhiram Avatar asked Dec 01 '15 11:12

Abhiram


People also ask

How do you think Kafka is storing consumer offset?

Kafka store the offset commits in a topic, when consumer commit the offset, kafka publish an commit offset message to an "commit-log" topic and keep an in-memory structure that mapped group/topic/partition to the latest offset for fast retrieval.

How do I find my consumer offset?

How to find the current consumer offset? Use the kafka-consumer-groups along with the consumer group id followed by a describe. You will see 2 entries related to offsets – CURRENT-OFFSET and LOG-END-OFFSET for the partitions in the topic for that consumer group.

How do you check the offset of a topic in Kafka?

Short Answer. If your Kafka topic is in Confluent Cloud, use the kafka-console-consumer command with the --partition and --offset flags to read from a specific partition and offset. You can also read messages from a specified partition and offset using the Confluent Cloud Console: Run it.

Which consumer in Kafka will commit the current offset?

By default, as the consumer reads messages from Kafka, it will periodically commit its current offset (defined as the offset of the next message to be read) for the partitions it is reading from back to Kafka.


2 Answers

The following straight command gives the enough details:

kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-second-application

You will get the details like this

TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                     HOST            CLIENT-ID
first_topic     0          4               4               0               consumer-1-7cb31cf3-1621-4635-8f95-6ae85215b31b /10.200.237.53  consumer-1
first_topic     1          3               3               0               consumer-1-7cb31cf3-1621-4635-8f95-6ae85215b31b /10.200.237.53  consumer-1
first_topic     2          3               3               0               consumer-1-7cb31cf3-1621-4635-8f95-6ae85215b31b /10.200.237.53  consumer-1
first-topic     0          4               4               0               -                                               -               -
like image 71
KayV Avatar answered Sep 28 '22 07:09

KayV


I'am using kafka 0.8.2 with offsets stored in kafka. This tools works good for me:

./kafka-run-class.sh kafka.tools.ConsumerOffsetChecker 
        --topic your-topic 
        --group your-consumer-group
        --zookeeper localhost:2181

You get all informations you need: topic size, consumer lag, owner.

like image 34
codejitsu Avatar answered Sep 28 '22 08:09

codejitsu