Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

What command shows all of the topics and offsets of partitions in Kafka?

Tags:

apache-kafka

I'm looking for a Kafka command that shows all of the topics and offsets of partitions. If it's dynamically would be perfect. Right now I'm using java code to see these information, but it's very inconvenient.

like image 363
Jack Avatar asked Jun 25 '15 22:06

Jack


People also ask

What is the command to find topics in Kafka?

admin. TopicCommand is a command-line tool that can alter, create, delete, describe and list topics in a Kafka cluster. TopicCommand can be executed using kafka-topics shell script (i.e. bin/kafka-topics.sh or bin\windows\kafka-topics.

Which file helps with offset details in Kafka?

Short Answer. If your Kafka topic is in Confluent Cloud, use the kafka-console-consumer command with the --partition and --offset flags to read from a specific partition and offset.

What is offset and partition in Kafka?

Offsets and Consumer PositionKafka maintains a numerical offset for each record in a partition. This offset acts as a unique identifier of a record within that partition, and also denotes the position of the consumer in the partition.


1 Answers

Kafka ships with some tools you can use to accomplish this.

List topics:

# ./bin/kafka-topics.sh --list --zookeeper localhost:2181 test_topic_1 test_topic_2 ... 

List partitions and offsets:

# ./bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --broker-info --group test_group --topic test_topic --zookeeper localhost:2181 Group           Topic                  Pid Offset          logSize         Lag             Owner test_group      test_topic             0   698020          698021          1              test_group-0 test_group      test_topic             1   235699          235699          0               test_group-1 test_group      test_topic             2   117189          117189          0               test_group-2 

Update for 0.9 (and higher) consumer APIs

If you're using the new apis, there's a new tool you can use: kafka-consumer-groups.sh.

./bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group count_errors --describe GROUP                          TOPIC                          PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             OWNER count_errors                   logs                           2          2908278         2908278         0               consumer-1_/10.8.0.55 count_errors                   logs                           3          2907501         2907501         0               consumer-1_/10.8.0.43 count_errors                   logs                           4          2907541         2907541         0               consumer-1_/10.8.0.177 count_errors                   logs                           1          2907499         2907499         0               consumer-1_/10.8.0.115 count_errors                   logs                           0          2907469         2907469         0               consumer-1_/10.8.0.126 
like image 196
chrsblck Avatar answered Nov 17 '22 05:11

chrsblck