I'm looking for a Kafka command that shows all of the topics and offsets of partitions. If it's dynamically would be perfect. Right now I'm using java code to see these information, but it's very inconvenient.
admin. TopicCommand is a command-line tool that can alter, create, delete, describe and list topics in a Kafka cluster. TopicCommand can be executed using kafka-topics shell script (i.e. bin/kafka-topics.sh or bin\windows\kafka-topics.
Short Answer. If your Kafka topic is in Confluent Cloud, use the kafka-console-consumer command with the --partition and --offset flags to read from a specific partition and offset.
Offsets and Consumer PositionKafka maintains a numerical offset for each record in a partition. This offset acts as a unique identifier of a record within that partition, and also denotes the position of the consumer in the partition.
Kafka ships with some tools you can use to accomplish this.
List topics:
# ./bin/kafka-topics.sh --list --zookeeper localhost:2181 test_topic_1 test_topic_2 ...
List partitions and offsets:
# ./bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --broker-info --group test_group --topic test_topic --zookeeper localhost:2181 Group Topic Pid Offset logSize Lag Owner test_group test_topic 0 698020 698021 1 test_group-0 test_group test_topic 1 235699 235699 0 test_group-1 test_group test_topic 2 117189 117189 0 test_group-2
If you're using the new apis, there's a new tool you can use: kafka-consumer-groups.sh
.
./bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group count_errors --describe GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG OWNER count_errors logs 2 2908278 2908278 0 consumer-1_/10.8.0.55 count_errors logs 3 2907501 2907501 0 consumer-1_/10.8.0.43 count_errors logs 4 2907541 2907541 0 consumer-1_/10.8.0.177 count_errors logs 1 2907499 2907499 0 consumer-1_/10.8.0.115 count_errors logs 0 2907469 2907469 0 consumer-1_/10.8.0.126
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With