Menu
How should one check if a specific consumer group, fully consumed a topic?
This is the equivalent of checking if a queue is empty in standard queuing system.
This isn't obvious as each consumer only see the partitions it is given hence it isn't aware if other consumers in it's group consumed their part.
I guess this should be done somehow by AdminUtils or ZkUtils but I cant seem to find the right way.
886
Use kafka-consumer-groups.sh to list all consumer groups.
The Kafka cluster retains all published messages—whether or not they have been consumed—for a configurable period of time. For example if the log retention is set to two days, then for the two days after a message is published it is available for consumption, after which it will be discarded to free up space.
You can use a KafkaConsumer node in a message flow to subscribe to a specified topic on a Kafka server. The KafkaConsumer node then receives messages that are published on the Kafka topic, as input to the message flow.
Please use kafka-consumer-groups.sh to check consumer offset.
(The kafka-consumer-offset-checker.sh has similar functionality but has been deprecated in 0.9.0.0.)
bin/kafka-consumer-groups.sh --bootstrap-server xxx:9092--group xxx-group-id --describe --new-consumer
GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG OWNER consumer.test.0529.095217 secured_topic 0 586 3333 2747 consumer-1_/10.139.0.4 consumer.test.0529.095217 secured_topic 3334 3334 0 consumer-1_/10.139.0.4 consumer.test.0529.095217 secured_topic 2 0 3333 3333 consumer-1_/10.139.0.4
121
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
