Check all the lags in Kafka

Question

I want to make sure that I do not lost any message in Kafka even if my consumers are down.

If I have one hour in log roll, if my consumers are down for more than one hour, I will lose messages which were not consumed.

Is there any way to check all the lags for all the partitions easily in Kafka 0.10.2.0?

I found many ways to none are working well.

Answer 1

You can use kafka-consumer-groups.sh command to find out the lag.

bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group group1

In this example i am saying show me all the topics that group1 is listening to and whats the lag, my consumer was down for last few min. and it has 4 pending messages so this is what i get