Clear a Kafka consumer group with stuck members

Question

I have the problem that a lot of my Kafka clients from one consumer group did not shutdown correctly and thus the Kafka cluster thinks they are still connected. Thus, I cannot connect to the consumer group with the new version of my client. It will be stuck in the rebalancing step.

According to documentation they should be removed after session.timeout.ms or maximum group.max.session.timeout.ms. At first I tried to set session.timeout.ms to 30000 milliseconds (30 seconds), but it was not listed on the startup of Kafka. group.max.session.timeout.ms was set to 300000 milliseconds (5 minutes) at that point in time. The consumers were not deleted after 30 seconds.

After that I tried to reduce group.max.session.timeout.ms to 30000 milliseconds (30 seconds) and restarted Kafka. However, again all clients just remained in the consumer group.

Now, it's about 2 hours later and the clients are still attached to the consumer group.

I tried to delete the consumer group with:

bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
    --delete --group GroupName

which gives me:

* Group 'GroupName' could not be deleted due to: NON_EMPTY_GROUP

Unfortunately, there does not seem to be a --force flag.

Next, I tried to list all members of the consumer group with this command:

bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
    --members --group GroupName --describe

This gives me 40-50 consumer group members (all of them must be inactive, because the only active consumer is stopped).

Is there a way how I can get Kafka to purge all consumers from this group or force it to delete the whole consumer group?

Answer 1

With

• Group 'GroupName' could not be deleted due to: NON_EMPTY_GROUP

meaning the consumer group is still actively connecting to the server You have to first kill the process before you delete the consumer group. This works for me.

ps -ef | grep GroupName | grep -v grep | awk '{print $2}' | xargs kill
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
--delete --group GroupName