Menu
Can a rolling deployment of a Kafka consumer group cause the group to freeze?
So let's consider this scenario,
So if you have a large enough cluster and it takes some time for the deployment to get completed on one machine (which is usually the case), Will this lead to a complete freeze in consumption?
If yes, What are the strategies to do a consumer group update in production
939
Kafka Rebalancing Consequences They will experience an interruption and subsequent lag between the latest messages consumed from the topic and the most recent messages available within the topic.
During the entire rebalancing process, i.e. as long as the partitions are not reassigned, consumers no longer process any data. By default, the rebalance timeout is fixed to 5 minutes which can be a very long period during which the increasing consumer-lag can become an issue.
If you want to prevent this from happening, you can either increase the timeout ( session.timeout.ms ), or make sure your consumer sends heartbeat more often ( heartbeat.interval.ms ). Heartbeats are basically embedded in poll() , thus, you need to make sure you call poll frequently enough.
Rebalance Triggers There are several causes for a consumer group rebalance to take place. A new consumer joins a consumer group, an existing consumer leaves a consumer group, or the broker thinks a consumer may have failed. As well as these, any other need for resources to be reassigned will trigger a rebalance.
Yes, that's definitely possible. There have been a number of recent improvements to mitigate the downtime during events like this. I'd recommend enabling one or both or the following features:
Static membership was added in 2.3 and can prevent a rebalance from occurring when a known member of the group is bounced. This requires both the client and the broker to be on version 2.3+
Incremental cooperative rebalancing enables the group to have faster rebalances AND allows individual members to continue consuming throughout the rebalance. You'll still see rebalances during a rolling deployment but they won't result in a complete freeze in consumption for the duration. This is completely client side so it will work with any brokers, but your clients should be on version 2.5.1+
195
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
