Menu
How long does kafka store the offset of a consumer-group after all consumers in that group fail? Is there a configuration variable for this?
981
As each message is received by Kafka, it allocates a message ID to the message. Kafka then maintains the message ID offset on a by consumer and by partition basis to track consumption. Kafka brokers keep track of both what is sent to the consumer and what is acknowledged by the consumer by using two offset values.
offsets. retention. minutes allows you to move the offset back to the beginning if it isn't changed within a set period of time.
Consumer offset is recorded in Kafka so if the consumer processing the partition in the consumer group goes down and when the consumer comes back, the consumer will read the offset to start reading the messages from the topic from where it is left off. This avoids duplication in message consumption.
Kafka maintains a numerical offset for each record in a partition. This offset acts as a unique identifier of a record within that partition, and also denotes the position of the consumer in the partition.
The right property name is:
offsets.retention.minutes
from https://kafka.apache.org/documentation/#brokerconfigs
114
The value can be configured in kafka broker using:
offsets.retention.minutes
The default is 24 hours.
See: the Kafka broker config docs.
35
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With