Menu
I have a kafka topic with 1 partition. if it had 100 messages in it the offset would be from 0.99.
According the kafka retention policy all of the messages will be wiped out after the specified period.
and i am sending 100 new messages to the topic once all have been wiped out(after retention period). Now, where would the new offset of the message start from? is it From 100 or from 0??
I am trying to understand whether the new offsets will be 100-199 or 0-99?
597
retention. minutes controlling how long Kafka will remember offsets in the special topic. The default value is 1,440 minutes (24 hours).
Consumer offset is recorded in Kafka so if the consumer processing the partition in the consumer group goes down and when the consumer comes back, the consumer will read the offset to start reading the messages from the topic from where it is left off. This avoids duplication in message consumption.
Kafka maintains a numerical offset for each record in a partition. This offset acts as a unique identifier of a record within that partition, and also denotes the position of the consumer in the partition.
Kafka allows for auto.offset.reset to take the value of: - latest - the default value, which represents the most recent offset available in a topic- earliest - the oldest offset available in a topic (ie. the oldest message)
Kafka honors the log retention policy by deleting log segments which satisfy the predicate and it will not never remove the active log segment which means the base offset of the active log segment will always be kept.
As for your example, if the first batch of 100 messages are all in a same log segment, Kafka will create a new log segment with the base offset of 100 and delete the old segment after a log-retention-period amount of time passed. So a new message starts offset at 100.
166
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
