Menu
I am new to kafka. I want to understand how kafka consumer behaves when offset specified by it is not present. Maybe the message at the given offset is deleted due to retention policy or consumer specified an invalid offset value.
855
Kafka maintains a numerical offset for each record in a partition. This offset acts as a unique identifier of a record within that partition, and also denotes the position of the consumer in the partition.
The answer is it depends on the offset retention period. The default retention period for message offsets in Kafka is one week (7 days). If Kafka was configured using the default, then to answer the questions above, the offset would begin at 32.
If the consumer crashes or is shut down, its partitions will be re-assigned to another member, which will begin consumption from the last committed offset of each partition. If the consumer crashes before any offset has been committed, then the consumer which takes over its partitions will use the reset policy.
By default, as the consumer reads messages from Kafka, it will periodically commit its current offset (defined as the offset of the next message to be read) for the partitions it is reading from back to Kafka.
Your consumer will use your auto.offset.reset setting to reset its offset to a valid position. If no reset option has been specified, then the Broker will throw an InvalidOffsetException.
126
It depends on the auto.offset.reset's mode that you have configured. Please find below about what will happen with each mode when you seek to an offset that doesn't exist in the given topic's partition.
auto.offset.reset=earliest
auto.offset.reset=latest
auto.offset.reset=none
4
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
