Menu
A Kafka Consumer by default periodically commits the current offsets unless it is turned off by disabling enable.auto.commit. According to the documentation you're then responsible for committing the offsets yourself. So when I want manual control, that seems to be the way to go, however the documentation also mentions the stored offsets and that if you want manual control you should disable enable.auto.offset.store and use rd_kafka_offsets_store() and leave the auto-commit untouched.
Can somebody explain why the latter approach is to be preferred? Disabling auto-commits should have the exact same effect?
268
Using auto-commit gives you “at least once” delivery: Kafka guarantees that no messages will be missed, but duplicates are possible. Auto-commit basically works as a cron with a period set through the auto.commit.interval.ms configuration property.
Apache Kafka Offset Commit activity notifies Kafka Consumer Trigger to commit given offset. This is useful in case you want offsets to be committed as soon as the record is processed in the flow. By default, offsets are committed only when flow is successfully executed.
Auto commit is enabled out of the box and by default commits every five seconds. For a simple data transformation service, “processed” means, simply, that a message has come in and been transformed and then produced back to Kafka.
By default, as the consumer reads messages from Kafka, it will periodically commit its current offset (defined as the offset of the next message to be read) for the partitions it is reading from back to Kafka.
Kafka - Auto Committing Offsets. For a consumer, we can enable/disable auto commit by setting enable.auto.commit = true/false. When set to true consumer's offset will be periodically committed in the background. The property auto.commit.interval.ms specifies the frequency in milliseconds that the consumer offsets are auto-committed to Kafka.
As an alternative to auto commit, offsets can also be managed manually. For this, auto commit should be disabled ( enable.auto.commit = false ). For manual committing KafkaConsumers offers two methods, namely commitSync () and commitAsync ().
If we do not need the duplicate copy data on the consumer front, then Kafka offset plays an important role. On the other hand, the committed offset means that the consumer has confirmed the processing position. Here, the processing term may vary from the Kafka architecture or project requirement.
For a consumer, we can enable/disable auto commit by setting enable.auto.commit = true/false When set to true consumer's offset will be periodically committed in the background. If this property is set to false then no offsets are committed
With enable.auto.commit=true librdkafka will commit the last stored offset for each partition at regular intervals, at rebalance, and at consumer shutdown.
The offsets that are used here, are taken from an in-memory offset store. This store will be updated automatically when enable.auto.offset.store=true.
If you set enable.auto.offset.store=false you can update this in-memory offset store by yourself via rd_kafka_offsets_store().
This is preferred over disabling enable.auto.commit because you do not have to reimplement calling commit at regular intervals yourself, but can rely on the already built-in logic instead.
You have manual control about whether or not offsets are committed either way, but disabling enable.auto.commit and calling commit yourself will most likely lead to more frequent commits.
70
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
