Kafka enable.auto.commit values for each consumer

Question

I just got in Kafka and I was wondering if we should have enable.auto.commit to true or false. Is there anything which drive us to a specific solution or it's something which is upon us. Is there any option to have default as true and then if for a consumer is needed for us to commit it after processing/transforimg to set it to false. I found out that we could use 2 different ConcurrentKafkaListenerContainerFactory , but is it the only solution?

Answer 1

It depends on how strongly you want to guarantee data is processed.

If you set it to true, and catch and ignore some exception in the middle of processing, you could lose up to hundreds of offsets, by default, per poll. You'd need to seek the consumer backwards to catch any missed data, but then re-process any successful events as well.

If you set it to false, and forget to commit, then you could re-process the same data again and again. If you commit after every message, then you'll slow down the consumer rather significantly.

You cannot change the value at runtime without closing the consumer thread(s) and re-initializing with new configs.