Kafka Streams - Processor context commit

Question

should we ever invoke processorContext.commit() in Processor implementation by ourselves? I mean invoking commit method inside scheduled Punctuator implementation or inside process method.

in which use cases should we do that, and do we need that at all? the question relates to both Kafka DSL with transform() and Processor API.

seems Kafka Streams handles it by itself, also invoking processorContext.commit() does not guarantee that it will be done immediately.

Answer 1

It is ok to call commit() -- either from the Processor or from a Punctuation -- that's why this API is offered.

While Kafka Streams commits on a regular (configurable) interval, you can request intermediate commits when you use it. One example use case would be, that you usually do cheap computation, but sometimes you do something expensive and want to commit asap after this operation instead of waiting for the next commit interval (to reduce the likelihood of a failure after the expensive operation and the next commit interval). Another use case would be, if you set the commit interval to MAX_VALUE what effectively "disables" regular commits and to decide when to commit base on your business logic.

I guess, that calling commit() is not necessary for most use cases thought.