Kafka Streams: What are the negative consequences of having a slow punctuate job?

Question

In our Kafka Streams topology, we have some punctuates, that may take a long time to run (several minutes).

What are the consequences of having such slow punctuates? Will the process in which they run stop processing records while punctuate runs? Can they cause re-balances to happen?

Answer 1

if your punctuator logic runs slowly like several minutes, Kafka stream will not consume new messages from specific partition (on which punctuator is executed) until punctuator completes execution. also long running punctuator shouldn't trigger rebalancing (rebalancing might occur due to other reasons)

whether punctuator on one partition affects others depends on whether you run application on single node or on multiple, and also on num.stream.threads property that you might specified for Kafka Streams. if your app is running on a single node, Kafka stream will use only single thread for all partitions (by default, num.stream.threads is 1). so in this case punctuator on specific partition blocks all other partitions as well. but in case you have multiple app instances and/or num.stream.threads greater than 1, if some of punctuators run quickly, then such partitions will not be affected from long running punctuators on other partitions.