How to handle backpressure in a Kafka Connect Sink?

Question

We build a custom Kafka Connect sink which in turn calls a remote REST API. How do I propagate backpressure to the Kafka Connect infrastructure, so put() is called less often in cases when the remote system is slower than the internal consumer delivers messages to put()? The Kafka connect documentation says that we should not block in put(), but block in flush(). But not blocking in put() means that we have to buffer data which surely leads to OOM exceptions at some point, if put() is called more often than flush(). I've seen that a kafka consumer is allowed to call pause() or block in the loop(). Is it possible to leverage this in a kafka connect sink?

Answer 1

I've seen that a kafka consumer is allowed to call pause() or block in the loop(). Is it possible to leverage this in a kafka connect sink?

The raw consumer is not exposed, so no. You could call /pause on the whole connector, though I'm not sure what happens to un-flushed messages at that point.

But not blocking in put() means that we have to buffer data which surely leads to OOM exceptions at some point

It can, sure, but that is really the only viable option for holding onto data for longer than necessary. For instance, this is how the S3 and HDFS connectors work.

rotate.interval.ms
The time interval in milliseconds to invoke file commits...

Your HTTP client connection is likely blocking anyway to make the request, is it not?

The alternative would be to make your HTTP server embed a Kafka consumer so it can poll messages itself and act on them locally rather than need to be sent requests over HTTP.