Kafka Connect - Failed to commit offsets and flush

Question

I had my Kafka Connectors paused and upon restarting them got these errors in my logs

[2020-02-19 19:36:00,219] ERROR WorkerSourceTask{id=wem-postgres-source-0} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter)
************
************
[2020-02-19 19:36:00,216] ERROR WorkerSourceTask{id=wem-postgres-source-0} Failed to flush, timed out while waiting for producer to flush outstanding 2389 messages (org.apache.kafka.connect.runtime.WorkerSourceTask)

I got this error multiple times with the number of outstanding messages changed. Then it stopped and haven't seen it again.

Do I need to take any action here or has Connect retried and committed the offsets and that is why the error has stopped?

Thanks

Answer 1

The error indicates that there are a lot of messages buffered and cannot be flushed before the timeout is reached. To address this issue you can

• either increase offset.flush.timeout.ms configuration parameter in your Kafka Connect Worker Configs
• or you can reduce the amount of data being buffered by decreasing producer.buffer.memory in your Kafka Connect Worker Configs. This turns to be the best option when you have fairly large messages.