Automatically reconnect failed tasks in Kafka-Connect

Question

I'm using a mongo-source plugin with Kafka-connect. I checked the source task state, and it was running and listening on a mongo collection.

I manually stopped mongod service and waited about 1 minute, then I start it back again.

I checked the source task to see if anything will fix itself, and after 30 minutes nothing seems to work.

Only after restarting the connector it started working again.

Since, mongo-source doesn't have the options to set retries + backoff when timeout, I searched for a configuration that will fit a simple scenario: restart failed task after X time using Kafka-connect configuration. couldn't find any.. :/ I can do that with a simple script, but there must be something in Kafka-connect that manages failed tasks. or even in mongo-source... I don't want it to fail so fast after just 1 minute... :/

Answer 1

There isn't any way other than using the REST API to find a failed task and submit a restart request - and then running this on a periodic basis. For example

curl -s "http://localhost:8083/connectors?expand=status" | \
  jq -c -M 'map({name: .status.name } +  {tasks: .status.tasks}) | .[] | {task: ((.tasks[]) + {name: .name})}  | select(.task.state=="FAILED") | {name: .task.name, task_id: .task.id|tostring} | ("/connectors/"+ .name + "/tasks/" + .task_id + "/restart")' | \
  xargs -I{connector_and_task} curl -v -X POST "http://localhost:8083"\{connector_and_task\}

Source: https://rmoff.net/2019/06/06/automatically-restarting-failed-kafka-connect-tasks/