Kafka sink connector: No tasks assigned, even after restart

Question

I am using Confluent 3.2 in a set of Docker containers, one of which is running a kafka-connect worker.

For reasons yet unclear to me, two of my four connectors - to be specific, hpgraphsl's MongoDB sink connector - stopped working. I was able to identify the main problem: The connectors did not have any tasks assigned, as could be seen by calling GET /connectors/{my_connector}/status. The other two connectors (of the same type) were not affected and were happily producing output.

I tried three different methods to get my connectors running again via the REST API:

• Pausing and resuming the connectors
• Restarting the connectors
• Deleting and the creating the connector under the same name, using the same config

None of the methods worked. I finally got my connectors working again by:

• Deleting and creating the connector under a different name, say my_connector_v2 instead of my_connector

What is going on here? Why am I not able to restart my existing connector and get it to start an actual task? Is there any stale data on the kafka-connect worker or in some kafka-connect-related topic on the Kafka brokers that needs to be cleaned?

I have filed an issue on the specific connector's github repo, but I feel like this might actually be general bug related to the intrinsics of kafka-connect. Any ideas?

Answer 1

I have faced this issue. If the resources are less for a SinkTask or SourceTask to start, this can happen.

Memory allocated to the worker may be less some time. By default workers are allocated 250MB. Please increase this. Below is an example to allocate 2GB memory for the worker running in distributed mode.

KAFKA_HEAP_OPTS="-Xmx2G" sh $KAFKA_SERVICE_HOME/connect-distributed $KAFKA_CONFIG_HOME/connect-avro-distributed.properties