Kafka Connect Confluent S3 Sink Connector: Class io.confluent.connect.avro.AvroConverter could not be found

Question

With this Kafka Connect Connector:

https://www.confluent.io/hub/confluentinc/kafka-connect-s3

I manually install that into /opt/kafka/plugins of my Kafka Connect Docker image. My purpose is to use Kafka Connect to write Avro records from Kafka topics to S3.

At runtime, with Kafka Connect, I get the error:

ERROR Stopping due to error (org.apache.kafka.connect.cli.ConnectDistributed) [main]
org.apache.kafka.common.config.ConfigException: Invalid value io.confluent.connect.avro.AvroConverter for configuration key.converter: Class io.confluent.connect.avro.AvroConverter could not be found.
    at org.apache.kafka.common.config.ConfigDef.parseType(ConfigDef.java:744)
    at org.apache.kafka.common.config.ConfigDef.parseValue(ConfigDef.java:490)
    at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:483)
    at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:108)
    at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:129)
    at org.apache.kafka.connect.runtime.WorkerConfig.<init>(WorkerConfig.java:452)
    at org.apache.kafka.connect.runtime.distributed.DistributedConfig.<init>(DistributedConfig.java:405)
    at org.apache.kafka.connect.cli.ConnectDistributed.startConnect(ConnectDistributed.java:95)
    at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:80)

After poking around for that class in Confluent open source, and looking at other Confluent Kafka Connect plugins, my solution is to copy kafka-connect-avro-converter-7.1.1.jar from the Confluent Kafka Connect Avro Converter plugin here:

https://www.confluent.io/hub/confluentinc/kafka-connect-avro-converter

into the confluentinc-kafka-connect-s3*/lib directory.

This seems to work, but it seems hacky, and it seems like there probably should be a better supported, simpler way of getting this to work. Is there a better way?

Answer 1

Ideally you should be using confluent-hub to add connectors and converters.

https://docs.confluent.io/platform/current/installation/docker/development.html#add-connectors-or-software

Otherwise, yes, you would download and extract the relevant packages from Maven Central or Confluent repositories, and adding to the plugin path.