Use Avro in KafkaConnect without Confluent Schema Registry

Question

We have vanilla apache Kafka setup in current infrastructure and we started logging some data that we want to process using Kafka Connect. Currently we use Avro for our message format, but there's no Schema Registry in our infrastructure. In future, we plan to replace current stack with Confluent and use Schema Registry and Connect, but for some time we need to deploy only Connect for that.

Is it possible to configure Connect sinks somehow so they use explicit avsc files or schema without connecting to Schema Registry and without using Confluent format with magic bytes and schema ID?

Answer 1

Yes, it is possible using the registryless-avro-converter on Github.

Follow the build instructions there, add a JAR to your plugin.path folder as other connectors are loaded, then setup like so

key.converter=me.frmr.kafka.connect.RegistrylessAvroConverter
key.converter.schema.path=/path/to/schema/file.avsc
value.converter=me.frmr.kafka.connect.RegistrylessAvroConverter
value.converter.schema.path=/path/to/schema/file.avsc

Note that this will require you to store/maintain/sync the schema files on all Connect workers, however

---

Alternatively, you can setup the Schema Registry with your vanilla Kafka - No reason to do some "Confluent migration" since the registry doesn't require any infrastructure changes other than your Serializer & Deserializer configs.