Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Use kafka-avro-console-producer with a schema already in the schema registry

I would like to use the kafka-avro-console-producer with the schema registry. I have big schemas (over 10k chars) and I can't really past them as a command line argument. Besides that I'd like to use the schema registry directly so I can use a specific schema id.

I'm thinking about something like this, but it doesn't work:

kafka-avro-console-producer \
 --broker-list <broker-list> \
 --topic <topic>  \
 --property schema.registry.url=http://localhost:8081 \
 --property value.schema=`curl http://localhost:8081/schemas/ids/419`
like image 889
0x26res Avatar asked Jan 03 '20 16:01

0x26res


People also ask

Does Avro require schema registry?

With the Kafka Avro Serializer, the schema is registered if needed and then it serializes the data and schema ID. The Kafka Avro Serializer keeps a cache of registered schemas from the Schema Registry their schema IDs.


2 Answers

For the current version of the CLI tool

kafka-avro-console-producer \
 --broker-list <broker-list> \
 --topic <topic>  \
 --property schema.registry.url=http://localhost:8081 \
 --property value.schema.id=419

For older version

You'll need to extract the schema from the API request using jq, for example

value.schema="$(curl http://localhost:8081/schemas/ids/419 | jq -r .schema)"
like image 111
OneCricketeer Avatar answered Sep 20 '22 00:09

OneCricketeer


You can use the property value.schema.id:

kafka-avro-console-producer \
 --broker-list <broker-list> \
 --topic <topic>  \
 --property schema.registry.url=http://localhost:8081 \
 --property value.schema.id=419
like image 33
mathiasfk Avatar answered Sep 20 '22 00:09

mathiasfk