Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Debezium, Kafka connect: is there a way to send only payload and not schema?

I have an outbox postgresql table and debezium connector in kafka connect that creates kafka messages based on the added rows to the table.

The problem I am facing is with the message format. This is the created message value:

{
  "schema": {
    "type": "string",
    "optional": true,
    "name": "io.debezium.data.Json",
    "version": 1
  },
  "payload": "{\"foo\": \"bar\"}"
}

But (because of consumer) I need the message to contain only the payload, like this:

{
  "\"foo\": \"bar\""
}

This is part of my kafka connector configuration:

"transforms": "outbox",
"transforms.outbox.type": "io.debezium.transforms.outbox.EventRouter",
"transforms.outbox.route.topic.replacement": "${routedByValue}",
"transforms.outbox.route.by.field": "aggregate_type",
"transforms.outbox.table.field.event.payload.id": "aggregate_id",
"transforms.outbox.table.fields.additional.placement": "payload_type:header:__TypeId__"

Is there any way to achieve this without creating custom transformer?

like image 647
ThreeDots Avatar asked Oct 31 '25 09:10

ThreeDots


1 Answers

It looks like you're using org.apache.kafka.connect.json.JsonConverter with schemas.enable=true for your value converter. When you do this it embeds the schema alongside the payload in the message.

If you set value.converter.schemas.enable=false you should get just the payload in your message.

Ref: Kafka Connect: Converters and Serialization Explained — JSON and Schemas

like image 87
Robin Moffatt Avatar answered Nov 03 '25 00:11

Robin Moffatt



Donate For Us

If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!