How do you set Kafka Rest Proxy Key Schema with JSON values?

Question

I'm using Kafka 1.1 and Kafka Rest Proxy 4.1.2. I have been keying records from inside Kafka streams using String keys. I want to use Rest Proxy to insert records to be joined, but the keys are getting escaped quotation marks put around them.

I'm sending a POST request to /topics/{someTopic} with Content-Type: application/vnd.kafka.json.v2+json which causes the issue.

With Content-Type: application/vnd.kafka.avro.v2+json and key_schema type: string, the keys do not have extra quotation marks around them, but I'd rather send json values.

This is what I am sending to the /topics endpoint.

{
    "records": [
        { 
            "key": "abc", 
            "value": {"animal": "dog"} 
        }
    ]
}

When I stream the data in Kafka streams, the key is coming out as \"abc\", and obviously isn't being joined with records with string keys abc.

Is there a way to specify a key schema with json values, so that my keys don't get escaped quotation marks around them?

Answer 1

When using the Content-Type: application/vnd.kafka.json.v2+json header, a JSON key will have escaped quotes around all strings so that it properly deserializes in a streams app. When using a simple key, strings do seem to be escape quoted while numeric keys are unmodified.

The Content-Type: application/vnd.kafka.binary.v2+json will produce your key value pairs exactly as you give them, without adding escaped quotes to string keys. You just need to base64 encode your keys and values.

Your example body becomes:

{
    "records": [{
        "key": "YWJj",
        "value": "eyJhbmltYWwiOiJkb2cifQ=="
    }]
}