Kafka - Broker: Message size too large

Question

I get Message size too large exception, when I try to send a message which is over 1 Mb size. The error appears in my client application, when I try to produce a message. After a little googling I found out that the settings should be changed in order to increase max message size. Well, I did that in /kafka/config/server.properties file. I added next 2 settings:

message.max.bytes=15728640
replica.fetch.max.bytes=15728640

Also, I added fetch.message.max.bytes=15728640 to the /kafka/config/consumer.properties file. All other settings remain default.

I did kafka server restart, but I'm still getting the same error.

P.S Kafka version is 1.1.0.

Answer 1

You have the right configuration however you need to also set max.request.size on the producer side.

props.put(ProducerConfig.MAX_REQUEST_SIZE_CONFIG, 15728640);

max.request.size The maximum size of a request in bytes. This setting will limit the number of record batches the producer will send in a single request to avoid sending huge requests. This is also effectively a cap on the maximum record batch size.

On the Broker side, you have already configured the below parameter that should work

message.max.bytes The largest record batch size allowed by Kafka.

replica.fetch.max.bytes The number of bytes of messages to attempt to fetch for each partition. This is not an absolute maximum if the first record batch in the first non-empty partition of the fetch is larger than this value, the record batch will still be returned to ensure that progress can be made. The maximum record batch size accepted by the broker is defined via message.max.bytes (broker config) or max.message.bytes (topic config).

On the topic side max.message.bytes which is not required in case you have already set message.max.bytes in the broker side

max.message.bytes - this is the largest size of the message the broker will allow being appended to the topic. This size is validated pre-compression. (Defaults to broker's message.max.bytes.)

Refrence https://kafka.apache.org/documentation/