Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Where to set maximum message size in Apache Kafka?

In which file I can increase the maximum message size for Kafka? (I'm sending records of String, byte[] and when I send byte[] of 770kb Kafka does not send the message)

like image 928
neb Avatar asked May 07 '18 05:05

neb


People also ask

How do I change the Max message size in Kafka?

Kafka Broker Configuration An optional configuration property, “message. max. bytes“, can be used to allow all topics on a Broker to accept messages of greater than 1MB in size. And this holds the value of the largest record batch size allowed by Kafka after compression (if compression is enabled).

How does Kafka calculate message size?

To know the amount of bytes received by a topic, you can measure this metric on the server side: kafka. server:type=BrokerTopicMetrics,name=BytesInPerSec or checking outgoing-byte-rate metric on the producer side.

Which is larger than 1048576 which is the value of the Max request size configuration?

RecordTooLargeException: The message is 1740572 bytes when serialized which is larger than 1048576, which is the value of the max. request. size configuration.

What is Max poll in Kafka?

Kafka consumer has a configuration max. poll. records which controls the maximum number of records returned in a single call to poll() and its default value is 500.


1 Answers

In kafka 0.11.0 following four config options need to be set

Broker config options (details) :

  1. message.max.bytes - The largest record batch size allowed by Kafka.

  2. replica.fetch.max.bytes - The number of bytes of messages to attempt to fetch for each partition.

Producer config options (details) :

  1. max.request.size - The maximum size of a request in bytes. It is advisable to match this value with (message.max.bytes).

Consumer config options (details) :

  1. max.partition.fetch.bytes - max number of bytes per partition returned by the server. should be larger than the max.message.size so consumer can read the largest message sent by the broker.
like image 82
Swapnil Avatar answered Sep 28 '22 08:09

Swapnil