Menu
In which file I can increase the maximum message size for Kafka? (I'm sending records of String, byte[] and when I send byte[] of 770kb Kafka does not send the message)
928
asked May 07 '18 05:05
Kafka Broker Configuration An optional configuration property, “message. max. bytes“, can be used to allow all topics on a Broker to accept messages of greater than 1MB in size. And this holds the value of the largest record batch size allowed by Kafka after compression (if compression is enabled).
To know the amount of bytes received by a topic, you can measure this metric on the server side: kafka. server:type=BrokerTopicMetrics,name=BytesInPerSec or checking outgoing-byte-rate metric on the producer side.
RecordTooLargeException: The message is 1740572 bytes when serialized which is larger than 1048576, which is the value of the max. request. size configuration.
Kafka consumer has a configuration max. poll. records which controls the maximum number of records returned in a single call to poll() and its default value is 500.
In kafka 0.11.0 following four config options need to be set
Broker config options (details) :
message.max.bytes - The largest record batch size allowed by Kafka.
replica.fetch.max.bytes - The number of bytes of messages to attempt to fetch for each partition.
Producer config options (details) :
Consumer config options (details) :
82
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
