Apache-Kafka, batch.size vs buffer.memory

Question

I'm trying to figure out the difference between the settings batch.size and buffer.memory in Kafka Producer.

As I understand batch.size: It's the max size of the batch that can be sent.

The documentation describes buffer.memory as: the bytes of memory the Producer can use to buffer records waiting to be sent.

I don't understand the difference between these two. Can someone explain?

Thanks

Answer 1

In my opinion,

batch.size: The maximum amount of data that can be sent in a single request. If batch.size is (32*1024) that means 32 KB can be sent out in a single request.

buffer.memory: if Kafka Producer is not able to send messages(batches) to Kafka broker (Say broker is down). It starts accumulating the message batches in the buffer memory (default 32 MB). Once the buffer is full, It will wait for "max.block.ms" (default 60,000ms) so that buffer can be cleared out. Then it's throw exception.

Answer 2

Both of these producer configurations are described on the Confluent documentation page as following:

• batch.size

Kafka producers attempt to collect sent messages into batches to improve throughput. With the Java client, you can use batch.size to control the maximum size in bytes of each message batch.

• buffer.memory

Use buffer.memory to limit the total memory that is available to the Java client for collecting unsent messages. When this limit is hit, the producer will block on additional sends for as long as max.block.ms before raising an exception.