max.poll.records
has been recently changed to 500 in kafka consumer config, but I am wondering how does this affect the consumer poll. Is it just an upper bound of the maximum number of records that could be fetched or the consumer waits till it gets 500 records.
Kafka consumer has a configuration max. poll. records which controls the maximum number of records returned in a single call to poll() and its default value is 500.
So the max. poll. records control the number of messages read at one poll. This allows us to tune the consumption based on the number of messages to be processed without timing out.
Increasing the number of partitions and the number of brokers in a cluster will lead to increased parallelism of message consumption, which in turn improves the throughput of a Kafka cluster; however, the time required to replicate data across replica sets will also increase.
The Kafka consumer poll() method fetches records in sequential order from a specified topic/partitions. This poll() method is how Kafka clients read data from Kafka. When the poll() method is called, the consumer will fetch records from the last consumed offset.
max.poll.records : Yes from new consumer this property is changed to 500 by default which means consumer can poll minimum 1 to max 500 records for each poll, and which means consumer will not wait when partition did not have sufficient data
fetch.min.bytes : By default it is 1 byte, consumer will wait if you increase this configuration.
The minimum amount of data the server should return for a fetch request. If insufficient data is available the request will wait for that much data to accumulate before answering the request. The default setting of 1 byte means that fetch requests are answered as soon as a single byte of data is available or the fetch request times out waiting for data to arrive. Setting this to something greater than 1 will cause the server to wait for larger amounts of data to accumulate which can improve server throughput a bit at the cost of some additional latency.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With