Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How does max.poll.records affect the consumer poll

max.poll.records has been recently changed to 500 in kafka consumer config, but I am wondering how does this affect the consumer poll. Is it just an upper bound of the maximum number of records that could be fetched or the consumer waits till it gets 500 records.

like image 817
Achilleus Avatar asked Nov 14 '18 21:11

Achilleus


People also ask

What is Max poll in Kafka?

Kafka consumer has a configuration max. poll. records which controls the maximum number of records returned in a single call to poll() and its default value is 500.

What is Max Poll records?

So the max. poll. records control the number of messages read at one poll. This allows us to tune the consumption based on the number of messages to be processed without timing out.

How do you increase consumer throughput in Kafka?

Increasing the number of partitions and the number of brokers in a cluster will lead to increased parallelism of message consumption, which in turn improves the throughput of a Kafka cluster; however, the time required to replicate data across replica sets will also increase.

How do consumer polls work?

The Kafka consumer poll() method fetches records in sequential order from a specified topic/partitions. This poll() method is how Kafka clients read data from Kafka. When the poll() method is called, the consumer will fetch records from the last consumed offset.


1 Answers

max.poll.records : Yes from new consumer this property is changed to 500 by default which means consumer can poll minimum 1 to max 500 records for each poll, and which means consumer will not wait when partition did not have sufficient data

fetch.min.bytes : By default it is 1 byte, consumer will wait if you increase this configuration.

The minimum amount of data the server should return for a fetch request. If insufficient data is available the request will wait for that much data to accumulate before answering the request. The default setting of 1 byte means that fetch requests are answered as soon as a single byte of data is available or the fetch request times out waiting for data to arrive. Setting this to something greater than 1 will cause the server to wait for larger amounts of data to accumulate which can improve server throughput a bit at the cost of some additional latency.

like image 167
Deadpool Avatar answered Nov 02 '22 12:11

Deadpool