Kafka long polling

Question

I'm looking at kafka to implement a low-latency message queue, and I've been reading up about consumer long polling. However, there are no examples about how to actually use long polling, or what options you need to set to enable it. How do you enable long polling using the kafka java api?

Answer 1

You don't have to enable it per say since it is the default behavior of the kafka consumer. What you need to set in your configuration is fetch.wait.max.ms.

Two values are important to achieve what you want:

• fetch.min.bytes: The broker will wait for this amount of data to fill BEFORE it sends the response to the consumer client.
• fetch.max.wait.ms: The broker will wait for this amount of time BEFORE sending a response to the consumer client, unless it has enough data to fill the response (fetch.message.min.bytes)

A response will be sent to the consumer once either of these values is reached. The longest possible fetch request time will always be fetch.max.wait.ms.

Look for more configuration options in Kafka Consumer Configs

The kafka user mailing list is a also good option for questions like this.