Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Kafka - What are the better alternatives than poll() to listen to a topic in Java?

I'm trying to create a consumer client in Java. I realized the poll() function is depreciated. What are the alternatives to listen to a topic of Kafka?

My code:

KafkaConsumer< String, UserSegmentPayload > kc = new KafkaConsumer<>(props2);
kc.subscribe(Collections.singletonList(topicName));
while (true) {
    ConsumerRecords<String, UserSegmentPayload> records = kc.poll(100);
    for (ConsumerRecord<String, UserSegmentPayload> record : records) {
        System.out.printf("offset = %d, key = %s, value = %s\n",
        record.offset(), record.key(), record.value());
    }
}
like image 608
Dante Adams Avatar asked Jan 28 '20 07:01

Dante Adams


People also ask

Is Kafka poll blocking?

Kafka-client poll(timeout) is blocking method, so that is not Real Async even if usr reactor-kafka #63.

Does Kafka use polling?

The Kafka consumer poll() method fetches records in sequential order from a specified topic/partitions. This poll() method is how Kafka clients read data from Kafka. When the poll() method is called, the consumer will fetch records from the last consumed offset.

Why does Kafka use polling?

The poll API is designed to ensure consumer liveness. As long as you continue to call poll, the consumer will stay in the group and continue to receive messages from the partitions it was assigned. Underneath the covers, the consumer sends periodic heartbeats to the server.


1 Answers

The reason poll() and poll(long) are deprecated is that they may block indefinitely (even in the second case, where a timeout is specified). The root cause for this behaviour is that an initial metadata update in those methods may block forever (see here). Instead, you should use the poll(Duration)-method of the KafkaConsumer. So, in your code, all you have to do is to replace kc.poll(100) with kc.poll(Duration.ofMillis(100)).

like image 196
KWer Avatar answered Oct 27 '22 13:10

KWer