Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Terminate Kafka Console Consumer when all the messages have been read

I know there has to be a way to do this, but I am not able to figure this out. I need to stop the kafka consumer once I have read all the messages from the queue.

Can somebody provide any info on this?

like image 349
divinedragon Avatar asked Dec 18 '14 10:12

divinedragon


People also ask

How do I exit Kafka console consumer?

Produce a message to a Kafka topic Repeat the same step to publish more messages. The message published to Kafka has a null key. Press Ctrl+C to exit.

How do I know if Kafka reads my message?

You can use ConsumerGroupCommand to check if certain consumer group has finished processing all messages in a particular topic: Zero lag for every partition will indicate that the messages have been consumed successfully, and offsets committed by the consumer.

How do I stop consuming messages from Kafka topic?

Stopping a Kafka Consumer We can use rest api to stop a running consumer. However, we need consumer id to stop the running consumer, so the consumer id needs to be sent. Then try to access the POST http://localhost:8080/api/kafka/registry/deactivate by sending the id parameter of the consumer you want to stop.

How do you cancel Kafka?

Stop the Kafka broker with Ctrl-C .


2 Answers

Currently, Kafka version 2.11-2.1.1 has a script called kafka-console-consumer.sh.

It has a new flag: --timeout-ms.

Basically, this flag is the maximum time to wait before exiting when there is no new log to wait. It's in millisecond term.

You can use this property to end you console consumer after reading all the messages.

like image 147
Planck35 Avatar answered Nov 08 '22 10:11

Planck35


You can pass parameter: -consumer-timeout-ms with a value when starting the consumer and it will throw an exception if no messages have been read during that time. For example, to stop the consumer if no new messages have arrived in the last 2 seconds: kafka.consumer.ConsoleConsumer -consumer-timeout-ms 2000

You can see this and all the other input options here

like image 22
Lundahl Avatar answered Nov 08 '22 12:11

Lundahl