Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

kafka 0.11 reset offset for consumer group by --to-datetime

I want to reset the offset of kafka consumer group by timestamp. But when I am using following command:

./kafka-consumer-groups.sh --bootstrap-server localhost:9092 --reset-offsets --to-datetime 2017-11-1907:52:43:00:000 --group <group_name> --topic <topic_name> --execute

I am getting the following error message:

Note: This will only show information about consumers that use the Java consumer API (non-ZooKeeper-based consumers).

how to reset offset according to time

like image 415
Dhruv Bansal Avatar asked Nov 20 '17 12:11

Dhruv Bansal


People also ask

How do you change or reset consumer offset in Kafka?

How to change consumer offset? Use the kafka-consumer-groups.sh to change or reset the offset. You would have to specify the topic, consumer group and use the –reset-offsets flag to change the offset.

How consumer remember Kafka offset?

consumers in a consumer group load balance record processing. consumers remember offset where they left off reading. consumers groups each have their own offset per partition.

What is auto offset reset in Kafka?

The auto offset reset consumer configuration defines how a consumer should behave when consuming from a topic partition when there is no initial offset. This is most typically of interest when a new consumer group has been defined and is listening to a topic for the first time.

What is Kafka consumer group offset?

Kafka Consumer Offsets As we know, each message in a Kafka topic has a partition ID and an offset ID attached to it. Therefore, in order to "checkpoint" how far a consumer has been reading into a topic partition, the consumer will regularly commit the latest processed message, also known as consumer offset.


2 Answers

Invoking

bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092
--group test-group --reset-offsets --all-topics --to-datetime 2017-08-04T00:00:00.000

can resets offsets to the earliest ones after the given datetime. Datetime format is yyyy-MM-ddTHH:mm:ss.xxx, 2017-08-04T00:00:00.000 for instance.

You could also reset offsets by duration. See an example below:

bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 
--group test-group --reset-offsets --all-topics --by-duration PT0H30M0S

--by-duration resets offsets to offset by duration from current timestamp. Format: 'PnDTnHnMnS'.

like image 179
amethystic Avatar answered Sep 19 '22 08:09

amethystic


It's not an error, but just a warning - because you specified --bootstrap-server option then changes will affect only consumers that are implemented using new Java API. If you have consumers that are built using other APIs, then you need to specify --zookeeper option instead.

like image 29
Alex Ott Avatar answered Sep 22 '22 08:09

Alex Ott