Menu
I'm considering using Apache Kafka as an event store for storing events within a microservice.
One thing that I read through various blogs is that Kafka can be considered to be a single source of truth, where Kafka log will store all the events for a given topic.
I was wondering if Kafka has the ability to replay messages since the beginning of time (in case there is a hard drive/network crash that occurs for example)?
(note that i see that there are some logs stored in the /tmp folder under a topic directory). Does anyone know of any command (if any) that can be invoked to replay the messages in the topic?
737
Create a new consumer group (with a new, unique group ID) of consumers. Bring up this consumer group and attach it to the topic that contains the messages to be replayed. The consumers in the consumer group will start consuming messages from the earliest offsets in the partitions, effective replaying the messages.
You can use the Kafka-console-consumer to view your messages. It provides a command line utility, bin/kafka-console-consumer.sh, that sends messages from a topic to an output file. To display a maximum number of messages by using: --from-beginning and --max-messages ${NUM_MESSAGES}.
Retries happen within the consumer poll for the batch. Consumer poll must complete before poll timeout, containing all retries, and total processing time (including REST calls & DB calls), retry delay and backoff, for all records in the batch. Default poll time is 5 minutes for 500 records in the batch.
Initially, Kafka only supported at-most-once and at-least-once message delivery. However, the introduction of Transactions between Kafka brokers and client applications ensures exactly-once delivery in Kafka.
Yes, you can seek to a specific offset, but
beginning of time
depends on the topic or broker configuration. IIRC, the default retention is 7 days.
Refer to the the Kafka documentation.
53
Yes, You can replay message. As Consumer have a control over resetting the offset. You can start reading messages from the beginning or if you know any existing offset value you can read it from there as well. Once the message is committed it will be in there in topic until its retention period is over. Default retention period is 7 days, however you can change it any time.
36
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
