Menu
I am writing log messages into a Kafka Topic and I want the retention of this topic to be permanent. I have seen in Kafka and Kafka Connect (_schemas, connect-configs, connect-status, connect-offsets, etc) that there are special topics that are not deleted by the log retention time. How do I enforce a topic to be like these other special topics? Is it the naming convention or some other properties?
Thanks
329
Method 1) If your Kafka allow plaintext connections (via port 9092) you can do it getting into a kafka pod. Method 2) You can do it getting into zookeeper pod. You can still use deprecated script kafka-topics.sh to change retention period for a topic.
Just like relational OLTP (online transactional processing) platforms, Kafka can now act as permanent database stores for transactional data.
log.retention.hours The most common configuration for how long Kafka will retain messages is by time. The default is specified in the configuration file using the log. retention. hours parameter, and it is set to 168 hours, the equivalent of one week.
The default value is 168 hours (seven days). This setting controls the period of time after which Kafka will force the log to roll, even if the segment file is not full. This ensures that the retention process is able to delete or compact old data.
If you want to retain all topics forever, you can set both log.retention.hours and log.retention.bytes to -1.
138
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
