How do I code to store the data permanently in Kafka?

Question

I have read the documentation but still unable to figure out where to insert the command so that my topic stores the logs forever?

Answer 1

There are properties like log.retention.hours, log.retention.bytes which controls for much time these logs will be retained. You can set both of these properties to -1 to retain your logs forever.

Answer 2

As @pulkit-singhal said, you can configure the retention period as a server default in the server.properties - for full details see the reference doc. You can also set it per-topic using kafka-topics.sh.

You also should be aware of the concept of Log Compaction, which is another way of enabling infinite retention and is commonly used. This gives you a way to retain for every given key, the latest value. Log compaction is enabled by setting log.cleanup.policy=compact

Which you use (-1 as the time & size retention policy, or log compaction) depends on your use case.