Apache Kafka persist all data

Question

When using Kafka as an event store, how is it possible to configure the logs never to lose data (v0.10.0.0) ?

I have seen the (old?) log.retention.hours, and I have been considering playing with compaction keys, but is there simply an option for kafka never to delete messages ?

Or is the best option to put a ridiculously high value for the retention period ?

Answer 1

You don't have a better option that using a ridiculously high value for the retention period.

Fair warning : Using an infinite retention will probably hurt you a bit.

For example, default behaviour only allows a new suscriber to start from start or end of a topic, which will be at least annoying in an event sourcing perspective.

Also, Kafka, if used at scale (let's say tens of thousands of messages per second), benefits greatly for high performance storage, the cost of which will be ridiculously high with an eternal retention policy.

FYI, Kafka provides tools (Kafka Connect e.g) to easily persist data on cheap data stores.