Kafka Streams: RocksDB TTL

Question

I understand that the default TTL is set to infinity (non-positive). However, if we need to retain data in the store for max of 2 days, can we do the override with the RocksDBConfigSetter interface implementation, that is options.setWalTtlSeconds(172800)? OR would it conflict with the Kafka streams internals?

Ref: https://docs.confluent.io/current/streams/developer-guide/config-streams.html#streams-developer-guide-rocksdb-config

Answer 1

This is currently not possible. Kafka Streams disables RocksDB's TTL feature in a hard-coded way for various technical reasons. There is also a ticket for this: https://issues.apache.org/jira/browse/KAFKA-4212

For now, you could use a windowed store to expire old record after 2 days. Ie, you do a stream.groupByKey().windowedBy(...).reduce(...) with a TimeWindow of 1ms and a "dummy" reduce that just return the latest value for a key.