kafka ktable - rocksdb access via java

Question

Ive been reading about ktables this morning in hopes of implementing a rolling-window key-value store. I can see where the latest release of kafka seems to hint that this is possible but I'm more wondering about accessing the key-value data from 'outside' applications.

Say I implement a kstreams application and it's consuming log data (or it's ilk) from a topic and merrily windowing, aggregating and producing to another topic. Now I want to look at this key-value data from some other process. The docs hint that the data is stored by rocksdb. Can I read this from 'outside' as a call to said db? Or is this data only available as a virtual construct to kstreams applications?

Answer 1

RocksDBStore shows how to access the RocksDB store from Java. Basically:

RocksDB db = RocksDB.open(options, dir.getAbsolutePath());

and then you can get() and put() and remove(). You can configure where KafkaStreams stores the RocksDB state, so that's basically all you need to know to work with the store outside of the KafkaStreams.

With regard to windows—depending on how your windowing is configured, multiple RocksDBStore called Segments are created are each window segment. You can access each of these stores as mentioned above.

Answer 2

Right now, there is no built-in support, but there are plans to expose the internal state (ie, KTable state) and make them queryable. See KIP-67 for more details.

Also, KTable state is written to a Kafka topic for fault-tolerance. Thus, you could also consume this topic and feed the data into an external database.