KTable/KStream memory consumption over time

Question

Is there a way to calculate how much heap (or any other) memory approximately will be used by KTable/KStream in java/scala application over time?

I have some specific assumptions and I'd like to know whether they are correct:

• Kafka streams use internal topics and RocksDB only.

• RocksDB is embeddable DB, so it uses heap memory of my application.

• KStream constantly deletes all records from RocksDB after they can no longer be used by any of processors in topology (e.g for join with specified JoinWindow) (== not so much memory used)

• KTable is fully stored in RocksDB (== in memory)

• When KTable receive null-value record it deletes record from RocksDB (== memory freed up)

Answer 1

It's hard to estimate. For general sizing, consider this guide: https://docs.confluent.io/current/streams/sizing.html

Kafka streams use internal topics and RocksDB only.

Yes. You can also replace RocksDB with in-memory stores (that are part of Kafka Streams) or implement your own custom stores.

RocksDB is embeddable DB, so it uses heap memory of my application.

RocksDB use off-heap memory and also spills to disk.

KStream constantly deletes all records from RocksDB after they can no longer be used by any of processors in topology (e.g for join with specified JoinWindow) (== not so much memory used)

It depends on the store type. For key-value stores (ie, "regular" KTables) data is not deleted (with the exception of explicit delete messages, so-call tombstones). For time-windowed/session-windowed KTables (result of windowed-aggregations) and joins, there is a retention period after which data is deleted.

KTable is fully stored in RocksDB (== in memory)

RocksDB also spills do disk. It's not in-memory only.

When KTable receive null-key record it deletes record from RocksDB (== memory freed up)

null-key records are not malformed. I assume you mean null-value record, so-called tombstone. Those are treated as deletes.