Menu
I'm trying to find out why my kafka-streams application runs out of memory. I already found out that rocksDB is consuming lots of native memory and I tried to restrict it with the following configuration:
# put index and filter blocks in blockCache to avoid letting them grow unbounded (https://github.com/facebook/rocksdb/wiki/Block-Cache#caching-index-and-filter-blocks)
cache_index_and_filter_blocks = true;
# avoid evicting L0 cache of filter and index blocks to reduce performance impact of putting them in the blockCache (https://github.com/facebook/rocksdb/wiki/Block-Cache#caching-index-and-filter-blocks)
pinL0FilterAndIndexBlocksInCache=true
# blockCacheSize should be 1/3 of total memory available (https://github.com/facebook/rocksdb/wiki/Setup-Options-and-Basic-Tuning#block-cache-size)
blockCacheSize=1350 * 1024 * 1024
# use larger blockSize to reduce index block size (https://github.com/facebook/rocksdb/wiki/RocksDB-Tuning-Guide#difference-of-spinning-disk)
blockSize=256 * 1024
but still the memory usage seems to grow unbounded and my container eventually gets OOMKilled.
I used jemalloc to profile the memory usage (like described here) and the result clearly shows that rocksDB is responsible but I have no clue how to further restrict the memory usage of rocksDB.
I don't know if it is helpful, but for completeness here are statistics gathered from a running rocksdb instance:
I'm glad for any hints
402
RocksDB Architecture memtable: an in-memory data-structure to ensure very low latency on reads. sorted static table files (SST files): where data is finally persisted.
The non-volatile cache is officially referred to in RocksDB as the SecondaryCache. By maintaining a SecondaryCache that's an order of magnitude larger than DRAM, fewer reads would be required from remote storage, thus reducing read latency as well as network bandwidth consumption.
Block cache is where RocksDB caches data in memory for reads. User can pass in a Cache object to a RocksDB instance with a desired capacity (size). A Cache object can be shared by multiple RocksDB instances in the same process, allowing users to control the overall cache capacity.
Are you seeing the memory usage grow quickly or over a longer period of time?
We have found and fixed a few RocksDB resource leaks that would cause memory leaks:
There are some indications that there may be others (https://issues.apache.org/jira/browse/KAFKA-8367), either in our usage of RocksDB or in RocksDB itself.
Oh, one other idea is that if you're using iterators from the state stores, either in your processors or in Interactive Query, you have to close them.
Beyond looking for leaks, I'm afraid I don't have too much insight into diagnosing RocksDB's memory usage. You could also restrict the Memtable size, but I don't think we set it very large by default anyway.
Hope this helps,
-John
139
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
