Menu
Basically I am looking for a equivalent of 1.x storage.local.target-heap-size.
I have a prometheus app(running in a 6 core, 32G box) that pulls about host metrics from 2000 VMs. Scrape interval 5mins.
Within few mins my RAM usage increases up to 29-30GBs and later in 10-15mins, it dies with OOM or server is Unavailable from UI.
Any way we can tell prometheus to use a defined max RAM ?
735
To reduce memory consumption, Prometheus can be configured to store fewer time series, by scraping fewer metrics or by attaching fewer labels to the time series. To see which series use the most memory, you can check the TSDB (time series database) status page in the Prometheus UI.
This fits chunks that are still being filled, plus the ones that are already filled and waiting to be written. You can expect RSS RAM usage to be at least 2.6kiB per local memory chunk. The chunks themselves are 1024 bytes, there is 30% of overhead within Prometheus, and then 100% on top of that to allow for Go's GC.
Prometheus can handle millions of time series. However, with the above mentioned default setting for storage. local. target-heap-size , you will be limited to about 200,000 time series simultaneously present in memory.
As of Prometheus 2.20 a good rule of thumb should be around 3kB per series in the head. This allows not only for the various data structures the series itself appears in, but also for samples from a reasonable scrape interval, and remote write.
Prometheus 2.x diesn't provide configs for limiting memory usage - it uses the memory needed for processing the current workload. There is a calculator for estimating the required memory for Prometheus 2.x depending on workload.
P.S. If you are struggling with Prometheus OOMs under your workload, then take a look at VictoriaMetrics. It requires by up to 7x less memory than Prometheus on the same workload. See https://valyala.medium.com/prometheus-vs-victoriametrics-benchmark-on-node-exporter-metrics-4ca29c75590f . There is also vmagent, which can be used for efficient scraping big number of Prometheus targets.
127
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
