Menu
The JVM Memory Pressure of my AWS Elasticsearch cluster has been increasing consistently. The pattern I see for the last 3 days is that it adds 1.1% every 1 hour. This is for one of the 3 master nodes I have provisioned.
All other metrics seem to be in the normal range. The CPU is under 10% and there are barely any indexing or search operations being performed.
I have tried clearing the cache for fielddata for all indices as mentioned in this document but that has not helped.
Can anyone help me understand what might be the reason for this?
804
You can lower the JVM memory pressure by reducing the number of shards or upgrading to a larger cluster.
The Elasticsearch process is very memory intensive. Elasticsearch uses a JVM (Java Virtual Machine), and close to 50% of the memory available on a node should be allocated to JVM. The JVM machine uses memory because the Lucene process needs to know where to look for index values on disk.
This is because the JVM steadily increases heap usage percentage until the garbage collection process frees up memory again. High heap usage occurs when the garbage collection process cannot keep up. An indicator of high heap usage is when the garbage collection is incapable of reducing the heap usage to around 30%.
Anything below 75% - the green range of the indicator - is good and there is no need to upgrade unless you expect an increase in requests.
Got this answer from AWS Support
I checked the particular metric and can also see the JVM increasing from the last few days. However, I do not think this is an issue as JVM is expected to increase over time. Also, the garbage collection in ES runs once the JVM reaches 75% (currently its around 69%), after which you would see a drop in the JVM metric of your cluster. If JVM is being continuously > 75 % and not coming down after GC's is a problem and should be investigated.
The other thing which you mentioned about clearing the cache for fielddata for all indices was not helping in reducing JVM, that is because the dedicated master nodes do not hold any indices data and their related caches. Clearing caches should help in reducing JVM on the data nodes.
166
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
