I am running an 8 cores, 32g RAM elasticsearch node with 5 shards, 400 million (small) documents.
Everything works great until I run an agg search, then shards start failing with:
java.lang.OutOfMemoryError: Java heap space
I have changed heap size with:
export ES_HEAP_SIZE=16g
(also ES_MAX_MEM
and ES_MIN_MEM
to same)
also changed the yml
file for elasticsearch:
bootstrap.mlockall: true
and even (recommended by install documents):
sudo sysctl -w vm.max_map_count=262144
Restart service and still no no impact, still java.lang.OutOfMemoryError: Java heap space
Any other suggestions? other than don't run agg queries?
query is:
https://localhost:9200/my_index_name/_search?search_type=count
{
"aggs": {
"distinct_hostname": {
"cardinality": {
"field": "hostname"
}
}
}
}
I think I have discovered the error. I was using 'service' to run elasticsearch and therefore my environment variables got stripped. I had to update the /etc/default/elasticsearch
file with the correcct env variables (specifically the ES_HEAP_SIZE=16g
).
So far it's running well and app is not erroring.
The correct way to update Java heap size for Elasticsearch 5 is not EXPORT _JAVA_OPTIONS
or EXPORT ES_HEAP_SIZE
or using command line parameters. As far as I can tell, all of these are overridden by a configuration file in your Elasticsearch install directory, config/jvm.options
. To change these settings you need to edit these lines in that file:
# Xms represents the initial size of total heap space
# Xmx represents the maximum size of total heap space
-Xms2g
-Xmx2g
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With