Elastic Search-Java.lang.OutOfMemoryError

Question

We are getting "Out of Memory Exception" frequently on elastic search instances. 3 machines are in Elastic search cluster with Similar configuration.

Environment: Windows Server 2012 R2 64 bit OS. Ram : 24gb Elastic search : 6.3.2 JVM Heap Size (jvm.options in elastic search): -Xms16g -Xmx16g

We have tried with 2g (Xms2g & Xmx2g) and 8g (Xms8g & Xmx8g) for min and max heap size but ended up with same error.

And as per the following link https://www.elastic.co/guide/en/elasticsearch/guide/current/heap-sizing.html

we are tried with (Xms12g & Xmx12g) for heap size i.e., 50% of 24gb RAM, but still fails.

[2018-09-14T04:14:37,545][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [Machine1] fatal error in thread [elasticsearch[Machine1][refresh][T#2]], exiting java.lang.OutOfMemoryError: Java heap space     at org.apache.lucene.util.fst.BytesStore.writeByte(BytesStore.java:89) ~[lucene-core-7.3.1.jar:7.3.1 ae0705edb59eaa567fe13ed3a222fdadc7153680
- caomanhdat - 2018-05-09 09:27:24]     at org.apache.lucene.util.fst.FST.<init>(FST.java:265) ~[lucene-core-7.3.1.jar:7.3.1 ae0705edb59eaa567fe13ed3a222fdadc7153680
- caomanhdat - 2018-05-09 09:27:24]     at org.apache.lucene.util.fst.Builder.<init>(Builder.java:157) ~[lucene-core-7.3.1.jar:7.3.1 ae0705edb59eaa567fe13ed3a222fdadc7153680
- caomanhdat - 2018-05-09 09:27:24]     at org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$PendingBlock.compileIndex(BlockTreeTermsWriter.java:456) ~[lucene-core-7.3.1.jar:7.3.1 ae0705edb59eaa567fe13ed3a222fdadc7153680
- caomanhdat - 2018-05-09 09:27:24]     at org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.writeBlocks(BlockTreeTermsWriter.java:633) ~[lucene-core-7.3.1.jar:7.3.1 ae0705edb59eaa567fe13ed3a222fdadc7153680
- caomanhdat - 2018-05-09 09:27:24]     at org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.finish(BlockTreeTermsWriter.java:934) ~[lucene-core-7.3.1.jar:7.3.1 ae0705edb59eaa567fe13ed3a222fdadc7153680
- caomanhdat - 2018-05-09 09:27:24]     at org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter.write(BlockTreeTermsWriter.java:346) ~[lucene-core-7.3.1.jar:7.3.1 ae0705edb59eaa567fe13ed3a222fdadc7153680
- caomanhdat - 2018-05-09 09:27:24]     at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.write(PerFieldPostingsFormat.java:140) ~[lucene-core-7.3.1.jar:7.3.1 ae0705edb59eaa567fe13ed3a222fdadc7153680
- caomanhdat - 2018-05-09 09:27:24]     at org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:108) ~[lucene-core-7.3.1.jar:7.3.1 ae0705edb59eaa567fe13ed3a222fdadc7153680
- caomanhdat - 2018-05-09 09:27:24]     at org.apache.lucene.index.DefaultIndexingChain.flush(DefaultIndexingChain.java:162) ~[lucene-core-7.3.1.jar:7.3.1 ae0705edb59eaa567fe13ed3a222fdadc7153680
- caomanhdat - 2018-05-09 09:27:24]     at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:452) ~[lucene-core-7.3.1.jar:7.3.1 ae0705edb59eaa567fe13ed3a222fdadc7153680
- caomanhdat - 2018-05-09 09:27:24]     at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:557) ~[lucene-core-7.3.1.jar:7.3.1 ae0705edb59eaa567fe13ed3a222fdadc7153680
- caomanhdat - 2018-05-09 09:27:24]     at org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:673) ~[lucene-core-7.3.1.jar:7.3.1 ae0705edb59eaa567fe13ed3a222fdadc7153680
- caomanhdat - 2018-05-09 09:27:24]     at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:453) ~[lucene-core-7.3.1.jar:7.3.1 ae0705edb59eaa567fe13ed3a222fdadc7153680
- caomanhdat - 2018-05-09 09:27:24]     at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:293) ~[lucene-core-7.3.1.jar:7.3.1 ae0705edb59eaa567fe13ed3a222fdadc7153680
- caomanhdat - 2018-05-09 09:27:24]     at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:268) ~[lucene-core-7.3.1.jar:7.3.1 ae0705edb59eaa567fe13ed3a222fdadc7153680
- caomanhdat - 2018-05-09 09:27:24]     at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:258) ~[lucene-core-7.3.1.jar:7.3.1 ae0705edb59eaa567fe13ed3a222fdadc7153680
- caomanhdat - 2018-05-09 09:27:24]     at org.apache.lucene.index.FilterDirectoryReader.doOpenIfChanged(FilterDirectoryReader.java:104) ~[lucene-core-7.3.1.jar:7.3.1 ae0705edb59eaa567fe13ed3a222fdadc7153680
- caomanhdat - 2018-05-09 09:27:24]     at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:140) ~[lucene-core-7.3.1.jar:7.3.1 ae0705edb59eaa567fe13ed3a222fdadc7153680
- caomanhdat - 2018-05-09 09:27:24]     at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:156) ~[lucene-core-7.3.1.jar:7.3.1 ae0705edb59eaa567fe13ed3a222fdadc7153680
- caomanhdat - 2018-05-09 09:27:24]     at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:58) ~[lucene-core-7.3.1.jar:7.3.1 ae0705edb59eaa567fe13ed3a222fdadc7153680
- caomanhdat - 2018-05-09 09:27:24]     at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:176) ~[lucene-core-7.3.1.jar:7.3.1 ae0705edb59eaa567fe13ed3a222fdadc7153680
- caomanhdat - 2018-05-09 09:27:24]     at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:253) ~[lucene-core-7.3.1.jar:7.3.1 ae0705edb59eaa567fe13ed3a222fdadc7153680
- caomanhdat - 2018-05-09 09:27:24]     at org.elasticsearch.index.engine.InternalEngine$ExternalSearcherManager.refreshIfNeeded(InternalEngine.java:281) ~[elasticsearch-6.3.2.jar:6.3.2]  at org.elasticsearch.index.engine.InternalEngine$ExternalSearcherManager.refreshIfNeeded(InternalEngine.java:256) ~[elasticsearch-6.3.2.jar:6.3.2]  at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:176) ~[lucene-core-7.3.1.jar:7.3.1 ae0705edb59eaa567fe13ed3a222fdadc7153680
- caomanhdat - 2018-05-09 09:27:24]     at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:253) ~[lucene-core-7.3.1.jar:7.3.1 ae0705edb59eaa567fe13ed3a222fdadc7153680
- caomanhdat - 2018-05-09 09:27:24]     at org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:1356) ~[elasticsearch-6.3.2.jar:6.3.2]     at org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:1335) ~[elasticsearch-6.3.2.jar:6.3.2]     at org.elasticsearch.index.shard.IndexShard.refresh(IndexShard.java:853) ~[elasticsearch-6.3.2.jar:6.3.2]   at org.elasticsearch.index.IndexService.maybeRefreshEngine(IndexService.java:696) ~[elasticsearch-6.3.2.jar:6.3.2]  at org.elasticsearch.index.IndexService.access$400(IndexService.java:97) ~[elasticsearch-6.3.2.jar:6.3.2]

Answer 1

Solved. After updating the jvm.options file with heap sizes (Xms12g & Xmx12g), for windows machine we need to re-install the elastic search service instance to reflect the changes for heap size as per the link https://www.elastic.co/guide/en/elasticsearch/reference/current/heap-size.html