Elasticsearch crashes after showing t: failed to read local state , exiting

Question

I am new to Elasticsearch . I am running only one node locally . Up until now it was working fine . Now as soon as it starts it is crashing . The error stack is as follows .

C:\MyApps\elasticsearch6.5.1\elasticsearch-oss-6.5.1\elasticsearch-6.5.1\bin>elasticsearch.bat [2018-12-10T14:20:49,956][INFO ][o.e.e.NodeEnvironment ] [J7qB0pK] using [1] data paths, mounts [[OS (C:)]], net usable_space [352.1gb], net total_space [475.8gb], types [NTFS] [2018-12-10T14:20:49,972][INFO ][o.e.e.NodeEnvironment ] [J7qB0pK] heap size [990.7mb], compressed ordinary object pointers [true] [2018-12-10T14:21:05,971][INFO ][o.e.n.Node ] [J7qB0pK] node name derived from node ID [J7qB0pK5R7S9adX1LCN2xA]; set [node.name] to override [2018-12-10T14:21:05,971][INFO ][o.e.n.Node ] [J7qB0pK] version[6.5.1], pid[16884], build[oss/zip/8c58350/2018-11-16T02:22:42.182257Z], OS[Windows 10/10.0/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_171/25.171-b11] [2018-12-10T14:21:05,971][INFO ][o.e.n.Node ] [J7qB0pK] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=C:\Users\ADITYA~1\AppData\Local\Temp\elasticsearch, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:logs/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -Delasticsearch, -Des.path.home=C:\MyApps\elasticsearch6.5.1\elasticsearch-oss-6.5.1\elasticsearch-6.5.1, -Des.path.conf=C:\MyApps\elasticsearch6.5.1\elasticsearch-oss-6.5.1\elasticsearch-6.5.1\config, -Des.distribution.flavor=oss, -Des.distribution.type=zip] [2018-12-10T14:21:07,596][INFO ][o.e.p.PluginsService ] [J7qB0pK] loaded module [aggs-matrix-stats] [2018-12-10T14:21:07,596][INFO ][o.e.p.PluginsService ] [J7qB0pK] loaded module [analysis-common] [2018-12-10T14:21:07,596][INFO ][o.e.p.PluginsService ] [J7qB0pK] loaded module [ingest-common] [2018-12-10T14:21:07,596][INFO ][o.e.p.PluginsService ] [J7qB0pK] loaded module [lang-expression] [2018-12-10T14:21:07,596][INFO ][o.e.p.PluginsService ] [J7qB0pK] loaded module [lang-mustache] [2018-12-10T14:21:07,596][INFO ][o.e.p.PluginsService ] [J7qB0pK] loaded module [lang-painless] [2018-12-10T14:21:07,596][INFO ][o.e.p.PluginsService ] [J7qB0pK] loaded module [mapper-extras] [2018-12-10T14:21:07,596][INFO ][o.e.p.PluginsService ] [J7qB0pK] loaded module [parent-join] [2018-12-10T14:21:07,614][INFO ][o.e.p.PluginsService ] [J7qB0pK] loaded module [percolator] [2018-12-10T14:21:07,619][INFO ][o.e.p.PluginsService ] [J7qB0pK] loaded module [rank-eval] [2018-12-10T14:21:07,622][INFO ][o.e.p.PluginsService ] [J7qB0pK] loaded module [reindex] [2018-12-10T14:21:07,627][INFO ][o.e.p.PluginsService ] [J7qB0pK] loaded module [repository-url] [2018-12-10T14:21:07,631][INFO ][o.e.p.PluginsService ] [J7qB0pK] loaded module [transport-netty4] [2018-12-10T14:21:07,633][INFO ][o.e.p.PluginsService ] [J7qB0pK] loaded module [tribe] [2018-12-10T14:21:07,637][INFO ][o.e.p.PluginsService ] [J7qB0pK] no plugins loaded [2018-12-10T14:21:14,718][ERROR][o.e.g.GatewayMetaState ] [J7qB0pK] failed to read local state, exiting... org.elasticsearch.ElasticsearchException: java.io.IOException: failed to read [id:153, file:C:\MyApps\elasticsearch6.5.1\elasticsearch-oss-6.5.1\elasticsearch-6.5.1\data\nodes\0\indices\0gZa7OvBS4GaYgFeY2sKaQ\_state\state-153.st] at org.elasticsearch.ExceptionsHelper.maybeThrowRuntimeAndSuppress(ExceptionsHelper.java:164) ~[elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:304) ~[elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.gateway.MetaStateService.loadFullState(MetaStateService.java:64) ~[elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.gateway.GatewayMetaState.<init>(GatewayMetaState.java:86) [elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.node.Node.<init>(Node.java:499) [elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.node.Node.<init>(Node.java:265) [elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) [elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) [elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) [elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) [elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:127) [elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-6.5.1.jar:6.5.1] at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-6.5.1.jar:6.5.1] at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) [elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:86) [elasticsearch-6.5.1.jar:6.5.1] Caused by: java.io.IOException: failed to read [id:153, file:C:\MyApps\elasticsearch6.5.1\elasticsearch-oss-6.5.1\elasticsearch-6.5.1\data\nodes\0\indices\0gZa7OvBS4GaYgFeY2sKaQ\_state\state-153.st] at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:298) ~[elasticsearch-6.5.1.jar:6.5.1] ... 14 more Caused by: org.elasticsearch.gateway.CorruptStateException: org.apache.lucene.index.CorruptIndexException: codec footer mismatch (file truncated?): actual footer=459461742 vs expected footer=-1071082520 (resource=BufferedChecksumIndexInput(SimpleFSIndexInput(path="C:\MyApps\elasticsearch6.5.1\elasticsearch-oss-6.5.1\elasticsearch-6.5.1\data\nodes\0\indices\0gZa7OvBS4GaYgFeY2sKaQ\_state\state-153.st"))) at org.elasticsearch.gateway.MetaDataStateFormat.read(MetaDataStateFormat.java:201) ~[elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:294) ~[elasticsearch-6.5.1.jar:6.5.1] ... 14 more Caused by: org.apache.lucene.index.CorruptIndexException: codec footer mismatch (file truncated?): actual footer=459461742 vs expected footer=-1071082520 (resource=BufferedChecksumIndexInput(SimpleFSIndexInput(path="C:\MyApps\elasticsearch6.5.1\elasticsearch-oss-6.5.1\elasticsearch-6.5.1\data\nodes\0\indices\0gZa7OvBS4GaYgFeY2sKaQ\_state\state-153.st"))) at org.apache.lucene.codecs.CodecUtil.validateFooter(CodecUtil.java:502) ~[lucene-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi - 2018-09-18 13:01:13] at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:414) ~[lucene-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi - 2018-09-18 13:01:13] at org.apache.lucene.codecs.CodecUtil.checksumEntireFile(CodecUtil.java:526) ~[lucene-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi - 2018-09-18 13:01:13] at org.elasticsearch.gateway.MetaDataStateFormat.read(MetaDataStateFormat.java:184) ~[elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:294) ~[elasticsearch-6.5.1.jar:6.5.1] ... 14 more [2018-12-10T14:21:14,734][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [J7qB0pK] uncaught exception in thread [main] org.elasticsearch.bootstrap.StartupException: ElasticsearchException[java.io.IOException: failed to read [id:153, file:C:\MyApps\elasticsearch6.5.1\elasticsearch-oss-6.5.1\elasticsearch-6.5.1\data\nodes\0\indices\0gZa7OvBS4GaYgFeY2sKaQ\_state\state-153.st]]; nested: IOException[failed to read [id:153, file:C:\MyApps\elasticsearch6.5.1\elasticsearch-oss-6.5.1\elasticsearch-6.5.1\data\nodes\0\indices\0gZa7OvBS4GaYgFeY2sKaQ\_state\state-153.st]]; nested: CorruptStateException[org.apache.lucene.index.CorruptIndexException: codec footer mismatch (file truncated?): actual footer=459461742 vs expected footer=-1071082520 (resource=BufferedChecksumIndexInput(SimpleFSIndexInput(path="C:\MyApps\elasticsearch6.5.1\elasticsearch-oss-6.5.1\elasticsearch-6.5.1\data\nodes\0\indices\0gZa7OvBS4GaYgFeY2sKaQ\_state\state-153.st")))]; nested: CorruptIndexException[codec footer mismatch (file truncated?): actual footer=459461742 vs expected footer=-1071082520 (resource=BufferedChecksumIndexInput(SimpleFSIndexInput(path="C:\MyApps\elasticsearch6.5.1\elasticsearch-oss-6.5.1\elasticsearch-6.5.1\data\nodes\0\indices\0gZa7OvBS4GaYgFeY2sKaQ\_state\state-153.st")))]; at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:140) ~[elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:127) ~[elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.5.1.jar:6.5.1] at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.5.1.jar:6.5.1] at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:86) ~[elasticsearch-6.5.1.jar:6.5.1] Caused by: org.elasticsearch.ElasticsearchException: java.io.IOException: failed to read [id:153, file:C:\MyApps\elasticsearch6.5.1\elasticsearch-oss-6.5.1\elasticsearch-6.5.1\data\nodes\0\indices\0gZa7OvBS4GaYgFeY2sKaQ\_state\state-153.st] at org.elasticsearch.ExceptionsHelper.maybeThrowRuntimeAndSuppress(ExceptionsHelper.java:164) ~[elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:304) ~[elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.gateway.MetaStateService.loadFullState(MetaStateService.java:64) ~[elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.gateway.GatewayMetaState.<init>(GatewayMetaState.java:86) ~[elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.node.Node.<init>(Node.java:499) ~[elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.node.Node.<init>(Node.java:265) ~[elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.5.1.jar:6.5.1] ... 6 more Caused by: java.io.IOException: failed to read [id:153, file:C:\MyApps\elasticsearch6.5.1\elasticsearch-oss-6.5.1\elasticsearch-6.5.1\data\nodes\0\indices\0gZa7OvBS4GaYgFeY2sKaQ\_state\state-153.st] at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:298) ~[elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.gateway.MetaStateService.loadFullState(MetaStateService.java:64) ~[elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.gateway.GatewayMetaState.<init>(GatewayMetaState.java:86) ~[elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.node.Node.<init>(Node.java:499) ~[elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.node.Node.<init>(Node.java:265) ~[elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.5.1.jar:6.5.1] ... 6 more Caused by: org.elasticsearch.gateway.CorruptStateException: org.apache.lucene.index.CorruptIndexException: codec footer mismatch (file truncated?): actual footer=459461742 vs expected footer=-1071082520 (resource=BufferedChecksumIndexInput(SimpleFSIndexInput(path="C:\MyApps\elasticsearch6.5.1\elasticsearch-oss-6.5.1\elasticsearch-6.5.1\data\nodes\0\indices\0gZa7OvBS4GaYgFeY2sKaQ\_state\state-153.st"))) at org.elasticsearch.gateway.MetaDataStateFormat.read(MetaDataStateFormat.java:201) ~[elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:294) ~[elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.gateway.MetaStateService.loadFullState(MetaStateService.java:64) ~[elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.gateway.GatewayMetaState.<init>(GatewayMetaState.java:86) ~[elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.node.Node.<init>(Node.java:499) ~[elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.node.Node.<init>(Node.java:265) ~[elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.5.1.jar:6.5.1] ... 6 more Caused by: org.apache.lucene.index.CorruptIndexException: codec footer mismatch (file truncated?): actual footer=459461742 vs expected footer=-1071082520 (resource=BufferedChecksumIndexInput(SimpleFSIndexInput(path="C:\MyApps\elasticsearch6.5.1\elasticsearch-oss-6.5.1\elasticsearch-6.5.1\data\nodes\0\indices\0gZa7OvBS4GaYgFeY2sKaQ\_state\state-153.st"))) at org.apache.lucene.codecs.CodecUtil.validateFooter(CodecUtil.java:502) ~[lucene-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi - 2018-09-18 13:01:13] at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:414) ~[lucene-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi - 2018-09-18 13:01:13] at org.apache.lucene.codecs.CodecUtil.checksumEntireFile(CodecUtil.java:526) ~[lucene-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi - 2018-09-18 13:01:13] at org.elasticsearch.gateway.MetaDataStateFormat.read(MetaDataStateFormat.java:184) ~[elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:294) ~[elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.gateway.MetaStateService.loadFullState(MetaStateService.java:64) ~[elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.gateway.GatewayMetaState.<init>(GatewayMetaState.java:86) ~[elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.node.Node.<init>(Node.java:499) ~[elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.node.Node.<init>(Node.java:265) ~[elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.5.1.jar:6.5.1] ... 6 more

I had same issue with Elasticsearch 6.3 . I downloaded 6.5.1 and it was working fine for sometime . But again this is happening . After sometime I have to host the Elasticsearch. Please help me to resolve the issue.

Answer 1

I faced almost the same situation and I tried to move the corrupted index to /tmp/ directory and rebooted the machine. After the reboot(where I have already set up elasticsearch to run automatically after reboot) all things good.

So on your case, here is the solution: 1) Move this file/entire index directory to some other location C:\MyApps\elasticsearch6.5.1\elasticsearch-oss-6.5.1\elasticsearch-6.5.1\data\nodes\0\indices\0gZa7OvBS4GaYgFeY2sKaQ_state\state-153.st

2) Then reboot the machine

3) Restart the elasticsearch process (if it's not added in crontab or scheduler)

4) That's it!

Good luck