Figure out Indexing error in elasticsearch?

Question

I am using ES 1.x version and having trouble to find the errors while indexing some document.

Some documents are not getting indexed and all I saw is below lines in ES logs.

stop throttling indexing: numMergesInFlight=2, maxNumMerges=3

now throttling indexing: numMergesInFlight=4, maxNumMerges=3

I did a quick google and understood the high level of these errors but would like to understand below:

1. Will ES retry the documents which were throttled?
2. Is there is any way to know the documents which were throttled by enabling some detailed logging and if yes, then in which classes?
3. I don't see any error message, apart from above INFO logs. Is there is a way to enable verbose logging for indexing which shows what exactly is going on during indexing?

Answer 1

The throttling messages you see in the logs are not the issue. throttling is happening in the background in order for elasticsearch to protect against segments explosion. see explanation here: https://www.elastic.co/guide/en/elasticsearch/guide/current/indexing-performance.html#segments-and-merging

The throttling does not drop messages, but just slows down indexing, which cause a back pressure to the indexers and external queues.

When indexing fail you should get an error response for the index/bulk request. in order to tell what is the issue you should inspect the responses ES provide for the index/bulk requests. Logs might not tell the full story, as it depends on the log level configuration, which is per module in ES.

Another option is that you are able to index, but the logs don't have the timestamp you think it does. check _cat/indices in order to see if the docs count increases when you index. if the doc counts increases it means that the indexed docs are there, and you need to refine your searches.