How to fetch more than 10000 records from Elasticsearch 7.X.X

Question

I need to fetch more than 10000 records from Elasticsearch but I'm unable to set the index.max_result_window in Elasticsearch 7.2 from python.

I had used the following command to set the window limit to 100000 in Elasticsearch V6 which was working.

es.indices.create(index=prod_index, body={"settings": {"index.mapping.total_fields.limit": 50000, "index.max_result_window" : 100000})

The same command is not working in Elasticsearch 7.2

Answer 1

It's better not to and that's the reason why they set it 10 000 as a max number. Increasing index.max-result-window is not very good idea which can lead to cluster latency or crashes. When you set a size, ES creates a heap of the same size before fetching the data. Those records will stay in RAM and unless you have great hardware and huge heap space, it's probably the best idea not to do it, because it can crash your cluster or slow it down.

Alternatives are to use scroll API ,From-size or Search-after (probably most preferable -

description)

You can check this solution. It helped me to fetch more then 700k documents without bringing down cluster. Also, you can check this answer.