Result window is too large, from + size must be less than or equal to: [10000] but was [100000]

Question

I got the following Error in elasticSearch:

[Result window is too large, from + size must be less than or equal to: [10000] but was [100000].

See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level parameter.] and i am not getting in which file we have to set

 index.max_result_window = 50000;

Answer 1

You can find here some references to official documentation for deep paging.

If you need to update the maximum result window in your elasticsearch instance, you can edit settings this way

curl -XPUT "http://localhost:9200/my_index/_settings" -d '{ "index" : { "max_result_window" : 500000 } }' -H "Content-Type: application/json"

as already discussed here, but pay attention to deep paging, because it could increase memory usage and degrade performance of elasticsearch.

In order to implement a more efficient search, you should take a look at:

• Scroll API
• Search After API

Answer 2

you can change max result window by setting index.max_result_window = 50000; in elasticsearch.yml file in etc/elasticsearch

Answer 3

Yes, increasing max_result_window can solve the issue but the elastic search doesn't recommend this solution because it could increase memory, and CPU usage and degrade the performance of the elastic search instance.

Why 10,000 is the limit for normal ES search API:

By default, the offset + limit is limited to 10,000. When paginating in this manner, Elasticsearch has to parse the query, build the search context, distribute the query to applicable shards, collate the results, skip past $offset items, then read out $limit items and destroy the search context for each page which means that the deeper we paginate, each page is more expensive than the page before it.

Two recommended solutions can be:

1. scroll API: Can be used to retrieve large numbers of results (or even all results) from a single search request. But if you don't have infinite pagination then this is not for you. Scroll API is recommended for efficient deep scrolling but scroll contexts are costly and it is not recommended to use it for real-time user requests. It reuses search context and position from one request to the next. You should use it when you need to paginate deeply
2. ES search after API: If you are showing a list of items using finite pagination, then this can help you.

But If you want to jump freely to a random page, above both is not for you.

In sort-term, you can solve the issue by just updating the max_result_window value in settings, but in long term, this should not be a solution.

If you want to jump freely to a random page:

You can educate your users that they don't need to click many times on pagination numbers to go to the next page, they can...

• Just add a date range filter to reduce the items and then perform pagination with limited pages.
• Or change the sorting order so that the last page comes on the first page