Which is better scrolls or search_after in elasticsearch to simulate random pagination?

Question

I want to randomly jump to a page of results from elasticsearch. There are three ways to paginate in elasticsearch:

• from/size - I can't use this because of the maximum depth limit of 10000.
• scroll API - I can use this but it has a cost of memory usage (keeping the search context alive) associated with it.
• search_after - I can also use this even it is less expensive than scrolls as it is stateless.

I know that anyway, Elasticsearch will sequentially read the data. Let's say if I wanted to get 99th page then elastic is going to read all 98 results to get the 99th result.

I can do one thing i.e. to reduce the data which I will sequentially get before the targeted data, in this case I will reduce the data returned for 98 pages and for the 99th one I will get the complete data.

My main question is "What if I don't have memory concerns then which approach would be faster to sequentially get 98 pages ?" (search_after or scrolls)

If I use scrolls I will be clearing it after every usage.

Answer 1

If you don't have memory concerns, then the simplest option is to increase the index setting index.max_result_window from 10000 to the number you require.

See https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules.html#dynamic-index-settings