Lucene: Why the search cost increase with the page increase?

Question

I am using ElasticSearch cluster with 1G cache for each machine. I configured the searcher that 5000/perpage.

When I search page 3, it cost about 400ms. But when I search page 300, the cost increased to more than 5000ms! About 60% cache was free at this situation.

Why the cost increased more than 10 times?

Answer 1

In order to page that far, Elasticsearch has to retrieve all previous pages, just to discard them at the end. A search with from: 1000, size: 100 is equivalent to a search with from: 0, size: 100100, only that you just get the last 100 results. That also means, that every document has to be scored, which is a potentially expensive operation.

There was a recent optimization, that should improve performance in such cases, when you use a scroll search, see this github issue.