End of search results using search_after parameter from Elastic Search API

Question

For a given date range in the query and with a search_after parameter I am able to successfully extract the relevant results. How do I figure out if I am at the end of the search results for the given date range and I dont have to continue querying with the search_after parameter.

Answer 1

There is a pretty cool "trick" that does not involve any additional queries or knowledge of the total number of results:

Say you have a page size of 20. Instead of asking elasticsearch for 20 results, ask it for 21.

• If you got 21 results back, only use the first 20 of them. But you now know that the next query will have at least one more result (If you use the sort values of the 20th result for the search_after parameter, not the 21st!).

• If you get 20 results or fewer, there will be no additional results.

This github issue gives some more details into why elasticsearch does not have this feature out of the box: https://github.com/elastic/elasticsearch/issues/22364

Answer 2

You can either keep querying until it starts returning zero results, or it does return the total, so you could keep a track of how many you've already retrieved and stop searching once you've met the total. (I do a combination of both)