Elasticsearch number of results changes with pagination

Question

I'm using Elasticsearch 7.6.0 and have paginated one of my queries. It seems to work well, and I can vary the number of results per page and the selected page using the search from and size parameters.

    query = 'sample query'
    items_per_page = 12
    page = 0

    es_query = {'query': {
        'bool': {
            'must': [{
                'multi_match': {
                    'query': query,
                    "fuzziness": "AUTO",
                    "operator": "and",
                    'fields': ['title^2', 'description']
                },
            }]
        }
    }, 'min_score': 5.0}

    res = es.search(index='my-index', body=es_query, size=items_per_page, from_=items_per_page*page)
    hits = sorted(res['hits']['hits'], key=lambda x: x['_score'], reverse=True)

    print(res['hits']['total']['value']) # This changes depending on the page provided

I've noticed that the number of results returned depends on the page provided, which makes no sense to me! The number of results also oscillates which further confuses me: Page 0, 233 items. Page 1, 157 items. Page 2, 157 items. Page 3, 233 items...

Why does res['hits']['total']['value'] depend on the size and from parameters?

Answer 1

The search is distributed and being sent to all the nodes holding shards matching the searched indices. Then all the results will be merged and returned. Sometimes, not all shards can be searched. This happens when

• The cluster is very busy
• The specific shard is not available due to recovery process
• The search has been optimized and the shard has been omitted.

In the response, there is a _shards section like this:

{
    "took": 1,
    "timed_out": false,
    "_shards":{
        "total" : 1,
        "successful" : 1,
        "skipped" : 0,
        "failed" : 0
    },
    "hits":{...}
}

Check if there is any value other than 0 for failed shards. If so, check the logs and cluster and index status.