Different Elasticsearch results for the same query

Question

I've setup Elasticsearch with 1 cluster á 4 nodes. Number of shards per index: 1; Number of replicas per index: 3

When I call a simple query like the following one multiple times I get different results (different total hits and different top 10 documents):

http://localhost:9200/index_name/_search?q=term

Different data on each shard? I like to have all shards up to date. What can I do?

This is the result of /_cluster/health:

{
  "cluster_name" : "secret",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 4,
  "number_of_data_nodes" : 4,
  "active_primary_shards" : 24,
  "active_shards" : 96,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0
}

As a temporary solution I rebuild the index through Ruby gem tire: ModelName.rebuild_index

But I need a long-term solution.

Answer 1

We ran into a similar problem and it turned out to be because Elasticsearch round-robins between different shards when searching. Each shard returns a slightly different _score because of slightly different indexing due to the way ES handles deleted documents in an index. In our case this meant similar results often placed slightly lower or higher in the results order, and, when combined with pagination (using from and size in the search query) it meant the same results were turning up on two separate "pages" or not at all from page to page.

We found an Elasticsearch article on consistent scoring which explains this quite neatly and implemented a preference parameter to ensure that we always get the same scores for a particular search by querying the same shards:

http://localhost:9200/index_name/_search?q=term&preference=blablabla

We also thought about using sorting, but Elasticsearch sorts results with the same scores by an internal Lucene document ID, ensuring that results with the same scores are always returned in the same order.