Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Elasticsearch multi search not returning correct number of results

I'm using Elasticsearch's multi search to run a list of queries through the python library. It seems to be working... mostly. The problem is that if I send up 200 queries, it returns 100 results (no errors). I've tried running even 20 queries and still get 10 results. For some reason it seems to always return only half of the queries. Which also means I don't know which query the results correlate to.

The queries all work individually but would take too long. There's no mention in the docs of the possibility of the response not containing the same number of queries so I'm not sure where to go here. I've stepped into the code as much as the python library will let me to verify all of the queries are being sent up and it seems to work as expected.

Thanks for any help or a push in the right direction!

Edit: Here's an example of the query I use.

   {  
   "filter":{  
      "and":[  
         {  
            "not":{  
               "term":{  
                  "uuid":"60507f9e-01c1-11e5-a369-34363bd16ec4"
               }
            }
         },
         {  
            "term":{  
               "brand_id":22212
            }
         }
      ]
   },
   "query":{  
      "bool":{  
         "minimum_should_match":1,
         "should":[  
            {  
               "match":{  
                  "name":{  
                     "query":"spork",
                     "boost":3
                  }
               }
            },
            {  
               "match":{  
                  "categories":{  
                     "slop":10,
                     "type":"phrase",
                     "query":"household plates, bowls, cups & flatware"
                  }
               }
            }
         ],
         "must_not":[  

         ],
         "must":[  
            {  
               "match":{  
                  "name_analyzed":{  
                     "boost":4,
                     "query":"spork",
                     "type":"phrase",
                     "slop":15
                  }
               }
            }
         ]
      }
   }
}

I create a list of however many queries like these I'd like to run and execute them as so, as per the python elasticsearch documentation:

elasticsearch.msearch(list_of_search_queries)
like image 533
kevin.w.johnson Avatar asked Aug 12 '15 23:08

kevin.w.johnson


People also ask

Why is Elasticsearch not returning all results?

The reason might be that you haven't provided the size parameter in the query. This limits the result count to 10 by default. Out of all the results the top 10 might be from the two index even thought the match is present in third index as well.

What is Elasticsearch DSL?

Elasticsearch DSL is a high-level library whose aim is to help with writing and running queries against Elasticsearch. It is built on top of the official low-level client ( elasticsearch-py ). It provides a more convenient and idiomatic way to write and manipulate queries.


1 Answers

Whew! That was annoying.

So the caveat between search and msearch is that elasticsearch expects two lines per query. The first line indicates the index (can be left blank if you've already specified it) and the second line is your actual query.

So when creating my list of queries, I just append {} right before it as so:

list_of_queries = []
for thing in list_of_things:
    list_of_queries.append({})
    list_of_queries.append(thing.get_query())

Then it works!

like image 160
kevin.w.johnson Avatar answered Sep 19 '22 16:09

kevin.w.johnson