how to deal with null values in an elastic search field_value_factor

Question

How do I deal with a null value on a property that I want to do a field_value_factor for? I want to weight by popularity but some records have a null value for that property. Do I have to set a minimum value of 10 on that property in that data itself? It seems kind of kludgy that way.

{
  "query": {
      "function_score": { 
        "query":{
          "multi_match" : {
            "query" : "new girl",
            "fields" : [ "title^1.2", "name"] 
          }
        },
        "field_value_factor": {
          "field":"popularity",
          "modifier":"log1p"
        },
        "boost_mode":"multiply"

      }
  }
}

Answer 1

ES's default behavior for null values is to not add the field value at all. However, you can set a default null_value instead, in your mapping. So in your case:

 ...
 "properties" : {
    ...     
    "popularity" : {
       "type" : "integer",
       "null_value" : "0"    # or whatever
       ...
       }
    ...
 }

Reference: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-core-types.html

"When there is a (JSON) null value for the field, use the null_value as the field value. Defaults to not adding the field at all."

I'm surprised ES is not throwing any errors. You should confirm the documents actually do (or don't) have your 'popularity' field. Try using Sense?

If you try to compute on a non-existent field, I'm pretty sure ES would throw a [Missing value for field [x]] exception. This is both from my own experience and from checking the source code that implements field_value_factor:

https://gitlab.devero.com/development/elasticsearchssl/commit/8fbd1bdd489a9d31d3bda249c3ed9715052e196d

Scroll down to: src/main/java/org/elasticsearch/common/lucene/search/function/FieldValueFactorFunction.java

and look at lines 53-87.