How to access nested property at script_fields on elastic 5.x

Question

I need to iterate over nested objects on painless script_fields for my query but notation doc['nestedProperty.property'] does not gave me the value, neither does using array notation doc['nestedProperty.property'][0]

Any idea on how to use this?

EDIT

A document example:

{
    "neestedProperty": [
      {
        "property": 12,
        "innerNeestedProperty": {
          "innerProperty1": 45,
          "innerProperty2": -45
        }
      }
    ]
}

example query:

{
  "query": {
    match_all: {}
  },
  "script_fields": {
    "scripted": {
      "script": {
        "inline": "doc['neestedProperty.property'] * params.multiplier",
        "params": {
          "multiplier": 100
        },
        "lang": "painless"
      }
    }
  }
}

Answer 1

The doc-notation apparently doesn't work on nested objects, but you could directly access the _source-object as Horst Seirer pointed out.

Complication is that accessing _source depends on the context where your script is executed; the ctx-variable is not always available. For scripted_fields, you can instead use params._source. But that won't be available on other contexts (e.g. in the query-part)

After you have the _source, accessing elements can be done with get, or using dot-notation, and the nested fields will be an array. So for example params._source.nestedProperty[0].property will get the value from the first nested object.

For scripted fields, you should return one object, but that can be an array. So in your example, I'd use something like this:

def returnval=[];
for (nested in params._source.nestedProperty) {
  returnval.add(nested['property']*params.multiplier)
}
return returnval;

Even though you're calling the source as a parameter, you don't have to add it in your parameter list.

Edit

The above is useful if you need to script from the point of view of your parent document (maybe you need to combine several nested documents).

There is however, a disadvantage in that you can only use the raw source, which means non-analyzed strings, and dates that are just strings, etc.

So often it may be a lot easier to just use the scripted fields on your nested documents directly, by including script_fields in an inner_hits section. Like this:

{
  "query": {
    "nested": {
      "path": "nestedProperty",
      "query": {
        "match_all": {}
      },
      "inner_hits": {
        "_source": true, 
        "script_fields": {
          "my_value": {
            "script": {
              "source": "doc['nestedProperty.property'].value*params.multiplier",
              "params": {
                "multiplier": 100
              }
            }
          }
        }
      }
    }
  }
}

What this code does is just looking at each of the nested documents individually (as they are internally stored as seperate documents), and run a script on them. And these (internally stored seperate) documents can use the doc-notation.

The result will give you your original documents, along with a section inner_hits that contains each of the nested documents with their source, and a scripted field my_value.

It may only feel weird that you are using the nested document as a seperate entry, yet still have to use the full path (nestedProperty.property)

And you have to mind the fact that this query as it is now only returns documents that have nested documents in them, whereas the code before would return the document with as scripted field an empty array. However, if you want all documents you can use a bool-query with a match_all-clause.

Finally, I don't know if this works on elastic 5 (as was the original question), but I've confirmed on 7.3, and according to the docs it should also work on e.g. 5.0.

And I know I'm late, but maybe I can help others looking for the same answer.

Answer 2

ACTUALLY, I found the solution on discuss.elastic.co here...

GET my_index/_search
{
  "script_fields": {
    "new_scripted_field_name": {
      "script": {
        "source": "doc['nestedProperty.property'].value"
      }
    }
  }
}

Contrary to @Emil Bode's answer, you can use doc, just that you must use dot notation rather than bracket notation.

I am using Elasticsearch v7 and the discuss.elastic.co post uses Elasticsearch v6.