Field [] used in expression does not exist in mappings

Question

The feature I try to fullfit is to create a metric in kibana that display the number of users "unvalidated". I send a log sent when a user registers, then a log when a user is validated.

So the count I want is the difference between the number of registered and the number of validated.

In kibana I cannot do such a math operation, so I found a workaround: I added a "scripted field" named "unvalidated" which is equal to 1 when a user registers and -1 when a user validates his account. The sum of the "unvalidated" field should be the number of unvalidated users.

This is the script I defined in my scripted field: doc['ctxt_code'].value == 1 ? 1 : doc['ctxt_code'].value == 2 ? -1 : 0

with:

• ctxt_code 1 as the register log

• ctxt_code 2 as the validated log

This setup works well when all my logs have a "ctxt_code", but when a log without this field is pushed kibana throws the following error:

Field [ctxt_code] used in expression does not exist in mappings

I can't understand this error because kibana says:

If a field is sparse (only some documents contain a value), documents missing the field will have a value of 0

which is the case.

Anyone has a clue ?

Answer 1

It's OK to have logs without the ctxt_code field... but you have to have a mapping for this field in your indices. I see you're querying multiple indices with logstash-*, so you are probably hitting one that does not have it.

You can include a mapping for your field in all indices. Just go into Sense and use this:

PUT logstash-*/_mappings/[your_mapping_name]
{
  "properties": {
    "ctxt_code": {
      "type": "short",           // or any other numeric type, including dates
      "index": "not_analyzed"    // Only works for non-analyzed fields.
    }
  }
}

If you prefer you can do it from the command line: CURL -XPUT 'http://[elastic_server]/logstash-*/_mappings/[your_mapping_name]' -d '{ ... same JSON ... }'