logstash check if field exists

Question

I have log files coming in to an ELK stack. I want to copy a field (foo) in order to perform various mutations on it, However the field (foo) isn't always present.

If foo doesn't exist, then bar still gets created, but is assigned the literal string "%{foo}"

How can I perform a mutation only if a field exists?

I'm trying to do something like this.

if ["foo"] {
  mutate {
    add_field => "bar" => "%{foo}
  }
}

Answer 1

To check if field foo exists:

1) For numeric type fields use:

 if ([foo]) {
    ...
 }

2) For types other than numeric like boolean, string use:

if ("" in [foo]) {
    ...
}

Answer 2

"foo" is a literal string.

[foo] is a field.

# technically anything that returns 'true', so good for numbers and basic strings:
if [foo] {
}

# contains a value
if [foo] =~ /.+/ {
}

Answer 3

On Logstash 2.2.2, the ("" in [field]) construct does not appear to work for me.

if ![field] { }

does, for a non-numerical field.

Answer 4

It's 2020 and none of the above answers are quite correct. I've been working with logstash since 2014 and expressions in filter were, are and will be a thing...

For example, you may have a boolean field with false value and with the above solutions you may not know if false is the value of the field or the resulting value of the expression because the field doesn't exists.

Workaround for checking if a field exists in all versions

I think all versions of logstash supports [@metadata] field. That is, a field that will not be visible for output plugins and lives only in the filtering state. So this is what I have to workaround:

filter {

  mutate {
    # we use a "temporal" field with a predefined arbitrary known value that
    # lives only in filtering stage.
    add_field => { "[@metadata][testField_check]" => "unknown arbitrary value" }

    # we copy the field of interest into that temporal field.
    # If the field doesn't exist, copy is not executed.
    copy => { "testField" => "[@metadata][testField_check]" }
  }


  # now we now if testField didn't exists, our field will have 
  # the initial arbitrary value
  if [@metadata][testField_check] == "unknown arbitrary value" {

    # just for debugging purpouses...
    mutate { add_field => { "FIELD_DID_NOT_EXISTED" => true }}

  } else {
    # just for debugging purpouses...
    mutate { add_field => { "FIELD_DID_ALREADY_EXISTED" => true }}
  }
}

Old solution for logstash prior version 7.0.0

Check my issue in github.

I've been struggling a lot with expressions in logstash. My old solution worked until version 7. This was for boolean fields, for instance:

filter {

  # if the field does not exists, `convert` will create it with "false" string. If
  # the field exists, it will be the boolean value converted into string.
  mutate { convert => {  "field" => "string" } }

  # This condition breaks on logstash > 7 (see my bug report). Before version 7,
  # this condition will be true if a boolean field didn't exists.
  if ![field] {
    mutate { add_field => { "field" => false } }
  }
  # at this stage, we are sure field exists, so make it boolean again
  mutate { convert => { "field" => "boolean" } }
}