have a grok filter create nested fields as a result

Question

I have a drupal watchdog syslog file that I want to parse into essentially two nested fields, the syslog part and the message part so that I get this result

syslogpart: {
  timestamp: "",
  host: "",
  ...
},
messagepart:{
  parsedfield1: "",
  parsedfield2: "",
  ...
}

I tried making a custom pattern that looks like this:

DRUPALSYSLOG (%{SYSLOGTIMESTAMP:date} %{SYSLOGHOST:logsource} %{WORD:program}: %{URL:domain}\|%{EPOCH:epoch}\|%{WORD:instigator}\|%{IP:ip}\|%{URL:referrer}\|%{URL:request}\|(?<user_id>\d+)\|\|)

and then run match => ['message', '%{DRUPALSYSLOG:drupal}'}

but I don't get a nested response, I get a textblock drupal: "ALL THE MATCHING FIELDS IN ONE STRING" and then all the matches separately as well but not nested under drupal but rather on the same level.

Answer 1

Yes, this is expected. I don't think there's a way to produce nested fields with grok. I suspect you'll have to use the mutate filter to move them into place.

mutate {
    rename => {
      "date" => "[drupal][date]"
      "instigator" => "[drupal][instigator]"
      ...
    }
  }

If you have a lot of fields it might be more convenient to use a ruby filter. This is especially true if you prefix Drupal fields with e.g. "drupal." – then you'd write a filter to move all fields with that prefix into a subfield with the same name.