conditional matching with grok for logstash

Question

I have php log of this format

[Day Mon DD HH:MM:SS YYYY] [Log-Type] [client <ipv4 ip address>] <some php error type>: <other msg with /path/of/a/php/script/file.php and something else>
[Day Mon DD HH:MM:SS YYYY] [Log-Type] [client <ipv4 ip address>] <some php error type>: <other msg without any file name in it>
[Day Mon DD HH:MM:SS YYYY] [Log-Type] [client <ipv4 ip address>] <some msg with out semicolon in it but /path/of/a/file inside the message>

This I am trying to send to Graylog2 after processing through logstash. Using this post here, I was able to start. now I would like to get some additional fields, so that my final version would look something like this.

{
       "message" => "<The entire error message goes here>",
      "@version" => "1",
    "@timestamp" => "converted timestamp from Day Mon DD HH:MM:SS YYYY",
          "host" => "<ipv4 ip address>",
       "logtime" => "Day Mon DD HH:MM:SS YYYY",
      "loglevel" => "Log-Type",
      "clientip" => "<ipv4 ip address>",
      "php_error_type" => "<some php error type>"
      "file_name_from_the_log" => "/path/of/a/file || /path/of/a/php/script/file.php"
      "errormsg" => "<the error message after first colon (:) found>"
}

I have the expression for individual line, or atleast I think these should be able to parse, using grokdebugger. something like this:

%{DATA:php_error_type}: %{DATA:message_part1}%{URIPATHPARAM:file_name}%{GREEDYDATA:errormsg}
%{DATA:php_error_type}: %{GREEDYDATA:errormsg}
%{DATA:message_part1}%{URIPATHPARAM:file_name}%{GREEDYDATA:errormsg}

But somehow I am finding it very difficult to make it work for the entire log file.

Any suggestion please? Also, not sure if there would be any other type of error messages coming in the log file. but the intention is to get the same format for all. Any suggestions how to tackle these logs to get the above mentioned format?

Answer 1

The grok filter can be configured with multiple patterns:

grok {
  match => [
    "message", "%{DATA:php_error_type}: %{DATA:message_part1}%{URIPATHPARAM:file_name}%{GREEDYDATA:errormsg}",
    "message", "%{DATA:php_error_type}: %{GREEDYDATA:errormsg}",
    "message", "%{DATA:message_part1}%{URIPATHPARAM:file_name}%{GREEDYDATA:errormsg}"
  ]
}

(Instead of a single filter with multiple patterns you could have multiple grok filters, but then you'd probably want to disable the _grokparsefailure tagging with tag_on_failure => [].)

Answer 2

If you have some part of your log line missing sometime you can use the following syntax :

(?:%{PATTERN1}|%{PATTERN2})

or

(?:%{PATTERN1}|)

To allow PATTERN1 OR ''. (empty)

Using this, you can have have only one pattern to manage :

grok {
   match => [
      "message", "(?:%{DATA:php_error_type}: |)(?:%{DATA:message_part1}:)(?:%{URIPATHPARAM:file_name}|)%{GREEDYDATA:errormsg}",
   ]
}

If you have problems, maybe replace %{DATA} by a more restrictive pattern.

You can also use this syntax (more regex like)

(?:%{PATTERN1})?

To debug a complex grok pattern, I recommend :

• https://grokconstructor.appspot.com/do/match (multiline option + multiple input lines at same time + others options)
• https://grokdebug.herokuapp.com/ (simpler to use)