How to process multiline log entry with logstash filter?

Question

Background:

I have a custom generated log file that has the following pattern :

[2014-03-02 17:34:20] - 127.0.0.1|ERROR| E:\xampp\htdocs\test.php|123|subject|The error message goes here ; array (   'create' =>    array (     'key1' => 'value1',     'key2' => 'value2',     'key3' => 'value3'   ), ) [2014-03-02 17:34:20] - 127.0.0.1|DEBUG| flush_multi_line 

The second entry [2014-03-02 17:34:20] - 127.0.0.1|DEBUG| flush_multi_line Is a dummy line, just to let logstash know that the multi line event is over, this line is dropped later on.

My config file is the following :

input {   stdin{} }  filter{   multiline{       pattern => "^\["       what => "previous"       negate=> true   }   grok{     match => ['message',"\[.+\] - %{IP:ip}\|%{LOGLEVEL:loglevel}"]   }    if [loglevel] == "DEBUG"{ # the event flush  line     drop{}   }else if [loglevel] == "ERROR"  { # the first line of multievent     grok{       match => ['message',".+\|.+\| %{PATH:file}\|%{NUMBER:line}\|%{WORD:tag}\|%{GREEDYDATA:content}"]      }   }else{ # its a new line (from the multi line event)     mutate{       replace => ["content", "%{content} %{message}"] # Supposing each new line will override the message field     }   }   }  output {   stdout{ debug=>true } } 

The output for content field is : The error message goes here ; array (

Problem:

My problem is that I want to store the rest of the multiline to content field :

The error message goes here ; array (   'create' =>    array (     'key1' => 'value1',     'key2' => 'value2',     'key3' => 'value3'   ), ) 

So i can remove the message field later.

The @message field contains the whole multiline event so I tried the mutate filter, with the replace function on that, but I'm just unable to get it working :( .

I don't understand the Multiline filter's way of working, if someone could shed some light on this, it would be really appreciated.

Thanks,

Abdou.

Answer 1

I went through the source code and found out that :

• The multiline filter will cancel all the events that are considered to be a follow up of a pending event, then append that line to the original message field, meaning any filters that are after the multiline filter won't apply in this case
• The only event that will ever pass the filter, is one that is considered to be a new one ( something that start with [ in my case )

Here is the working code :

input {    stdin{} }    filter{       if "|ERROR|" in [message]{ #if this is the 1st message in many lines message       grok{         match => ['message',"\[.+\] - %{IP:ip}\|%{LOGLEVEL:loglevel}\| %{PATH:file}\|%{NUMBER:line}\|%{WORD:tag}\|%{GREEDYDATA:content}"]       }        mutate {         replace => [ "message", "%{content}" ] #replace the message field with the content field ( so it auto append later in it )         remove_field => ["content"] # we no longer need this field       }     }      multiline{ #Nothing will pass this filter unless it is a new event ( new [2014-03-02 1.... )         pattern => "^\["         what => "previous"         negate=> true     }      if "|DEBUG| flush_multi_line" in [message]{       drop{} # We don't need the dummy line so drop it     } }  output {   stdout{ debug=>true } } 

Cheers,

Abdou