Logstash grok multiline message

Question

My logs are formatted like this:

2014-06-19 02:26:05,556 INFO ok
2014-06-19 02:27:05,556 ERROR
 message:space exception
         at line 85
 solution:increase space
          remove files   

There are 2 types of events:

-log on one line like the first

-log on multiple line like the second

I am able to process the one line event, but I am not able to process the second type, where I would like to stock the message in one variable and the solution in another.

This is my config:

input {
 file {
    path => ["logs/*"]
    start_position => "beginning"
    codec => multiline {
                   pattern => "^%{TIMESTAMP_ISO8601} "
                   negate => true
                   what => previous
    }       
 }
}
filter {
 #parsing of one line event
 grok {
 patterns_dir => "./patterns"
 match=>["message","%{TIMESTAMP_ISO8601:timestamp} %{WORD:level} ok"]
 }
#the parsing fail, so we assumed we are in multiline events, now I process them and I am stuck when I am getting to the new line.
if "_grokparsefailure" in [tags] {
 grok {
 patterns_dir => "./patterns"
 match=>["message","%{TIMESTAMP_ISO8601:timestamp} %{WORD:level}\r\n"]
 }
}

}

So this is what I have done, and I would like to have in my console output the following:

{
"@timestamp" => "2014-06-19 00:00:00,000"
"path" => "logs/test.log"
"level"=>"INFO"
},
{
"@timestamp" => "2014-06-19 00:00:00,000"
"path" => "logs/test.log"
"level"=>"ERROR"
"message" => "space exception at line 85"
"solution"=>"increase space remove files"
}

Concretely, I would like to get all the expression between two words ("message" and "solution" for the message variable, "solution" and the end of event for the solution variable), and that no matter if the expression is on one or multiple lines.

Thanks in advance

Answer 1

As for multiline grok, it's best to use special flag for pattern string:

grok {
    match => ["message", "(?m)%{SYSLOG5424LINE}"]
}

Answer 2

It looks like you have two issues:

You need to correctly combine your multilines:

filter
{
    multiline
   {
        pattern => "^ "
        what => "previous"
   }
}

This will combine any line that begins with a space into the previous line. You may end up having to use a "next" instead of a "previous".

Replace Newlines

I don't believe that grok matches across newlines.

I got around this by doing the following in your filter section. This should go before the grok section:

mutate
{
    gsub => ["message", "\n", "LINE_BREAK"]
}

This allowed me to grok multilines as one big line rather than matching only till the "\n".