Fluentbit - parsing log with multiple format (log+json)

Question

I'm trying to send to elasticsearch the following log line through fluentbit, but I didn't find the right solution to extract both time and the json structure after the [MLP] part:

2020-12-29 08:00:03,230 INFO [http-nio-3410-exec-7] c.e.m.p.PushManager$ImportResponseImpl - [MLP] {"component":{"name":"importserv","version":"5.4.2"},"details":{"feed":"SomeFEED"},"elapsedMs":354,"event":"import","id":"1.0.58855123705431","name":"image1.png","objType":"Image","outcome":"OK","uuid":"1234566573234242342-123434234-12342"}

Did someone try to achieve something like that? Comments and suggestions are welcome.

Thanks!

Answer 1

You can use regex parser for this.

[PARSER]
    Format regex
    Name logging-parser
    Regex ^(?<time>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3}) (?<level>\S+) .* (?<capturedJson>{.*})$
    Decode_Field json capturedJson
    Time_Format %FT%H:%M:%S,%L
    Time_Key time

First, write a regex that captures your json in a separate group. You can use Rubular for testing the expression. Here is an example that parses your input, json is captured as capturedJson: https://rubular.com/r/NAby4NlVomkdWy

Next, add Decode_Field json capturedJson - this will parse the json captured in the group and extract it's fields, so they are searchable.

Code above is untested for your scenario but we use a similar one to parse envoy logs that also contain json.