grok parse multiple lines, for example exception stack trace

Question

I have multiple(three) types of log in my log file. One of the types has some own prints + exception stack trace. The example is listed below:

Multiple lines example:
    2018-04-27 10:53:17 [http-nio-8088-exec-4] - ERROR - app-info-exception-info - params:{"cardid":"111111111","txamt":10,"ip":"192.168.16.89","stationcode":"0002","inputuserid":1,"organcode":"99999"} java.lang.NullPointerException: null
        at com.datalook.group.BusinessHandler.handler(BusinessHandler.java:93) ~[classes/:?]
        at com.datalook.group.BusinessGroupController.businessGroup(BusinessGroupController.java:51) [classes/:?]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_77]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_77]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_77]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_77]

I have a patterns to parse it, it is:

#pattern:
(?<timestamp>[\d\-\s\:]+)\s\[(?<threadname>[\w\-\d]+)\]\s-\s(?<loglevel>[\w]+)\s\-\s(?<appinfo>app-info-exception-info)\s-\s(?<params>params):(?<jsonstr>[\"\w\d\,\:\.\{\}]+)\s(?<exceptionname>[\w\d\.]+Exception):\s(?<exceptiondetail>[\w\d\.]+)\n\t(?<extralines>at[\s\w\.\d\~\?\n\t\(\)\_\[\]\/\:\-]+)\n

Pattern has error(actually not error, but not parse wholely or as expected) in parse the multiline exception stack trace, mostly in last two parts(exceptiondetail (null in this case) and extralines(those lines starting with space or tabs plus 'at', or lines after first line of stack trace)). Any better idea than I did?

In filebeat.yml, I have following configured:

# The regexp Pattern that has to be matched. The example pattern matches all lines starting with [
  multiline.pattern: '^[[:space:]]'

  # Defines if the pattern set under pattern should be negated or not. Default is false.
  multiline.negate: false

  multiline.match: after

Any idea to improve parse multiple lines (exception stacktrace)?

Answer 1

How about making it simpler? assigning extra data (all lines start with at) to GREEDYDATA into a single field using (?m)?

For example, if this is your log,

2018-04-27 10:53:17 [http-nio-8088-exec-4] - ERROR - app-info-exception-info - params:{"cardid":"111111111","txamt":10,"ip":"192.168.16.89","stationcode":"0002","inputuserid":1,"organcode":"99999"} java.lang.NullPointerException: null
        at com.datalook.group.BusinessHandler.handler(BusinessHandler.java:93) ~[classes/:?]
        at com.datalook.group.BusinessGroupController.businessGroup(BusinessGroupController.java:51) [classes/:?]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_77]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_77]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_77]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_77]

You can parse it as,

%{TIMESTAMP_ISO8601:timestamp} \[%{DATA:threadname}\] - %{LOGLEVEL:loglevel} - app-info-exception-info - params:%{SPACE}\{\"%{DATA:jsondata}\"\} %{DATA:excentionname}: %{DATA:exceptiondetail}\n(?m)%{GREEDYDATA:extralines}

which will output,

{
  "timestamp": [
    [
      "2018-04-27 10:53:17"
    ]
  ],
  "YEAR": [
    [
      "2018"
    ]
  ],
  "MONTHNUM": [
    [
      "04"
    ]
  ],
  "MONTHDAY": [
    [
      "27"
    ]
  ],
  "HOUR": [
    [
      "10",
      null
    ]
  ],
  "MINUTE": [
    [
      "53",
      null
    ]
  ],
  "SECOND": [
    [
      "17"
    ]
  ],
  "ISO8601_TIMEZONE": [
    [
      null
    ]
  ],
  "threadname": [
    [
      "http-nio-8088-exec-4"
    ]
  ],
  "loglevel": [
    [
      "ERROR"
    ]
  ],
  "SPACE": [
    [
      ""
    ]
  ],
  "jsondata": [
    [
      "cardid":"111111111","txamt":10,"ip":"192.168.16.89","stationcode":"0002","inputuserid":1,"organcode":"99999"
    ]
  ],
  "excentionname": [
    [
      "java.lang.NullPointerException"
    ]
  ],
  "exceptiondetail": [
    [
      "null"
    ]
  ],
  "extralines": [
    [
      "        at com.datalook.group.BusinessHandler.handler(BusinessHandler.java:93) ~[classes/:?]\n        at com.datalook.group.BusinessGroupController.businessGroup(BusinessGroupController.java:51) [classes/:?]\n        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_77]\n        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_77]\n        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_77]\n        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_77]"
    ]
  ]
}

you can replace (?m) with %{SPACE} to break each line that start with at into its own field as well.