Datadog - group by substring of logs

Question

I am trying to create a "Top List" visualization in DataDog and I would like to graph my data which should be grouped by error code. This error code is a substring in logs. An example of a line in the log is given below. I have tried to group my data by message but this is not working, I would like to group my data by substring of the message. Can someone guide me on this?

...Server Error {"error":{"code":1001,"type":"MATCH","message":"Invoke failed: Failed... ...Server Error {"error":{"code":2001,"type":"MATCH","message":"Invoke failed: Failed...

Currently I get the visualization as follows

1.0 .....error:{"code":1001.....
1.0 .....error:{"code":1001.....
1.0 .....error:{"code":2001.....

1.0 is the number of occurrence

Rather I want the visualization as follows

2.0 .....error:{"code":1001.....
1.0 .....error:{"code":2001.....

2.0 will be total 2 occurrences of error 1001 and 1.0 will be the occurrences of error 2001

Answer 1

First you'll want to make sure your logs are well structured (which you can control in Datadog's processing pipelines). Effectively you'll want to parse out the "code" values into some "error code" attribute.

If your log events are in this format...

2020-01-01 12:10:10 myservername - Server Error {"error":{"code":1001,"type":"MATCH","message":"Invoke failed: Failed"}}

...Then all you need is a fairly simple grok parser rule, thanks to the "json" filter function. Something like this would get you where you want (note the %{data::json} part, that's what parses the in-log JSON).

myrulename %{date("yyyy-mm-dd' 'HH:MM:ss"):timestamp} %{notSpace:hostname} - Server Error %{data::json}

Once you've configured this, your logs will also have an attribute called "error.code" with a value of 2001 or 1001 or whatever.

Second you'll want to create a facet for that new error.code attribute so that you can make toplist / timeseries / etc. graphs grouped out by your "error code" facet.