using Grok to skip parts of message or logs

Question

I have just started using grok for logstash and I am trying to parse my log file using grok filter. My logline is something like below

03-30-2017 13:26:13 [00089] TIMER XXX.TimerLog: entType [organization], queueType [output], memRecno = 446323718, audRecno = 2595542711, elapsed time = 998ms

I want to capture only the initial date/time stamp, entType [organization], and elapsed time = 998ms.

However, it looks like I have to match pattern for every word and number in the line. Is there a way I can skip it ? I tried to look everywhere but couldn't find anything. Kindly help.

Answer 1

As per Charles Duffy's comment.

There are 2 ways of doing this: The GREEDYDATA way (?:.*):

grok {
  match => {"message" => "^%{DATE_US:dte}\s*%{TIME:tme}\s*\[%{GREEDYDATA}elapsed time\s*=\s*%{BASE10NUM}"
}

Or, telling it to ignore a match and look for the next one in the list.

grok {
  break_on_match => false
  match => { "message" => "^%{DATE_US:dte}\s*%{TIME:tme}\s*\[" }
  match => { "message" => "elapsed time\s*=\s*%{BASE10NUM:elapsedTime}"
}

You can then rejoin the date & time into a single field and convert it to a timestamp.

Answer 2

As Charles Duffy suggested, you can simply bypass data you don't need.

You can use .* to do that.

Following will produce the output you want,

%{DATE_US:dateTime}.*entType\s*\[%{WORD:org}\].*elapsed time\s*=\s*%{BASE10NUM}

Explanation:

• \s* matches space character.
• \[ is bypassing [ character.
• %{WORD:org} defines a word boundary and place it in a new field org

Outputs

{
  "dateTime": [
    [
      "03-30-2017"
    ]
  ],
  "MONTHNUM": [
    [
      "03"
    ]
  ],
  "MONTHDAY": [
    [
      "30"
    ]
  ],
  "YEAR": [
    [
      "2017"
    ]
  ],
  "org": [
    [
      "organization"
    ]
  ],
  "BASE10NUM": [
    [
      "998"
    ]
  ]
}

Click for a list of all available grok patterns