Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How can I have logstash drop all events that do not match a group of regular expressions?

Tags:

logstash

I'm trying to match event messages with several regular expressions. I was going for the use of grep filter, but its deprecated so I'm trying for the drop with negation.

The functionality I'm looking for is to have all events dropped unless the message matches several regular expressions.

The filter bellow does not work, but tested individually both expressions work fine. What am I missing?

filter {    
    if ([message] !~ ' \[critical\]:  ' or [message] !~ '\[crit\]: ') {
        drop { }
    }
}
like image 628
lnaia Avatar asked Dec 12 '14 12:12

lnaia


People also ask

What is regex in Logstash?

Grok is filter within Logstash that is used to parse unstructured data into something structured and queryable. Regular expression is a sequence of characters that define a search pattern.

What is grok pattern in Logstash?

Put simply, grok is a way to match a line against a regular expression, map specific parts of the line into dedicated fields, and perform actions based on this mapping. Built-in, there are over 200 Logstash patterns for filtering items such as words, numbers, and dates in AWS, Bacula, Bro, Linux-Syslog and more.

What is grok pattern?

A grok pattern is like a regular expression that supports aliased expressions that can be reused. This processor comes packaged with many reusable patterns. If you need help building patterns to match your logs, you will find the Grok Debugger tool quite useful! The Grok Constructor is also a useful tool.


1 Answers

I was reading a bit more and went along with painting the events with grok by adding a tag and dropping them in the end, if the tag was not there:

filter {
  grok {
    add_tag => [ "valid" ]
    match => [ 
      "message", ".+ \[critical\]: ?(.+)",
      "message", ".+ \[crit\]: ?(.+) ",
      "message", '.+ (Deadlock found.+) ',
      "message", "(.+: Could not record email: .+) "
    ]
  }

  if "valid" not in [tags] {            
    drop { }
  }

  mutate {
    remove_tag => [ "valid" ]
  }
} 
like image 195
lnaia Avatar answered Apr 28 '23 22:04

lnaia