how to make two condition check in logstash and write better configuration file

Question

I am using logstash 1.4.2,

I have logstash-forwarder.conf in client log-server like this

{
    "network": {
      "servers": [ "xxx.xxx.xxx.xxx:5000" ],
      "timeout": 15,
      "ssl ca": "certs/logstash-forwarder.crt"
    },
  "files": [
       {
          "paths": [ "/var/log/messages" ],
          "fields": { "type": "syslog" }
        },
        {

          "paths": [ "/var/log/secure" ],
          "fields": { "type": "linux-syslog" }
        }
         ]
}

=========================================================

In logstash server

1. filter.conf

filter {
  if [type] == "syslog" {
date {
        locale => "en"
        match => ["syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss"]
        timezone => "Asia/Kathmandu"
        target => "@timestamp"
        add_field => { "debug" => "timestampMatched"}
   }
    grok {
      match => { "message" => "\[%{WORD:messagetype}\]%{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    }
    syslog_pri { }
  }
if [type] == "linux-syslog" {
date {
        locale => "en"
        match => ["syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss"]
        timezone => "Asia/Kathmandu"
        target => "@timestamp"
        add_field => { "debug" => "timestampMatched"}
   }
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    }
    syslog_pri { }
mutate { replace => [ "syslog_timestamp", "%{syslog_timestamp} +0545" ] }

  }
}

=======================================================

2. output.conf

output {
    if [messagetype] == "WARNING" {
 elasticsearch { host => "xxx.xxx.xxx.xxx" }
  stdout { codec => rubydebug }
}

 if [messagetype] == "ERROR" {
 elasticsearch { host => "xxx.xxx.xxx.xxx" }
  stdout { codec => rubydebug }
}

if [type] == "linux-syslog" {
 elasticsearch { host => "xxx.xxx.xxx.xxx" }
  stdout { codec => rubydebug }
}

}

=======================================================

I want all the logs to forward from /var/log/secure and only ERROR and WARNING log from /var/log/messages, I know this is not a good configuration. I want someone to show me a better way to do this.

Answer 1

I prefer to make decisions about events in the filter block. My input and output blocks are usually quite simple. From there, I see two options.

Use the drop filter

The drop filter causes an event to be dropped. It won't ever make it to your outputs:

filter {
    #other processing goes here

    if [type] == "syslog" and [messagetype] not in ["ERROR", "WARNING"] {
        drop {}
    }
}

The upside of this is that it's very simple.

The downside is that the event is just dropped. It won't be output at all. Which is fine, if that's what you want.

Use a tag

Many filters allow you to add tags, which are useful for communicating decisions between plugins. You could attach a tag telling your output block to send the event to ES:

filter {
    #other processing goes here

    if [type] == "linux-syslog" or [messagetype] in ["ERROR", "WARNING"] {
        mutate {
            add_tag => "send_to_es"
        }
    }
}

output {
    if "send_to_es" in [tags] {
        elasticsearch {
            #config goes here
        }
    }
}

The upside of this is that it allows fine control.

The downside of this is that it's a bit more work, and your ES data ends up a little bit polluted (the tag will be visible and searchable in ES).