how to implement the unit or integration tests for logstash configuration?

Question

With the logstash 1.2.1 one can now have conditional to do various stuff. Even the earlier version's conf file can get complicated if one is managing many log files and implement metric extraction.

After looking at this comprehensive example, I really wondered my self, how can I detect any breakages in this configuration?

Any ideas.

Answer 1

For a syntax check, there is --configtest:

java -jar logstash.jar agent --configtest --config <yourconfigfile>

To test the logic of the configuration you can write rspec tests. This is an example rspec file to test a haproxy log filter:

require "test_utils"

describe "haproxy logs" do
  extend LogStash::RSpec

  config <<-CONFIG
  filter {
    grok {
       type            => "haproxy"
       add_tag         => [ "HTTP_REQUEST" ]
       pattern         => "%{HAPROXYHTTP}"
    }
    date {
       type            => 'haproxy'
       match           => [ 'accept_date', 'dd/MMM/yyyy:HH:mm:ss.SSS' ]
    }
  }
  CONFIG

  sample({'@message' => '<150>Oct 8 08:46:47 syslog.host.net haproxy[13262]: 10.0.1.2:44799 [08/Oct/2013:08:46:44.256] frontend-name backend-name/server.host.net 0/0/0/1/2 404 1147 - - ---- 0/0/0/0/0 0/0 {client.host.net||||Apache-HttpClient/4.1.2 (java 1. 5)} {text/html;charset=utf-8|||} "GET /app/status HTTP/1.1"',
          '@source_host' => '127.0.0.1',
          '@type' => 'haproxy',
          '@source' => 'tcp://127.0.0.1:60207/',

      }) do

    insist { subject["@fields"]["backend_name"] } == [ "backend-name" ]
    insist { subject["@fields"]["http_request"] } == [ "/app/status" ]
    insist { subject["tags"].include?("HTTP_REQUEST") }
    insist { subject["@timestamp"] } == "2013-10-08T06:46:44.256Z"
    reject { subject["@timestamp"] } == "2013-10-08T06:46:47Z"
  end
end

This will, based on a given filter configuration, run input samples and test if the expected output is produced.

To run the test, save the test as haproxy_spec.rb and run `logstash rspec:

java -jar logstash.jar rspec haproxy_spec.rb

There are lots of spec examples in the Logstash source repository.