What are some best practices for handling large logstash configuration files?

Question

I about to deploy a logstash instance that will handle a variety of inputs and do multiple filter actions. The configuration file will most likely end up having a lot of if-then statements given the complexity and number of the inputs.

My questions are:

1. Is there any way to make the configuration file more 'modular'? In a programming sense, I would create functions/subroutines so that I could test independently. I've thought about dynamically creating mini-configuration files that I can use for testing. These mini files could then be combined into one production configuration.

2. Are there any "best practices" for testing, deploying and managing more complicated Logstash configurations?

Thanks!

Answer 1

There's no support for functions/subroutines per se. I break up different filters into separate files to keep a logical separation and avoid having gigantic files. I also have inputs and outputs in different files. That way I can combine all filters with debug inputs/output, for example

input {
  stdin {}
}

output {
  stdout {
    codec => rubydebug
  }
}

and invoke Logstash by hand to inspect the results of given input. Since filter ordering matters I'm using the fact that Logstash reads configuration files in alphabetical order, so the files are named NN-some-descriptive-name.conf, where NN is an integer.

I've also written a script that automates this process by letting you write a spec with test inputs and the expected resulting messages, and if there's a mismatch it'll bail out with an error and display the diff. I may be able to open source it.

As for deployment, use any configuration management system like Puppet, Chef, SaltStack, Ansible, CFEngine, or similar that you're familiar with. I'm quite happy with Ansible.

Answer 2

As @Magnus Bäck stated, the answer to 1. is no. currently there is no support for functions.

But as for your second question, there is a way to make the logstash configuration more modular. you can split the configuration file to multiple files, and point logstash to the files directory.

check the directory option in logstash man:

-f, --config CONFIG_PATH      Load the logstash config from a specific file
                                  or directory.  If a direcory is given, all
                                  files in that directory will be concatonated
                                  in lexicographical order and then parsed as a
                                  single config file. You can also specify
                                  wildcards (globs) and any matched files will
                                  be loaded in the order described above.