Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Can you use environment variables in config file for fluentd

I was wondering how to use env vars in the Fluentd config, I tried:

<match **>
type elasticsearch
logstash_format true
logstash_prefix $ENV_VAR
host ***
port ***
include_tag_key true
tag_key _key
</match>

but it doesn't work, any idea?

like image 314
Daniel Conde Garcia Avatar asked Dec 01 '14 17:12

Daniel Conde Garcia


1 Answers

EDIT:

Here is a much better solution:

If you pass "--use-v1-config" option to Fluentd, this is possible with the "#{ENV['env_var_name']" like this:

<match foobar.**> # ENV["FOO"] is foobar
  type elasticsearch
  logstash_prefix "#{ENV['FOO']}"
  logstash_format true
  include_tag_key true
  tag_key _key
  host ****
  port ****
</match>

Old, kludgey answer is here.

  1. Install fluent-plugin-record-reformer and fluent-plugin-forest
  2. Update your config as follows.

<match hello.world>
  type record_reformer
  tag ${ENV["FOO"]}.${tag_prefix[-1]} # adding the env variable as a tag prefix
</match>

<match foobar.**> # ENV["FOO"] is foobar
  type forest
  subtype elasticsearch
  <template>
    logstash_prefix ${tag_parts[0]}
    logstash_format true
    include_tag_key true
    tag_key _key
    host ****
    port ****
  </template>
</match>

In particular, do NOT use <match **> there. That would catch all events and will lead to behaviors that are hard to debug.

like image 150
Kiyoto Tamura Avatar answered Oct 10 '22 02:10

Kiyoto Tamura