Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Can Filebeat use multiple config files?

I have several applications running on a single server. I'd like to use filebeat to ship the logs of each of them to logstash. However, for the sake of configuration management, I'd like to be able to add configuration to filebeat for each app separately.

Logstash reads its config from a conf.d directory. It's my understanding that one can add files there and they get combined when logstash loads them. Is there any similar feature for filebeat? Or am I stuck with maintaining a single filebeat.yml file per server?

I'm running both filebeat and logstash as services on CentOS 7, using the yum/rpm packages from elastic's repositories. Filebeat is version 1.3.1 and logstash is version 2.4.0.

like image 914
izrik Avatar asked Oct 11 '16 18:10

izrik


People also ask

Can Filebeat have multiple inputs?

You can have as many inputs as you want but you can only have one output, you will need to send your logs to a single logstash and from there you can send them to other places. Save this answer. Show activity on this post. Filebeat does not support sending the same data to multiple logstash servers simultaneously.

What kind of data does Filebeat collect?

Filebeat is a lightweight shipper for forwarding and centralizing log data. Installed as an agent on your servers, Filebeat monitors the log files or locations that you specify, collects log events, and forwards them either to Elasticsearch or Logstash for indexing.

What is registry in Filebeat?

I know that "registry" is for "tracking files that filebeat is harvesting or is harvested", but for details, how to understand it. Take following as example, what does "timestamp" and "ttl" mean ?


1 Answers

Yes, Filebeat has a conf.d like feature, but it is not enabled by default. Filebeat will look inside of the declared directory for additional *.yml files that contain prospector configurations. The configuration varies by Filebeat major version.

Filebeat 7.x:

The behavior is the same as 6.x, but the config option is filebeat.config.inputs instead of filebeat.config.prospectors.

# /etc/filebeat/filebeat.yml
filebeat.config.inputs:
  enabled: true
  path: inputs.d/*.yml

Then create individual config files for each app that's generating logs.

# /etc/filebeat/inputs.d/someapp.yml
- paths:
  - /var/log/someapp/stdout.log
  fields:
    app: someapp

Filebeat 6.x:

You specify a path option in the filebeat.config.prospectors section of the filebeat.yml file.

filebeat.config.prospectors:
  enabled: true
  path: /etc/filebeat/conf.d/*.yml

/etc/filebeat/conf.d/someapp.yml

Note that this file does not contain filebeat.prospectors like it did in earlier versions.

- paths:
    - /var/log/someapp/stdout.log
  fields:
    app: someapp

Filebeat 1.x and 5.x:

You declare the directory inside of the main filebeat.yml using the config_dir option.

filebeat:
  config_dir: /etc/filebeat/conf.d

/etc/filebeat/conf.d/someapp.yml

filebeat:
  prospectors:
    - paths:
        - /var/log/someapp/stdout.log
      fields:
        app: someapp
like image 127
A J Avatar answered Sep 25 '22 17:09

A J