Tail multiple logs fluentd

Question

I'm trying to tail multiple logs in fluentd with the following configuration:

<source>
  type tail
  tag es.workers.worker1

  format /^\[(?<timestamp>.*? .*?) (?<log_level>[INFO|ERROR][^\]]*)\] (?<message>.*)$/

  path /var/log/upstart/worker1.log
  pos_file /var/lib/fluentd/pos/-var-log-upstart-worker1.log.pos

</source>
<source>
  type tail
  tag es.workers.worker2

  format /^\[(?<timestamp>.*? .*?) (?<log_level>[INFO|ERROR][^\]]*)\] (?<message>.*)$/

  path /var/log/upstart/worker2.log
  pos_file /var/lib/fluentd/pos/-var-log-upstart-worker2.log.pos

</source>
<source>
  type tail
  tag es.workers.worker3

  format /^\[(?<timestamp>.*? .*?) (?<log_level>[INFO|ERROR][^\]]*)\] (?<message>.*)$/

  path /var/log/upstart/worker3.log
  pos_file /var/lib/fluentd/pos/-var-log-upstart-worker3.log.pos

</source>
<source>
  type tail
  tag es.workers.worker4

  format /^\[(?<timestamp>.*? .*?) (?<log_level>[INFO|ERROR][^\]]*)\] (?<message>.*)$/

  path /var/log/upstart/worker4.log
  pos_file /var/lib/fluentd/pos/-var-log-upstart-worker4.log.pos

</source>

This isn't working. Usually (but not always), I'm only getting logs of the first file. Sometimes it's a different file, but it's always only one. Any ideas as to what's going on? I'm not getting any meaningful errors in the fluentd error log.

Answer 1

tailing multiple files can be done like this (the tag will be based in the file name)

<source>
  @type tail
  @id in_tail_container_logs
  path /var/lib/docker/containers/*/*-json.log
  pos_file /fluentd/log/containers.log.pos
  time_format "%Y-%m-%dT%H:%M:%S.%L%Z"
  keep_time_key true
  read_from_head true
  tag "docker.*"
  format json
</source>

or like this

<source>
  @type tail
  @id in_tail_fos_logs
  @label @LOGS
  path /www/web/log/*.log,/www/web2/log/*.log,/www/web3/log/*.log   
  exclude_path ["/www/web/log/logstash_*.log"]
  pos_file /var/log/td-agent/logs.log.pos
  time_format "%Y-%m-%dT%H:%M:%S.%L%Z"
  read_from_head true
  tag "rowlogs.*"
  format none
</source>