Does Fluentd support log rotation for file output?

Question

The current setup I am working with is a Docker compose stack with multiple containers. These containers send their logging information to a logging container (inside the compose stack) running the Fluentd daemon. The configuration for Fluentd consists of one in_forward source that collects the logs and writes them to separate files, depending on the container. My Fluentd configuration file looks similar to this:

<source>
  @type forward
  port 24224
  bind 0.0.0.0
</source>

<match container1>
   @type copy
   <store>
     @type file
     path /fluentd/log/container1.*.log
     format single_value
     message_key "log"
   </store>
</match>

...

My docker-compose.yml file looks something like this:

version: '3'

services:

  container1:
    build: ./container1
    container_name: "container1" 
    depends_on:
     - "logger" 
    logging:
      driver: "fluentd"
      options:
        tag: container1  
    networks:
      static-net:
        ipv4_address: 172.28.0.4  


  ...


  logger:
    build: ./logger
    container_name: "logger"
    ports:
     - "24224:24224"
     - "24224:24224/udp"
    volumes:
     - ./logger/logs:/fluentd/log
    networks:
      static-net:
        ipv4_address: 172.28.0.5          

networks:
  static-net:
    ipam:
      driver: default
      config:
       - subnet: 172.28.0.0/16

Everything works as expected, but I would ideally like to set Fluentd to keep a certain number of log files. I can change the size of the log files by configuring the chunk_limit_size parameter in a buffer section. However, even though I want this option, I still do not want Fluentd writing an endless amount of files. The buffer_queue_limit and overflow_action in the buffer configuration do not seem to affect anything. This application will be running continuously once deployed, so log rotation is a necessity. Several questions I have:

1. Does Fluentd support log rotation for writing logs to files? If so, what parameters do I set in the Fluentd configuration file?
2. If not, can I configure Docker in such a way that I can utilize its log rotation of the json logging driver for Fluentd?
3. And if that is not possible, is there a way to add log rotation into Fluentd via a plugin or perhaps in the Fluentd docker container itself (or a sidecar container)?

Answer 1

When you are using fluentd logging driver for docker then there is no container log files, there are only fluentd logs, and to rotate them you can use this link. If you want docker to keep logs and to rotate them, then you have to change your stackfile from:

    logging:
      driver: "fluentd"
      options:
        tag: container1  

to

  logging:
   driver: "json-file"
   options:
      max-size: "5m" #max size of log file
      max-file: "2" #how many files should docker keep

in fluentd you have to use the in_tail plugin instead of forward (fluentd should have access to log files in /var/lib/docker/containers/*/*-json.log )

<source>
  type tail
  read_from_head true
  pos_file fluentd-docker.pos
  path /var/lib/docker/containers/*/*-json.log
  tag docker.*
  format json
</source>