Is there a way to specify file size limit for docker logs on Google Container Optimized OS?

Question

I had a very long running (and verbose) container job on Google's Container-Optimized OS (COS) that eventually generated enough logs to fill the disk.

To my knowledge there isn't a way to rotate / limit log file size while using COS.

Inspecting the running container, it appears that it writes an ever-growing file to /var/lib/docker/containers/ (mounted on the stateful partition) and that HostConfig.LogConfig.Config is empty.

I ended up having to SSH in and manually delete the multi-gigabyte log file to make the VM operational again.

I read through https://cloud.google.com/compute/docs/containers/configuring-options-to-run-containers and as far as I can tell there isn't a way to (say) pass --log-opt max-size=XX as per the Docker documentation: https://docs.docker.com/config/containers/logging/json-file/

Is there some way to pass that flag? Failing that, are there recommendations on how to rotate logs / limit log size / avoid hitting this problem?

Answer 1

Currently there isn't any way to pass that flag directly to your container. That being said there is a workaround that might work for you.

It requires you to set the flag globally in /etc/docker/daemon.json It will then apply to all containers started on that VM. You can do that with a startup script, by using this fragment:

cat <<EOF > /etc/docker/daemon.json
{
  "live-restore": true,
  "storage-driver": "overlay2",
  "log-opts": {
    "max-size": "10m"
  }
}
EOF
systemctl restart docker

See https://cloud.google.com/compute/docs/startupscript#providing_startup_script_contents_directly for instructions on startup scripts. Please refer to docker's documentation on the "log-opts" flag: https://docs.docker.com/config/containers/logging/json-file/#usage

Answer 2

You can limit the logs file size for each service in docker-compose.yml.

Just need to add these lines:

services:
  app:
    container_name: app
    image: node
    restart: always
    volumes:
      - ./app:/home/node/app
    working_dir: /home/node/app
    ports:
      - 3000:3000
    networks:
      - main
    command: "npm start" 
    logging:
      driver: "json-file"
      options:
        max-file: "5"   # number of files or file count
        max-size: "10m" # file size

  db:
    ...
    logging:
      driver: "json-file"
      options:
        max-file: "3"   # number of files or file count
        max-size: "10m" # file size