how to reconnect to a docker logs --follow where the log file was deleted

Question

I have a docker container running in a small AWS instance with limited disk space. The logs were getting bigger, so I used the commands below to delete the evergrowing log files:

sudo -s -H
find /var -name "*json.log" | grep docker | xargs -r rm
journalctl --vacuum-size=50M

Now I want to see what's the behaviour of one of the running docker containers, but it claims the log file has disappeared (from the rm command above):

ubuntu@x-y-z:~$ docker logs --follow name_of_running_docker_1
error from daemon in stream: Error grabbing logs: open /var/lib/docker/containers/d9562d25787aaf3af2a2bb7fd4bf00994f2fa1a4904979972adf817ea8fa57c3/d9562d25787aaf3af2a2bb7fd4bf00994f2fa1a4904979972adf817ea8fa57c3-json.log: no such file or directory

I would like to be able to see again what's going on in the running container, so I tried:

sudo touch /var/lib/docker/containers/d9562d25787aaf3af2a2bb7fd4bf00994f2fa1a4904979972adf817ea8fa57c3/d9562d25787aaf3af2a2bb7fd4bf00994f2fa1a4904979972adf817ea8fa57c3-json.log

And again docker follow, but while interacting with the software that should produce logs, I can see that nothing is happening.

Is there any way to rescue the printing into the log file again without killing (rebooting) the containers?

Answer 1

Is there any way to rescue the printing into the log file again without killing (rebooting) the containers?

Yes, but it's more of a trick than a real solution. You should never interact with /var/lib/docker data directly. As per Docker docs:

part of the host filesystem [which] is managed by Docker (/var/lib/docker/volumes/ on Linux). Non-Docker processes should not modify this part of the filesystem.

---

For this trick to work, you need to configure your Docker Daemon to keep containers alive during downtime before first running our container. For example, by setting your /etc/docker/daemon.json with:

{
  "live-restore": true
}

This requires Daemon restart such as sudo systemctl restart docker.

Then create a container and delete its .log file:

$ docker run --name myhttpd -d httpd:alpine
$ sudo rm $(docker inspect myhttpd -f '{{ .LogPath }}')

# Docker is not happy
$ docker logs myhttpd
error from daemon in stream: Error grabbing logs: open /var/lib/docker/containers/xxx-json.log: no such file or directory

Restart Daemon (with live restore), this will cause Docker to somehow re-take management of our container and create our log file back. However, any logs generate before log file deletion are lost.

$ sudo systemctl restart docker
$ docker logs myhttpd # works! and log file is created back

Note: this is not a documented or official Docker feature, simply a behavior I observed with my own experimentations using Docker 19.03. It may not work with other Docker versions

With live restore enabled, our container process keeps running even though Docker Daemon is stopped. On Docker daemon restart, it probably somehow try to re-read from the still alive process stdout and stderr and redirect output to our log file (hence re-creating it)