Why I cannot get output of tail of a file created by docker build

Question

Docker info :

Containers: 18
 Running: 18
 Paused: 0
 Stopped: 0
Images: 188
Server Version: 1.13.1
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: aa8187dbd3b7ad67d8e5e3a15115d3eef43a7ed1
runc version: 9df8b306d01f59d3a8029be411de015b7304dd8f
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.9.0-1-amd64
Operating System: Debian GNU/Linux 9 (stretch)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 15.56 GiB
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Experimental: false
Insecure Registries:
   127.0.0.0/8
Live Restore Enabled: false

Docker version : Docker version 1.13.1, build 092cba372

I have this docker file :

FROM debian:latest
RUN touch /var/log/mylog.log
CMD ["tail", "-F", "/var/log/mylog.log"]

Building it with docker build . -t test/test and running it with docker run -ti test/test will tail the file in stdout like this (ignore the tail warning) :

$ docker run --name test -t test/test
tail: unrecognized file system type 0x794c7630 for '/var/log/mylog.log'.   please report this to [email protected]. reverting to polling

Executing the following command will write to the file /var/log/mylog.log which is a file being followed by tail :

docker exec -ti test bash -c "echo 'asd' >> /var/log/mylog.log"

Unfortunately, there is no output in the other terminal even if cat shows that there is content in the file :

$ docker exec -ti test bash -c "cat /var/log/mylog.log"
asd

Although, if I create the file with PID1 instead of creating it inside the dockerfile, I get to see the content of the file with the tail. I can also get the content of the tail if I docker stop test && docker start test with earlier commands.

What's going on exactly ? Is there something different when creating files in the docker build vs the running script in the live container ?

Answer 1

The same issue occurs for me using the overlay2 storage driver on the Docker 4.9.8-moby Alpine release.

It seems like CMD tail is opening the /var/log/mylog.log file from the overlay layer that RUN touch /var/log/mylog.log creates.

When you append to the log, a "new" file is created in the topmost overlay layer that the container uses for any file system changes made on top of the image while running, and this new file is actually being appended to. tail is not able to pick up the changeover correctly though, with either -f or -F.

The docker start and docker stop resolves the problem as the tail process starts again after /var/log/mylog.log has been updated and is then pointing at the "new" file in the container overlay layer. Using a slightly different CMD would workaround the issue in a similar way:

CMD ["sh", "-c", "touch /var/log/mylog.log && tail -f /var/log/mylog.log"]

The debian:testing image includes coreutils-8.26-2 with the fix for supporting overlays magic number to remove that warning message, but still exhibits the same behaviour.

It's most likely an overlay issue to be fixed in the kernel. coreutils might be able to work around the issue when using -F.

What you are attempting is a bit of an edge case in Docker though. Containers that use tail as the foreground process usually complete a bunch of work in a script before running tail, which includes running the commands that create the log file to be tailed. Might be why not many people have picked this up.

Answer 2

Your CMD json syntax is invalid, you are missing comas. Otherwise, it works fine in my environment:

$ cat df.tail
FROM debian:latest
RUN touch /var/log/mylog.log
CMD ["tail", "-F", "/var/log/mylog.log"]

$ docker build -t test-tail -f df.tail .
Sending build context to Docker daemon 265.7 kB
Step 1/3 : FROM debian:latest
 ---> 7b0a06c805e8
Step 2/3 : RUN touch /var/log/mylog.log
 ---> Using cache
 ---> 412295d58dc8
Step 3/3 : CMD tail -F /var/log/mylog.log
 ---> Using cache
 ---> d7e9c7fee4a0
Successfully built d7e9c7fee4a0

$ docker run -t --rm --name test-tail test-tail
asd
^C
$

From a second window, this was run after the above docker run command:

$ docker exec test-tail bash  -c "echo 'asd' >> /var/log/mylog.log"