Where exactly do the logs of kubernetes pods come from (at the container level)?

Question

I'm looking to redirect some logs from a command run with kubectl exec to that pod's logs, so that they can be read with kubectl logs <pod-name> (or really, /var/log/containers/<pod-name>.log). I can see the logs I need as output when running the command, and they're stored inside a separate log directory inside the running container.

Redirecting the output (i.e. >> logfile.log) to the file which I thought was mirroring what is in kubectl logs <pod-name> does not update that container's logs, and neither does redirecting to stdout.

When calling kubectl logs <pod-name>, my understanding is that kubelet gets them from it's internal /var/log/containers/ directory. But what determines which logs are stored there? Is it the same process as the way logs get stored inside any other docker container?

Is there a way to examine/trace the logging process, or determine where these logs are coming from?

Answer 1

Logs from the STDOUT and STDERR of containers in the pod are captured and stored inside files in /var/log/containers. This is what is presented when kubectl log is run.

In order to understand why output from commands run by kubectl exec is not shown when running kubectl log, let's have a look how it all works with an example:

First launch a pod running ubuntu that are sleeping forever:

$> kubectl run test --image=ubuntu --restart=Never -- sleep infinity

Exec into it

$> kubectl exec -it test bash

Seen from inside the container it is the STDOUT and STDERR of PID 1 that are being captured. When you do a kubectl exec into the container a new process is created living alongside PID 1:

root@test:/# ps -auxf
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         7  0.0  0.0  18504  3400 pts/0    Ss   20:04   0:00 bash
root        19  0.0  0.0  34396  2908 pts/0    R+   20:07   0:00  \_ ps -auxf
root         1  0.0  0.0   4528   836 ?        Ss   20:03   0:00 sleep infinity

Redirecting to STDOUT is not working because /dev/stdout is a symlink to the process accessing it (/proc/self/fd/1 rather than /proc/1/fd/1).

root@test:/# ls -lrt /dev/stdout
lrwxrwxrwx 1 root root 15 Nov  5 20:03 /dev/stdout -> /proc/self/fd/1

In order to see the logs from commands run with kubectl exec the logs need to be redirected to the streams that are captured by the kubelet (STDOUT and STDERR of pid 1). This can be done by redirecting output to /proc/1/fd/1.

root@test:/# echo "Hello" > /proc/1/fd/1

Exiting the interactive shell and checking the logs using kubectl logs should now show the output

$> kubectl logs test
Hello