Kubernetes liveness probe: can a pod monitor its own stdout?

Question

My idea was to implement a liveness probe as a command, and use something like

$ grep something ERROR

from inside a pod, so that if in the output of a pod, a line containing ERROR exists, the liveness probe fails.

Is this possible? If not, is it possible if I add another container in the same pod, to monitor the first container?

Answer 1

You could query the Kubernetes API server.

The request looks like this:

GET /api/v1/namespaces/{namespace}/pods/{name}/log

To use the token that's usually mounted in a Pod, you can call it like this:

curl https://kubernetes/api/v1/namespaces/default/pods/$HOSTNAME/log -k \
     -H "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)"

Answer 2

Sure you can have more then one container per pod. It's just not a standard approach because it violate the "one process per container" principle. You can read this article explaining multi-container pods in Kubernetes, what are the use cases and so on.

apiVersion: v1
kind: Pod
metadata:
  name: test
spec:
  containers:
  - name: 1st
    image: nginx
    volumeMounts:
    - name: html
      mountPath: /tmp/html
    livenessProbe:
      exec:
        command:
        - cat
        - /tmp/html/test
      initialDelaySeconds: 10
      periodSeconds: 3
  - name: 2nd
    image: debian
    volumeMounts:
    - name: html
      mountPath: /tmp/html
    command: ["/bin/sh", "-c"]
    args:
      - while true; do
          date >> /tmp/html/test;
          sleep 5;
        done
  volumes:
  - name: html
    emptyDir: {}

In this example pod have two containers. Container 1st running nginx with mounted /tmp/html and livenessProbe which after 10 seconds from start checks if file /tmp/html/test exists every 3 seconds and if it's missing it restarts the container. And 2ndcontainer running debian with /tmp/html/ mounted, but it is also adding data entry into file /tmp/html/test each 5 seconds.

In the above example if you remove the file manually and probe caches it it will restart 1st container.

You would have to tailor this example into your particular needs, for example use grep -q ERROR /tml/html/test which if succeed removes the /tmp/html/test or change the probe itself.