How can I diagnose why a k8s pod keeps restarting?

Question

I deploy a elasticsearch to minikube with below configure file:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: elasticsearch
spec:
  replicas: 1
  selector:
    matchLabels:
      name: elasticsearch
  template:
    metadata:
      labels:
        name: elasticsearch
    spec:
      containers:
        - name: elasticsearch
          image: elasticsearch:7.10.1
          ports:
            - containerPort: 9200
            - containerPort: 9300

I run the command kubectl apply -f es.yml to deploy the elasticsearch cluster.

$ kubectl get pod
NAME                            READY   STATUS    RESTARTS   AGE
elasticsearch-fb9b44948-bchh2   1/1     Running   5          6m23s

The elasticsearch pod keep restarting every a few minutes. When I run kubectl describe pod command, I can see these events:

Events:
  Type     Reason     Age                    From               Message
  ----     ------     ----                   ----               -------
  Normal   Scheduled  7m11s                  default-scheduler  Successfully assigned default/elasticsearch-fb9b44948-bchh2 to minikube
  Normal   Pulled     3m18s (x5 over 7m11s)  kubelet            Container image "elasticsearch:7.10.1" already present on machine
  Normal   Created    3m18s (x5 over 7m11s)  kubelet            Created container elasticsearch
  Normal   Started    3m18s (x5 over 7m10s)  kubelet            Started container elasticsearch
  Warning  BackOff    103s (x11 over 5m56s)  kubelet            Back-off restarting failed container

The last event is Back-off restarting failed but I don't know why it restarts the pod. Is there any way I can check why it keeps restarting?

Answer 1

The first step (kubectl describe pod) you've already done. As a next step I suggest checking container logs: kubectl logs <pod_name>. 99% you get the reason from logs in this case (I bet on bootstrap check failure).

When neither describe pod nor logs do not have anything about the error, I get into the container with 'exec': kubectl exec -it <pod_name> -c <container_name> sh. With this you'll get a shell inside the container (of course if there IS a shell binary in it) ans so you can use it to investigate the problem manually. Note that to keep failing container alive you may need to change command and args to something like this:

command:
  - /bin/sh
  - -c
args:
  - cat /dev/stdout

Be sure to disable probes when doing this. A container may restart if liveness probe fails, you will see that in kubectl describe pod if it happen. Since your snippet doesn't have any probes specified, you can skip this.

Answer 2

Checking logs of the pod using kubectl logs podname gives clue about what could go wrong.

ERROR: [2] bootstrap checks failed
[1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
[2]: the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured
ERROR: Elasticsearch did not exit normally - check the logs at /usr/share/elasticsearch/logs/docker-cluster.log

Check this post for a solution