Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to debug evicted pod on k8s

I have searched some but could not get much as I am new to k8s. My pods are evicted and I get message as:

"Status:        Failed
Reason:         Evicted
Message:        The node was low on resource: nodefs."

Any help how can I figure out what is going on?

like image 370
AhmFM Avatar asked Nov 26 '18 19:11

AhmFM


People also ask

How do you debug an eviction pod?

Run kubectl describe pod <pod name> and look for the node name of this pod. Followed by kubectl describe node <node-name> that will show what type of resource cap the node is hitting under Conditions: section. From my experience this happens when the host node runs out of disk space.

How do I check logs of eviction pods in Kubernetes?

The command result reveals that many pods have evicted status after running kubectl get pods. The results of the check will be saved in the node's kubelet logs. To find the information, run cat /var/paas/sys/log/kubernetes/kubelet. log | grep -i Evicted -C3.

What happens when a pod is evicted?

This one will start to failed Pods until the node's used resources are under the eviction threshold, which means that the Kubelet will terminate all Pod's containers and set its PodPhase as Failed. If a Deployment manages the evicted Pod, the Deployment creates another Pod to be scheduled by Kubernetes.


1 Answers

Run kubectl describe pod <pod name> and look for the node name of this pod. Followed by kubectl describe node <node-name> that will show what type of resource cap the node is hitting under Conditions: section.

From my experience this happens when the host node runs out of disk space.

like image 199
ZPrime Avatar answered Oct 19 '22 17:10

ZPrime