Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to list Kubernetes recently deleted pods?

Tags:

kubernetes

Is there a way to get some details about Kubernetes pod that was deleted (stopped, replaced by new version).

I am investigating bug. I have logs with my pod name. That pod does not exist anymore, it was replaced by another one (with different configuration). New pod resides in same namespace, replication controller and service as old one.

Commands like

kubectl  get pods kubectl  get pod <pod-name>  

work only with current pods (live or stopped).

How I could get more details about old pods? I would like to see

  1. when they were created
  2. which environment variables they had when created
  3. why and when they were stopped
like image 287
Bartosz Bilicki Avatar asked Nov 16 '16 15:11

Bartosz Bilicki


People also ask

How can I recover my deleted pods in Kubernetes?

As of today, kubectl get pods -a is deprecated, and as a result you cannot get deleted pods. You can then investigate further issues within your logging pipeline if you have one in place.

How do I check my Kubernetes pod history?

In case that a pod restarts, and you wanted to check the logs of the previous run, what you need to do is to use the --previous flag: kubectl logs nginx-7d8b49557c-c2lx9 --previous.

What happens when Kubernetes pod is deleted?

They will be scheduled onto other nodes by their controller (Deployment, ReplicaSet, etc.). This command is blocking and will return when the pods have been removed. You can run multiple drain commands in separate shells to drain multiple nodes, but only one at a time will execute.

How do I check for eviction pods in Kubernetes?

The command result reveals that many pods have evicted status after running kubectl get pods. The results of the check will be saved in the node's kubelet logs. To find the information, run cat /var/paas/sys/log/kubernetes/kubelet. log | grep -i Evicted -C3.


2 Answers

As of today, kubectl get pods -a is deprecated, and as a result you cannot get deleted pods.

What you can do though, is to get a list of recently deleted pod names - up to 1 hour in the past unless you changed the ttl for kubernetes events - by running:

kubectl get event -o custom-columns=NAME:.metadata.name | cut -d "." -f1

You can then investigate further issues within your logging pipeline if you have one in place.

like image 152
iomv Avatar answered Oct 19 '22 20:10

iomv


as far as i know you may not get the pod details once the pod is deleted. can i know what is the usecase?

example:

  1. if a pod created using - kubectl run busybox-test-pod-status --image=busybox --restart=Never -- /bin/false you will have a pod with status termiated:error
  2. if a pod is created using - kubectl run busybox-test-pod-status --image=busybox --restart=Never -- /bin/true you will have pod with status terminated:Complted
  3. if container in a pods restarts: the pod will be alive and you can get the logs of previous container (only the previous container) using kubectl logs --container < container_name > --previous=true < pod_name >
  4. if you doing an upgrade of you app and you are creating pods using deployments. if the update deployment "say a new image". pod will be terminated and new pod will be created. you can get the pod details from depoyment yaml. if you want to get details of previous pod you have see "spec" section of previous deployment yaml
like image 33
Sandeep kumar singh Avatar answered Oct 19 '22 20:10

Sandeep kumar singh