Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

"Ghost" kubernetes pod stuck in terminating

The situation

I have a kubernetes pod stuck in "Terminating" state that resists pod deletions

NAME                             READY STATUS       RESTARTS   AGE
...
funny-turtle-myservice-xxx-yyy   1/1   Terminating  1          11d
...

Where funny-turtle is the name of the helm release that have since been deleted.

What I have tried

try to delete the pod.

Output: pod "funny-turtle-myservice-xxx-yyy" deleted Outcome: it still shows up in the same state. - also tried with --force --grace-period=0, same outcome with extra warning

warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.

try to read the logs (kubectl logs ...).

Outcome: Error from server (NotFound): nodes "ip-xxx.yyy.compute.internal" not found

try to delete the kubernetes deployment.

but it does not exist.

So I assume this pod somehow got "disconnected" from the aws API, reasoning from the error message that kubectl logs printed.

I'll take any suggestions or guidance to explain what happened here and how I can get rid of it.

EDIT 1

Tried to see if the "ghost" node was still there (kubectl delete node ip-xxx.yyy.compute.internal) but it does not exist.

like image 716
Yann Pellegrini Avatar asked Jul 27 '18 13:07

Yann Pellegrini


People also ask

Why do pods get stuck in terminating state?

A pod is stuck in a terminating state as the configmap mounted as a volume fails to unmount when trying to clean the subPath mount for it.

How do you remove all terminating pods in Kubernetes?

First, confirm the name of the node you want to remove using kubectl get nodes , and make sure that all of the pods on the node can be safely terminated without any special procedures. Next, use the kubectl drain command to evict all user pods from the node.

How do you force restart a pod?

A pod is the smallest unit in Kubernetes (K8S). They should run until they are replaced by a new deployment. Because of this, there is no way to restart a pod, instead, it should be replaced.


2 Answers

Try removing the finalizers from the pod:

kubectl patch pod funny-turtle-myservice-xxx-yyy -p '{"metadata":{"finalizers":null}}'
like image 182
jaxxstorm Avatar answered Oct 18 '22 23:10

jaxxstorm


In my case, the solution proposed by the accepted answer did not work, it kept stuck in "Terminating" status. What did the trick for me was:

kubectl delete pods <pod> --grace-period=0 --force
like image 21
João Matos Avatar answered Oct 18 '22 23:10

João Matos