I have a kubernetes pod stuck in "Terminating" state that resists pod deletions
NAME READY STATUS RESTARTS AGE
...
funny-turtle-myservice-xxx-yyy 1/1 Terminating 1 11d
...
Where funny-turtle
is the name of the helm release that have since been deleted.
Output: pod "funny-turtle-myservice-xxx-yyy" deleted
Outcome: it still shows up in the same state.
- also tried with --force --grace-period=0
, same outcome with extra warning
warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.
Outcome: Error from server (NotFound): nodes "ip-xxx.yyy.compute.internal" not found
but it does not exist.
So I assume this pod somehow got "disconnected" from the aws API, reasoning from the error message that kubectl logs
printed.
I'll take any suggestions or guidance to explain what happened here and how I can get rid of it.
Tried to see if the "ghost" node was still there (kubectl delete node ip-xxx.yyy.compute.internal
) but it does not exist.
A pod is stuck in a terminating state as the configmap mounted as a volume fails to unmount when trying to clean the subPath mount for it.
First, confirm the name of the node you want to remove using kubectl get nodes , and make sure that all of the pods on the node can be safely terminated without any special procedures. Next, use the kubectl drain command to evict all user pods from the node.
A pod is the smallest unit in Kubernetes (K8S). They should run until they are replaced by a new deployment. Because of this, there is no way to restart a pod, instead, it should be replaced.
Try removing the finalizers from the pod:
kubectl patch pod funny-turtle-myservice-xxx-yyy -p '{"metadata":{"finalizers":null}}'
In my case, the solution proposed by the accepted answer did not work, it kept stuck in "Terminating" status. What did the trick for me was:
kubectl delete pods <pod> --grace-period=0 --force
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With