Pod not terminating

Question

We have a Rancher Kubernetes cluster where sometimes the pods get stuck in terminating status when we try to delete the corresponding deployment, as shown below.

$  kubectl get deployments
NAME                              DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
...
storage-manager-deployment        1         0         0            0           1d
...
$ kubectl delete deployments storage-manager-deployment
kubectl delete deployments storage-manager-deployment
deployment.extensions "storage-manager-deployment" deleted
C-c C-c^C
$ kubectl get po 
NAME                                             READY     STATUS   RESTARTS   AGE
...
storage-manager-deployment-6d56967cdd-7bgv5      0/1   Terminating      0      23h
...
$ kubectl delete pods storage-manager-deployment-6d56967cdd-7bgv5 --grace-period=0 --force
warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.
pod "storage-manager-deployment-6d56967cdd-7bgv5" force deleted
C-c C-c^C  

Both the delete commands (for the deployment and the pod) get stuck and need to be stopped manually.

We have tried both

kubectl delete pod NAME --grace-period=0 --force

and

kubectl delete pod NAME --now

without any luck.

We have also set fs.may_detach_mounts=1, so it seems that all the similar questions already on StackOverflow don't apply to our problem.

If we check the node on which the incriminated pod runs, it does not appear in the docker ps list.

Any suggestion?

Thanks

Answer 1

Check the pod spec for an array: 'finalizers'

finalizers:
- cattle-system

If this exists, remove it, and the pod will terminate.