How do I force delete Namespaces stuck in Terminating?
apiVersion: v1 kind: Namespace metadata: name: delete-me spec: finalizers: - foregroundDeletion
kubectl delete ns delete-me
It is not possible to delete delete-me
.
The only workaround I've found is to destroy and recreate the entire cluster.
None of these work or modify the Namespace. After any of these the problematic finalizer still exists.
kubectl apply
Apply:
apiVersion: v1 kind: Namespace metadata: name: delete-me spec: finalizers:
$ kubectl apply -f tmp.yaml namespace/delete-me configured
The command finishes with no error, but the Namespace is not udpated.
The below YAML has the same result:
apiVersion: v1 kind: Namespace metadata: name: delete-me spec:
kubectl edit
kubectl edit ns delete-me
, and remove the finalizer. Ditto removing the list entirely. Ditto removing spec
. Ditto replacing finalizers
with an empty list.
$ kubectl edit ns delete-me namespace/delete-me edited
This shows no error message but does not update the Namespace. kubectl edit
ing the object again shows the finalizer still there.
kubectl proxy &
kubectl proxy &
curl -k -H "Content-Type: application/yaml" -X PUT --data-binary @tmp.yaml http://127.0.0.1:8001/api/v1/namespaces/delete-me/finalize
As above, this exits successfully but does nothing.
kubectl delete ns delete-me --force --grace-period=0
This actually results in an error:
warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (Conflict): Operation cannot be fulfilled on namespaces "delete-me": The system is ensuring all content is removed from this namespace. Upon completion, this namespace will automatically be purged by the system.
However, it doesn't actually do anything.
In the test cluster I set up to debug this issue, I've been waiting over a week. Even if the Namespace might eventually decide to be deleted, I need it to be deleted faster than a week.
The Namespace is empty.
$ kubectl get -n delete-me all No resources found.
etcdctl
$ etcdctl --endpoint=http://127.0.0.1:8001 rm /namespaces/delete-me Error: 0: () [0]
I'm pretty sure that's an error, but I have no idea how to interpret that. It also doesn't work. Also tried with --dir
and -r
.
ctron/kill-kube-ns
There is a script for force deleting Namespaces. This also does not work.
$ ./kill-kube-ns delete-me Killed namespace: delete-me $ kubectl get ns delete-me NAME STATUS AGE delete-me Terminating 1h
POST
ing the edited resource to /finalizeReturns a 405. I'm not sure if this is the canonical way to POST to /finalize though.
This appears to be a recurring problem and none of these resources helped.
Kubernetes bug
Problem: Deleting a namespace To do so, click on the three dots shown at the right of the namespace and select Delete. Check after a few minutes (or days, months, years). It still shows up as terminating.
Resolving the problem If the issue is not resolved, you can manually delete your namespace that is stuck in the Terminating state. Edit your tmp. json file. Remove the kubernetes value from the finalizers field and save the file.
Determine the namespace for a node by selecting any object/node that uses the namespace. In the Namespace Editor, select the namespace you want to remove. Click Delete.
The kubectl proxy
try is almost correct, but not quite. It's possible using JSON instead of YAML does the trick, but I'm not certain.
The JSON with an empty finalizers list:
~$ cat ns.json { "kind": "Namespace", "apiVersion": "v1", "metadata": { "name": "delete-me" }, "spec": { "finalizers": [] } }
Use curl
to PUT
the object without the problematic finalizer.
~$ curl -k -H "Content-Type: application/json" -X PUT --data-binary @ns.json http://127.0.0.1:8007/api/v1/namespaces/delete-me/finalize { "kind": "Namespace", "apiVersion": "v1", "metadata": { "name": "delete-me", "selfLink": "/api/v1/namespaces/delete-me/finalize", "uid": "0df02f91-6782-11e9-8beb-42010a800137", "resourceVersion": "39047", "creationTimestamp": "2019-04-25T17:46:28Z", "deletionTimestamp": "2019-04-25T17:46:31Z", "annotations": { "kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"v1\",\"kind\":\"Namespace\",\"metadata\":{\"annotations\":{},\"name\":\"delete-me\"},\"spec\":{\"finalizers\":[\"foregroundDeletion\"]}}\n" } }, "spec": { }, "status": { "phase": "Terminating" } }
The Namespace is deleted!
~$ kubectl get ns delete-me Error from server (NotFound): namespaces "delete-me" not found
I loved this answer extracted from here
In one terminal:
kubectl proxy
In another terminal:
kubectl get ns delete-me -o json | \ jq '.spec.finalizers=[]' | \ curl -X PUT http://localhost:8001/api/v1/namespaces/delete-me/finalize -H "Content-Type: application/json" --data @-
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With