Kubernetes Namespaces stuck in Terminating status

Question

We have one cluster where it seems that namespaces never want to be deleted completely and now can't re-create custom-metrics namespace to be able to collect custom metrics to properly setup HPA. I fully understand that I can create another namespace with all custom-metrics resources, but a little concerned with the overall health of the cluster, given that the namespaces get stuck in "Terminating" state

$ kubectl get ns
NAME             STATUS        AGE
cert-manager     Active        14d
custom-metrics   Terminating   7d
default          Active        222d
nfs-share        Active        15d
ingress-nginx    Active        103d
kube-public      Active        222d
kube-system      Active        222d
lb               Terminating   4d
monitoring       Terminating   6d
production       Active        221d

I already tried to export namespaces to JSON, delete finalizers and re-create using edited JSON files. also tried to kubectl edit ns custom-metrics and delete "- kubernetes" finalizer. all to no avail.

does anyone have any other recommendations on how else I can try to destroy these "stuck" namespaces"

curl to https://master-ip/api/v1/namespace/...../finalize doesn't seem to work on Google Kubernetes Engine for me, I'm assuming these operations are not allowed on GKE cluster

Trying things like doesn't work as well:

$ kubectl delete ns custom-metrics --grace-period=0 --force

warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (Conflict): Operation cannot be fulfilled on namespaces "custom-metrics": The system is ensuring all content is removed from this namespace. Upon completion, this namespace will automatically be purged by the system.

and there're no resources listed in this namespaces at all: kubectl get all -n custom-metrics or looping through all api-resources in this namespace shows no resources exist at all: kubectl api-resources --namespaced=true -o name | xargs -n 1 kubectl get -n custom-metrics

Answer 1

I did something similar to rahul.tripathi except the curl did not work for me - I followed https://medium.com/@craignewtondev/how-to-fix-kubernetes-namespace-deleting-stuck-in-terminating-state-5ed75792647e which does the following:

NAMESPACE=
kubectl get namespace $NAMESPACE -o json > $NAMESPACE.json
sed -i -e 's/"kubernetes"//' $NAMESPACE.json
kubectl replace --raw "/api/v1/namespaces/$NAMESPACE/finalize" -f ./$NAMESPACE.json

Voila! Namespace is deleted

Update: One-liner version of this solution (requires jq)

NAMESPACE= ; kubectl get namespace $NAMESPACE -o json | jq 'del(.spec.finalizers[0])' | kubectl replace --raw "/api/v1/namespaces/$NAMESPACE/finalize" -f -

Update #2: Terraform version

resource "kubernetes_namespace" "this" {
  for_each = toset( var.namespaces )
  metadata {
    name = each.key
  }
  provisioner "local-exec" {
    when = destroy
    command    = "nohup ${path.module}/namespace-finalizer.sh ${each.key} 2>&1 &"
  }
}

namespace-finalizer.sh

sleep 30; kubectl get namespace $1 && kubectl get namespace $1 -o json | jq 'del(.spec.finalizers[0])' | kubectl replace --raw "/api/v1/namespaces/$1/finalize" -f -

Answer 2

Looks like this is a known issue with people having mixed results trying a mix of different things:

• Bounce the kube-controller-manager
• Bounce all the kubelets
• Bounce the whole cluster
• kubectl delete ns <name> --grace-period=0 --force
• Patching finalizers "null" everywhere.

Some more background but at the pod level here too.

Answer 3

The only solution that worked for me was:

1. kubectl get namespace annoying-namespace-to-delete -o json > tmp.json

2. edit tmp.json and remove"kubernetes" from "spec": { "finalizers":[]}

3. curl -k -H "Content-Type: application/json" -X PUT --data-binary @tmp.json https://kubernetes-cluster-ip/api/v1/namespaces/annoying-namespace-to-delete/finalize

and this should delete your namespace,