Cancel or undo deletion of Persistent Volumes in kubernetes cluster

Question

Accidentally tried to delete all PV's in cluster but thankfully they still have PVC's that are bound to them so all PV's are stuck in Status: Terminating.

How can I get the PV's out of the "terminating" status and back to a healthy state where it is "bound" to the pvc and is fully working?

The key here is that I don't want to lose any data and I want to make sure the volumes are functional and not at risk of being terminated if claim goes away.

Here are some details from a kubectl describe on the PV.

$ kubectl describe pv persistent-vol-1
Finalizers:      [kubernetes.io/pv-protection foregroundDeletion]
Status:          Terminating (lasts 1h)
Claim:           ns/application
Reclaim Policy:  Delete

Here is the describe on the claim.

$ kubectl describe pvc application
Name:          application
Namespace:     ns
StorageClass:  standard
Status:        Bound
Volume:        persistent-vol-1

Answer 1

It is, in fact, possible to save data from your PersistentVolume with Status: Terminating and RetainPolicy set to default (delete). We have done so on GKE, not sure about AWS or Azure but I guess that they are similar

We had the same problem and I will post our solution here in case somebody else has an issue like this.

Your PersistenVolumes will not be terminated until there is a pod, deployment or to be more specific - a PersistentVolumeClaim using it.

The steps we took to remedy our broken state:

Once you are in the situation lke the OP, the first thing you want to do is to create a snapshot of your PersistentVolumes.

In GKE console, go to Compute Engine -> Disks and find your volume there (use kubectl get pv | grep pvc-name) and create a snapshot of your volume.

Use the snapshot to create a disk: gcloud compute disks create name-of-disk --size=10 --source-snapshot=name-of-snapshot --type=pd-standard --zone=your-zone

At this point, stop the services using the volume and delete the volume and volume claim.

Recreate the volume manually with the data from the disk:

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: name-of-pv
spec:
  accessModes:
    - ReadWriteOnce
  capacity:
    storage: 10Gi
  gcePersistentDisk:
    fsType: ext4
    pdName: name-of-disk
  persistentVolumeReclaimPolicy: Retain

Now just update your volume claim to target a specific volume, the last line of the yaml file:

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-pvc
  namespace: my-namespace
  labels:
    app: my-app
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  volumeName: name-of-pv

Answer 2

Edit: This only applies if you deleted the PVC and not the PV. Do not follow these instructions if you deleted the PV itself or the disk may be deleted!

I found myself in this same situation due to a careless mistake. It was with a statefulset on Google Cloud/GKE. My PVC said terminating because the pod referencing it was still running and the PV was configured with a retain policy of Deleted. I ended up finding a simpler method to get everything straightened out that also preserved all of the extra Google/Kubernetes metadata and names.

First, I would make a snapshot of your disk as suggested by another answer. You won't need it, but if something goes wrong, the other answer here can then be used to re-create a disk from it.

The short version is that you just need reconfigure the PV to "Retain", allow the PVC to get deleted, then remove the previous claim from the PV. A new PVC can then be bound to it and all is well.

Details:

1. Find the full name of the PV:

    kubectl get pv

2. Reconfigure your PV to set the reclaim policy to "Retain": (I'm doing this on Windows so you may need to handle the quotes differently depending on OS)

    kubectl patch pv <your-pv-name-goes-here> -p "{\"spec\":{\"persistentVolumeReclaimPolicy\":\"Retain\"}}"

3. Verify that the status of the PV is now Retain.
4. Shutdown your pod/statefulset (and don't allow it to restart). Once that's finished, your PVC will get removed and the PV (and the disk it references!) will be left intact.
5. Edit the PV:

    kubectl edit pv <your-pv-name-goes-here>

6. In the editor, remove the entire "claimRef" section. Remove all of the lines from (and including) "claimRef:" until the next tag with the same indentation level. The lines to remove should look more or less like this:

      claimRef:
        apiVersion: v1
        kind: PersistentVolumeClaim
        name: my-app-pvc-my-app-0
        namespace: default
        resourceVersion: "1234567"
        uid: 12345678-1234-1234-1234-1234567890ab

7. Save the changes and close the editor. Check the status of the PV and it should now show "Available".
8. Now you can re-create your PVC exactly as you originally did. That should then find the now "Available" PV and bind itself to it. In my case, I have the PVC defined with my statefulset as a volumeClaimTemplate so all I had to do was "kubectl apply" my statefulset.

Answer 3

You can check out this tool, it will update the Terminating PV's status in etcd back to Bound.

The way it works has been mentioned by Anirudh Ramanathan in his answer.

Be sure to back up your PV first.