State of PV/PVC after Pod is Deleted in Kubernetes

Question

I have a Kubernetes cluster with some pods deployed (DB, Frontend, Redis). A part that I can't fully grasp is what happens to the PVC after the pod is deleted.

For example, if I delete POD_A which is bound to CLAIM_A I know that CLAIM_A is not deleted automatically. If I then try to recreate the POD, it is attached back to the same PVC but the all the data is missing.

Can anyone explain what happens, I've looked at the official documentation but not making any sense at the moment.

Any help is appreciated.

Answer 1

PVCs have a lifetime independent of pods. If PV still exists it may be because it has ReclaimPolicy set to Retain in which case it won't be deleted even if PVC is gone.

PersistentVolumes can have various reclaim policies, including “Retain”, “Recycle”, and “Delete”. For dynamically provisioned PersistentVolumes, the default reclaim policy is “Delete”. This means that a dynamically provisioned volume is automatically deleted when a user deletes the corresponding PersistentVolumeClaim. This automatic behavior might be inappropriate if the volume contains precious data. Notice that the RECLAIM POLICY is Delete (default value), which is one of the two reclaim policies, the other one is Retain. (A third policy Recycle has been deprecated). In case of Delete, the PV is deleted automatically when the PVC is removed, and the data on the PVC will also be lost.

In that case, it is more appropriate to use the “Retain” policy. With the “Retain” policy, if a user deletes a PersistentVolumeClaim, the corresponding PersistentVolume is not be deleted. Instead, it is moved to the Released phase, where all of its data can be manually recovered.

This may also happens too when persistent volume is protected. You should be able to cross verify this:

Command:

$ kubectl describe pvc PVC_NAME | grep Finalizers

Output:

Finalizers:    [kubernetes.io/pvc-protection]

You can fix this by setting finalizers to null using kubectl patch:

$ kubectl patch pvc PVC_NAME -p '{"metadata":{"finalizers": []}}' --type=merge

EDIT:

A PersistentVolume can be mounted on a host in any way supported by the resource provider. Each PV gets its own set of access modes describing that specific PV’s capabilities.

The access modes are:

• ReadWriteOnce – the volume can be mounted as read-write by a single node
• ReadOnlyMany – the volume can be mounted read-only by many nodes
• ReadWriteMany – the volume can be mounted as read-write by many nodes

In the CLI, the access modes are abbreviated to:

• RWO - ReadWriteOnce
• ROX - ReadOnlyMany
• RWX - ReadWriteMany

So if you recreated pod and scheduler put it on different node and your PV has reclaim policy set to ReadWriteOnce it is normal that you cannot access your data.

Claims use the same conventions as volumes when requesting storage with specific access modes. My advice is to edit PV access mode to ReadWriteMany.

$ kubectl edit pv your_pv

You should be updating the access mode in PersistentVolume as shown below

   accessModes:
      - ReadWriteMany