Persistent Volume creation via kubectl stuck at Pending states

Question

I am trying to create pv and pvc resources via kuberctl create -f pv-definition.json but k8s doesn't make any progress and it just kept reporting Pending states for them.

Looked at kubectl get events and logs from api and controller, but I don't see anything related to messages to pv or pvc creation. I have the disk created beforehand on gcloud and I have verified that the name and disk size matches.

The pv definition is very basic:

{
  "apiVersion": "v1",
  "kind": "PersistentVolume",
  "metadata": {
    "name": "test-0b-pv"
  },
  "spec": {
    "accessModes": [
      "ReadWriteOnce"
    ],
    "capacity": {
      "storage": "50Gi"
    },
    "gcePersistentDisk": {
      "fsType": "ext4",
      "pdName": "test-0b"
    },
    "persistentVolumeReclaimPolicy": "Retain"
  }
}

Does anyone have any suggestion for debugging? I have restarted k8s-master services to no avail.

Answer 1

Most of the above answers wrote a solution from perspective of inconsistent or incorrect storage class especially in case it was not provided in PersistentVolumeClaims however we may have different reason here (described below)

Problem Statement:

If you have deleted PersistentVolumeClaim and then re-create it again with the same definition, it will be Pending forever, why?

Explanation:

• we have "persistentVolumeReclaimPolicy": "Retain" in PersistentVolume (the default is retain policy)
• In case we have deleted PersistentVolumeClaim, the PersistentVolume still exists and the volume is considered released. But it is not yet available for another claim because the previous claimant's data remains on the volume.
• so you need to manually reclaim the volume with the following steps:

Solution:

1. Delete the PersistentVolume (associated underlying storage asset/resource like EBS, GCE PD, Azure Disk, ...etc will NOT be deleted, still exists)
2. (Optional) Manually clean up the data on the associated storage asset accordingly
3. (Optional) Manually delete the associated storage asset (EBS, GCE PD, Azure Disk, ...etc)

Please note that if you still need same data, simply create a new PersistentVolume with same storage asset definition then you should be good to create PersistentVolumeClaim again.

One last thing to mention, Retain is not the only option for persistentVolumeReclaimPolicy, here are some other options that you may need to use or try based on use-case scenarios:

Recycle: performs a basic scrub on the volume (e.g., rm -rf //*) - makes it available again for a new claim. Only NFS and HostPath support recycling.

Delete: Associated storage asset such as AWS EBS, GCE PD, Azure Disk, or OpenStack Cinder...etc volume is deleted

For more information, please check kubernetes documentation

Answer 2

I had the same issue and solved it by adding storageClassName: value explicitly to both PV and PVC. For the value, I used standard.

In this case, the problem might be caused by DefaultStorageClass configuration. For in-depth details, please see: https://kubernetes.io/docs/concepts/storage/persistent-volumes/#class-1

Answer 3

I found this confusing too... I was deploying a redis server using helm. I created a PV, but redis PVC wouldn't take it. I was like... whaaaat??

then I found this in the docs..

PVCs don’t necessarily have to request a class. A PVC with its storageClassName set equal to "" is always interpreted to be requesting a PV with no class, so it can only be bound to PVs with no class (no annotation or one set equal to "").

Ahh... So not expressing a preference, is expressing a preference for no preference. (Yeah, that's like, not confusing at all).

So i created a PV like using a file called pv-volume.yaml

kind: PersistentVolume
apiVersion: v1
metadata:
  name: task-pv-volume
  labels:
    type: local
spec:
  storageClassName:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/mnt/data"

then

kubectl apply -f pv-volume.yaml

And viola! The redis pod took it.

Answer 4

You should check whether there is a storageclass set up for your cluster:

kubectl get sc

Since, you are not specifying any storageClassName here, it will try to get the default one. If the default one doesn't exist or is not gce but something else then it will remain in pending state.

You will need to first register a storageclass for gce and than it should work. You can create storage class for gce like this:

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: gce
provisioner: kubernetes.io/gce-pd
parameters:
  type: pd-standard
  replication-type: none

and then you will need to add storageClassName in PVC to make it work