WaitForFirstConsumer PersistentVolumeClaim waiting for first consumer to be created before binding

Question

I setup a new k8s in a single node, which is tainted. But the PersistentVolume can not be created successfully, when I am trying to create a simple PostgreSQL.

There is some detail information below.

---

The StorageClass is copied from the official page: https://kubernetes.io/docs/concepts/storage/storage-classes/#local

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: local-storage
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer

The StatefulSet is:

kind: StatefulSet
apiVersion: apps/v1beta1
metadata:
  name: postgres
spec:
  serviceName: postgres
  replicas: 1
  ...
  volumeClaimTemplates:
    - metadata:
        name: postgres-data
      spec:
        storageClassName: local-storage
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 1Gi

About the running StorageClass:

$ kubectl describe storageclasses.storage.k8s.io
Name:            local-storage
IsDefaultClass:  No
Annotations:     kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{},"name":"local-storage"},"provisioner":"kubernetes.io/no-provisioner","volumeBindingMode":"WaitForFirstConsumer"}

Provisioner:           kubernetes.io/no-provisioner
Parameters:            <none>
AllowVolumeExpansion:  <unset>
MountOptions:          <none>
ReclaimPolicy:         Delete
VolumeBindingMode:     WaitForFirstConsumer
Events:                <none>

About the running PersistentVolumeClaim:

$ kubectl describe pvc
Name:          postgres-data-postgres-0
Namespace:     default
StorageClass:  local-storage
Status:        Pending
Volume:
Labels:        app=postgres
Annotations:   <none>
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode:    Filesystem
Events:
  Type       Reason                Age                            From                         Message
  ----       ------                ----                           ----                         -------
  Normal     WaitForFirstConsumer  <invalid> (x2 over <invalid>)  persistentvolume-controller  waiting for first consumer to be created before binding

K8s versions:

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.4", GitCommit:"c27b913fddd1a6c480c229191a087698aa92f0b1", GitTreeState:"clean", BuildDate:"2019-02-28T13:37:52Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.1", GitCommit:"eec55b9ba98609a46fee712359c7b5b365bdd920", GitTreeState:"clean", BuildDate:"2018-12-13T10:31:33Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"linux/amd64"}

Answer 1

The app is waiting for the Pod, while the Pod is waiting for a PersistentVolume by a PersistentVolumeClaim. However, the PersistentVolume should be prepared by the user before using.

My previous YAMLs are lack of a PersistentVolume like this:

kind: PersistentVolume
apiVersion: v1
metadata:
  name: postgres-data
  labels:
    type: local
spec:
  storageClassName: local-storage
  capacity:
    storage: 1Gi
  local:
    path: /data/postgres
  persistentVolumeReclaimPolicy: Retain
  accessModes:
    - ReadWriteOnce
  storageClassName: local-storage
  nodeAffinity:
    required:
      nodeSelectorTerms:
        - matchExpressions:
          - key: app
            operator: In
            values:
              - postgres

The local path /data/postgres should be prepared before using. Kubernetes will not create it automatically.

Answer 2

I just ran into this myself and was completely thrown for a loop until I realized that the StorageClass's VolumeBindingMode was set to WaitForFirstConsumer vice my intended value of Immediate. This value is immutable so you will have to:

1. Get the storage class yaml:

   kubectl get storageclasses.storage.k8s.io gp2 -o yaml > gp2.yaml
   

   or you can also just copy the example from the docs here (make sure the metadata names match). Here is what I have configured:

   apiVersion: storage.k8s.io/v1
   kind: StorageClass
   metadata:
     name: gp2
   provisioner: kubernetes.io/aws-ebs
   parameters:
     type: gp2
   reclaimPolicy: Delete
   allowVolumeExpansion: true
   mountOptions:
     - debug
   volumeBindingMode: Immediate
   

2. And delete the old StorageClass before recreating it with the new volumeBindingMode set to Immediate.

Note: The EKS clsuter may need perms to create cloud resources like EBS or EFS. Assuming EBS you should be good with arn:aws:iam::aws:policy/AmazonEKSClusterPolicy.

After doing this you should have no problem creating and using dynamically provisioned PVs.

Answer 3

For me the problem was mismatched accessModes fields in the PV and PVC. PVC was requesting RWX/ReadWriteMany while PV was offering RWO/ReadWriteOnce.

Answer 4

The accepted answer didn't work for me. I think it's because the app key won't be set before the the StatefulSet's Pods are deployed, preventing the PersistentVolumeClaim to match the nodeSelector (preventing the Pods to start with the error didn't find available persistent volumes to bind.). To fix this deadlock, I defined one PersistentVolume for each node (this may not be ideal but it worked):

apiVersion: v1
kind: PersistentVolume
metadata:
  name: postgres-data-node1
  labels:
    type: local
spec:
[…]
  nodeAffinity:
    required:
      nodeSelectorTerms:
        - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
              - node1