Kubernetes - pod has unbound immediate PersistentVolumeClaims

Question

I'm using mysql Kubernetes statefulset, i mapped PVs to host directory (CentOS 8 VM) but getting " pod has unbound immediate PersistentVolumeClaims"

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mysql-container
spec:
  serviceName: mysql
  replicas: 1 
  selector:
    matchLabels:
      app: mysql-container
  template:
    metadata:
      labels:
        app: mysql-container
    spec:
      containers:
      - name: mysql-container
        image: mysql:dev
        imagePullPolicy: "IfNotPresent"
        envFrom:
          - secretRef:
             name: prod-secrets
        ports:
        - containerPort: 3306
         # container (pod) path
        volumeMounts:
          - name: mysql-persistent-storage
            mountPath: /var/lib/mysql


      volumes:
        - name: mysql-persistent-storage
          persistentVolumeClaim:
           claimName: mysql-pvc
 volumeClaimTemplates:
    - metadata:
        name: data
      spec:

        storageClassName: localstorage
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 3Gi
        selector:
           matchLabels:
            type: local

Storage class is defaulr and no events in PV

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:

  name: localstorage

provisioner: kubernetes.io/no-provisioner
volumeBindingMode: Immediate
reclaimPolicy: Delete
allowVolumeExpansion: True


kind: PersistentVolume
apiVersion: v1
metadata:
  name: mysql-01
  labels:
    type: local
spec:
  storageClassName: localstorage
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/mnt/mysql01"

---
kind: PersistentVolume
apiVersion: v1
metadata:
  name: mysql-02
  labels:
    type: local
spec:
  storageClassName: localstorage
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/mnt/mysql02"

Storage class is default one

get sc
NAME                     PROVISIONER                    RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
localstorage (default)   kubernetes.io/no-provisioner   Delete          Immediate           true                   35m

PVC also shows no events:

    Name:          data-mysql-0
    Namespace:     default
    StorageClass:  localstorage
    Status:        Pending
    Volume:        mysql-storage
    Labels:        app=mysql
    Annotations:   <none>
    Finalizers:    [kubernetes.io/pvc-protection]
    Capacity:      0
    Access Modes:
    VolumeMode:    Filesystem
    Mounted By:    mysql-0
    Events:        <none>


Name:            mysql-01
Labels:          type=local
Annotations:     kubectl.kubernetes.io/last-applied-configuration:
                   {"apiVersion":"v1","kind":"PersistentVolume","metadata":{"annotations":{},"labels":{"type":"local"},"name":"mysql-01"},"spec":{"accessMode...
Finalizers:      [kubernetes.io/pv-protection]
StorageClass:    localstorage
Status:          Available
Claim:
Reclaim Policy:  Retain
Access Modes:    RWO
VolumeMode:      Filesystem
Capacity:        10Gi
Node Affinity:   <none>
Message:
Source:
    Type:          HostPath (bare host directory volume)
    Path:          /mnt/mysql01
    HostPathType:
Events:            <none>


Name:            mysql-02
Labels:          type=local
Annotations:     kubectl.kubernetes.io/last-applied-configuration:
                   {"apiVersion":"v1","kind":"PersistentVolume","metadata":{"annotations":{},"labels":{"type":"local"},"name":"mysql-02"},"spec":{"accessMode...
Finalizers:      [kubernetes.io/pv-protection]
StorageClass:    localstorage
Status:          Available
Claim:
Reclaim Policy:  Retain
Access Modes:    RWO
VolumeMode:      Filesystem
Capacity:        10Gi
Node Affinity:   <none>
Message:
Source:
    Type:          HostPath (bare host directory volume)
    Path:          /mnt/mysql02
    HostPathType:
Events:            <none>

Pod is in pending state:

>       Events:
>          Type     Reason            Age                From               Message
>          ----     ------            ----               ----               -------
>          Warning  FailedScheduling  27s (x2 over 27s)  default-scheduler  error while running >"VolumeBinding" filter plugin for pod "mysql-0": pod has unbound immediate PersistentVolumeClaims

Can someone point out what else should be done here, thanks

Answer 1

PersistentVolumeClaims will remain unbound indefinitely if a matching PersistentVolume does not exist. The PersistentVolume is matched with accessModes and capacity. In this case capacity the PV is 10Gi whereas PVC has capacity of 3Gi.

The capacity in the PV needs to same as in the claim i.e 3Gi to fix the unbound immediate PersistentVolumeClaims issue.

Answer 2

The mentioned error can be caused for multiple reasons - below are few options which I encountered.

---

Example 1

The persistentvolume-controller has failed to find a PV with a capacity size which is equal or higher then the value that was specified in the PVC.

So if we take this example:

  # PVC
  resources:
     requests:
       storage: 3Gi


  # PV
  capacity:
    storage: 10Gi 

So:
If PV capacity >= PVC capacity then PVC should be bound to PV.

if Not then we'll get the unbound immediate PersistentVolumeClaims error in the pod level and no volume plugin matched name when describing the PVC.

---

Example 2

The number of PVC is higher then the PV.

For example if only one PV is created (or the others were deleted):

$ kubectl get pv
NAME             CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                      STORAGECLASS    REASON   AGE
mongo-local-pv   50Gi       RWO            Retain           Bound    default/mongo-persistent-storage-mongo-0   local-storage            106m

We could see that some workloads (Pods or Stateful sets) will be stuck on pending:

$ kubectl get pods
NAME      READY   STATUS    RESTARTS   AGE
mongo-0   2/2     Running   0          3m38s
mongo-1   0/2     Pending   0          3m23s

$ kubectl get pvc
NAME                               STATUS    VOLUME           CAPACITY   ACCESS MODES   STORAGECLASS    AGE
mongo-persistent-storage-mongo-0   Bound     mongo-local-pv   50Gi       RWO            local-storage   80m
mongo-persistent-storage-mongo-1   Pending                                              local-storage   45m

We'll get the mentioned error on the pending resources.

---

Example 3

If the scheduler failed to match a node to the PV.

When using local volumes the nodeAffinity of the PV is required and should be a value of an existing node in the cluster:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: local-mongo-pv
  .
  .
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - node-which-doesnt-exists  # <----- Will lead to the error

---

Example 4

Old PVs with the same name and different configuration were already exist on the cluster and the new PVC is created according to them.

When working with local volume the administrator must perform manually clean up and set up the local volume again each time for reuse.

(*) This local static provisioner was created to help with the PV lifecycle.