Error while using local persistent volumes in statefulset pod

Question

I am trying to use local persistent volume mentioned in https://kubernetes.io/blog/2018/04/13/local-persistent-volumes-beta/ for creating my statefulset pod. But when my pod tries to claim volume. I am getting following error :

Events:
  Type     Reason            Age                 From               Message
  ----     ------            ----                ----               -------
  Warning  FailedScheduling  4s (x243 over 20m)  default-scheduler  0/2 nodes are available: 1 node(s) didn't find available persistent volumes to bind, 1 node(s) had taints that the pod didn't tolerate.

Following are storage classes and persistent volumes I have created:

storageclass-kafka-broker.yml

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: kafka-broker
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer

storageclass-kafka-zookeeper.yml

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: kafka-zookeeper
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer

pv-zookeeper.yml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: example-local-pv-zookeeper
spec:
  capacity:
    storage: 2Gi
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: kafka-zookeeper
  local:
    path: /D/kubernetes-mount-path
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - my-node

pv-kafka.yml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: example-local-pv
spec:
  capacity:
    storage: 200Gi
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: kafka-broker
  local:
    path: /D/kubernetes-mount-path
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - my-node

Following is the pod 50pzoo.yml using this volume :

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: pzoo
  namespace: kafka
spec:
  selector:
    matchLabels:
      app: zookeeper
      storage: persistent
  serviceName: "pzoo"
  replicas: 1
  updateStrategy:
    type: OnDelete
  template:
    metadata:
      labels:
        app: zookeeper
        storage: persistent
      annotations:
    spec:
      terminationGracePeriodSeconds: 10
      initContainers:
      - name: init-config
        image: solsson/kafka-initutils@sha256:18bf01c2c756b550103a99b3c14f741acccea106072cd37155c6d24be4edd6e2
        command: ['/bin/bash', '/etc/kafka-configmap/init.sh']
        volumeMounts:
        - name: configmap
          mountPath: /etc/kafka-configmap
        - name: config
          mountPath: /etc/kafka
        - name: data
          mountPath: /var/lib/zookeeper/data
      containers:
      - name: zookeeper
        image: solsson/kafka:2.0.0@sha256:8bc5ccb5a63fdfb977c1e207292b72b34370d2c9fe023bdc0f8ce0d8e0da1670
        env:
        - name: KAFKA_LOG4J_OPTS
          value: -Dlog4j.configuration=file:/etc/kafka/log4j.properties
        command:
        - ./bin/zookeeper-server-start.sh
        - /etc/kafka/zookeeper.properties
        ports:
        - containerPort: 2181
          name: client
        - containerPort: 2888
          name: peer
        - containerPort: 3888
          name: leader-election
        resources:
          requests:
            cpu: 10m
            memory: 100Mi
        readinessProbe:
          exec:
            command:
            - /bin/sh
            - -c
            - '[ "imok" = "$(echo ruok | nc -w 1 -q 1 127.0.0.1 2181)" ]'
        volumeMounts:
        - name: config
          mountPath: /etc/kafka
        - name: data
          mountPath: /var/lib/zookeeper/data
      volumes:
      - name: configmap
        configMap:
          name: zookeeper-config
      - name: config
        emptyDir: {}
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: kafka-zookeeper
      resources:
        requests:
          storage: 1Gi

Following is the kubectl get events command output

[root@quagga kafka-kubernetes-testing-single-node]# kubectl get events --namespace kafka
LAST SEEN   FIRST SEEN   COUNT     NAME                           KIND                    SUBOBJECT   TYPE      REASON                 SOURCE                        MESSAGE
1m          1m           1         pzoo.15517ca82c7a4675          StatefulSet                         Normal    SuccessfulCreate       statefulset-controller        create Claim data-pzoo-0 Pod pzoo-0 in StatefulSet pzoo success
1m          1m           1         pzoo.15517ca82caed9bc          StatefulSet                         Normal    SuccessfulCreate       statefulset-controller        create Pod pzoo-0 in StatefulSet pzoo successful
13s         1m           9         data-pzoo-0.15517ca82c726833   PersistentVolumeClaim               Normal    WaitForFirstConsumer   persistentvolume-controller   waiting for first consumer to be created before binding
9s          1m           22        pzoo-0.15517ca82cb90238        Pod                                 Warning   FailedScheduling       default-scheduler             0/2 nodes are available: 1 node(s) didn't find available persistent volumes to bind, 1 node(s) had taints that the pod didn't tolerate.

Output of kubectl get pv is :

[root@quagga kafka-kubernetes-testing-single-node]# kubectl get pv
NAME                         CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM     STORAGECLASS      REASON    AGE
example-local-pv             200Gi      RWO            Retain           Available             kafka-broker                4m
example-local-pv-zookeeper   2Gi        RWO            Retain           Available             kafka-zookeeper             4m

Answer 1

it was a silly mistake. I was mentioning my-node in node name values in pv files. Modifying it to correct node name solved my issue.

Answer 2

Thanks for sharing! I did the same mistake. I guess the k8s docu could state this a bit clearer (though it's quite obviuos), so it's a copy paste pitfall.

And to be a bit clearer: if you have a cluster with 3 nodes, then you need to create three different named PVs and provide the correct nodename for 'my-node' (kubectl get nodes). The only reference between your volumeClaimTemplate and your PV is the name of the storage class.

I took something like "local-pv-node-X" as PV name, so when I look at the PV section in the kubernetes dashboard, I can see directly where this volume is located on.

You migh update your listing with the hint on 'my-note ;-)