Running MongoDB on Kubernetes Minikube with local persistent storage

Question

I am currently trying to reproduce this tutorial on Minikube:

http://blog.kubernetes.io/2017/01/running-mongodb-on-kubernetes-with-statefulsets.html

I updated the configuration files to use a hostpath as a persistent storage on minikube node.

kind: PersistentVolume
apiVersion: v1
metadata:
  name: pv0001
  labels:
    type: local
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/tmp"

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: myclaim
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi

---
apiVersion: v1
kind: Service
metadata:
  name: mongo
  labels:
    name: mongo
spec:
  ports:
  - port: 27017
    targetPort: 27017
  clusterIP: None
  selector:
    role: mongo
---
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: mongo
spec:
  serviceName: "mongo"
  replicas: 3
  template:
    metadata:
      labels:
        role: mongo
        environment: test
    spec:
      terminationGracePeriodSeconds: 10
      containers:
        - name: mongo
          image: mongo
          command:
            - mongod
            - "--replSet"
            - rs0
            - "--smallfiles"
            - "--noprealloc"
          ports:
            - containerPort: 27017
          volumeMounts:
            - name: myclaim
              mountPath: /data/db
        - name: mongo-sidecar
          image: cvallance/mongo-k8s-sidecar
          env:
            - name: MONGO_SIDECAR_POD_LABELS
              value: "role=mongo,environment=test"
  volumeClaimTemplates:
    - metadata:
        name: myclaim

Which result in the following:

kubectl get pv
NAME                                       CAPACITY   ACCESSMODES   RECLAIMPOLICY   STATUS      CLAIM             REASON    AGE
pv0001                                     1Gi        RWO           Retain          Available                               17s
pvc-134a6c0f-1565-11e7-9cf1-080027f4d8c3   1Gi        RWO           Delete          Bound       default/myclaim             11s

kubectl get pvc
NAME      STATUS    VOLUME                                     CAPACITY   ACCESSMODES   AGE
myclaim   Bound     pvc-134a6c0f-1565-11e7-9cf1-080027f4d8c3   1Gi        RWO           14s

kubectl get svc
NAME         CLUSTER-IP   EXTERNAL-IP   PORT(S)     AGE
kubernetes   10.0.0.1     <none>        443/TCP     3d
mongo        None         <none>        27017/TCP   53s

kubectl get pod
No resources found.


kubectl describe service mongo
Name:           mongo
Namespace:      default
Labels:         name=mongo
Selector:       role=mongo
Type:           ClusterIP
IP:         None
Port:           <unset> 27017/TCP
Endpoints:      <none>
Session Affinity:   None
No events.


kubectl get statefulsets
NAME      DESIRED   CURRENT   AGE
mongo     3         0         4h


kubectl describe statefulsets mongo
Name:           mongo
Namespace:      default
Image(s):       mongo,cvallance/mongo-k8s-sidecar
Selector:       environment=test,role=mongo
Labels:         environment=test,role=mongo
Replicas:       0 current / 3 desired
Annotations:        <none>
CreationTimestamp:  Thu, 30 Mar 2017 18:23:56 +0200
Pods Status:        0 Running / 0 Waiting / 0 Succeeded / 0 Failed
No volumes.
Events:
  FirstSeen LastSeen    Count   From        SubObjectPath   Type    Reason      Message
  --------- --------    -----   ----        -------------   --------------      -------
  1s        1s      4   {statefulset }          WarningFailedCreate pvc: myclaim-mongo-0, error: PersistentVolumeClaim "myclaim-mongo-0" is invalid: [spec.accessModes: Required value: at least 1 access mode is required, spec.resources[storage]: Required value]
  1s        1s      4   {statefulset }          WarningFailedCreate pvc: myclaim-mongo-1, error: PersistentVolumeClaim "myclaim-mongo-1" is invalid: [spec.accessModes: Required value: at least 1 access mode is required, spec.resources[storage]: Required value]
  1s        0s      4   {statefulset }          WarningFailedCreate pvc: myclaim-mongo-2, error: PersistentVolumeClaim "myclaim-mongo-2" is invalid: [spec.accessModes: Required value: at least 1 access mode is required, spec.resources[storage]: Required value]


kubectl get ev | grep mongo
29s        1m          15        mongo      StatefulSet               Warning   FailedCreate              {statefulset }          pvc: myclaim-mongo-0, error: PersistentVolumeClaim "myclaim-mongo-0" is invalid: [spec.accessModes: Required value: at least 1 access mode is required, spec.resources[storage]: Required value]
29s        1m          15        mongo      StatefulSet               Warning   FailedCreate              {statefulset }          pvc: myclaim-mongo-1, error: PersistentVolumeClaim "myclaim-mongo-1" is invalid: [spec.accessModes: Required value: at least 1 access mode is required, spec.resources[storage]: Required value]
29s        1m          15        mongo      StatefulSet               Warning   FailedCreate              {statefulset }          pvc: myclaim-mongo-2, error: PersistentVolumeClaim "myclaim-mongo-2" is invalid: [spec.accessModes: Required value: at least 1 access mode is required, spec.resources[storage]: Required value]

kubectl describe pvc myclaim
Name:       myclaim
Namespace:  default
StorageClass:   standard
Status:     Bound
Volume:     pvc-134a6c0f-1565-11e7-9cf1-080027f4d8c3
Labels:     <none>
Capacity:   1Gi
Access Modes:   RWO
No events.

minikube version: v0.17.1

It seems that the service is not able to load pods, which makes it complicated to debug with kubectl logs. Is there something wrong with the way I am creating a persistent volume on my node ?

Thanks a lot

Answer 1

TL; DR

In the situation described in the question the problem was that the Pods for the StatefulSet did not start up at all therefore the Service had no targets. The reason for not starting up was:

WarningFailedCreate pvc: myclaim-mongo-0, error: PersistentVolumeClaim "myclaim-mongo-0" is invalid: [spec.accessModes: Required value: at least 1 access mode is required, spec.resources[storage]: Required value]`

And since the volume by default is defined as required the Pod won't start without it. So edit the StatefulSet's volumeClaimTemplate to have:

volumeClaimTemplates:
- metadata:
    name: myclaim
  spec:
    accessModes: [ "ReadWriteOnce" ]
    resources:
      requests:
        storage: 1Gi

(There is no need to create the PersistentVolumeClaim manually.)

More general solution

If can't connect a Service try this command:

kubectl describe service myservicename

And if you see something like this in the output:

Endpoints:      <none>

That means there are no targets (usually Pods) running or the targets are not ready. To find out which one is the case do:

kubectl describe endpoint myservicename

It will list all endpoints, ready or not. If not ready, investigate the readinessProbe in the Pod. If doesn't exist then try to find out why by looking at the StatefulSet (Deployment, ReplicaSet, ReplicationController, etc) itself for messages (the Events section):

kubectl describe statefulset mystatefulsetname

This information is available if you do:

kubectl get ev | grep something

If you are sure they are running and ready then the labels on the Pods and the Service do not match up.