CrashLoopBackOff (postgres) - GCP

Question

Error in text:

Error from server: Get https://10.128.15.203:10250/containerLogs/default/postgres-54db6bdb8b-cmrsb/postgres: EOF

How could I solve this issue? And what can be reason? I've used this tutorial for configuring all stuff.

kubectl describe pods postgres-54db6bdb8b-cmrsb

Name:           postgres-54db6bdb8b-cmrsb
Namespace:      default
Priority:       0
Node:           gke-booknotes-pool-2-c1d23e62-r6nb/10.128.15.203
Start Time:     Sat, 14 Dec 2019 23:27:20 +0700
Labels:         app=postgres
                pod-template-hash=54db6bdb8b
Annotations:    kubernetes.io/limit-ranger: LimitRanger plugin set: cpu request for container postgres
Status:         Running
IP:             10.56.1.3
IPs:            <none>
Controlled By:  ReplicaSet/postgres-54db6bdb8b
Containers:
  postgres:
    Container ID:   docker://1a607cfb9a8968d708ff79419ec8bfc7233fb5ad29fb1055034ddaacfb793d6a
    Image:          postgres:10.4
    Image ID:       docker-pullable://postgres@sha256:9625c2fb34986a49cbf2f5aa225d8eb07346f89f7312f7c0ea19d82c3829fdaa
    Port:           5432/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       ContainerCannotRun
      Message:      error while creating mount source path '/mnt/data': mkdir /mnt/data: read-only file system
      Exit Code:    128
      Started:      Sat, 14 Dec 2019 23:54:00 +0700
      Finished:     Sat, 14 Dec 2019 23:54:00 +0700
    Ready:          False
    Restart Count:  25
    Requests:
      cpu:  100m
    Environment Variables from:
      postgres-config  ConfigMap  Optional: false
    Environment:       <none>
    Mounts:
      /var/lib/postgresql/data from postgredb (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-t48dw (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  postgredb:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  postgres-pv-claim
    ReadOnly:   false
  default-token-t48dw:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-t48dw
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age                     From                                         Message
  ----     ------     ----                    ----                                         -------
  Normal   Scheduled  32m                     default-scheduler                            Successfully assigned default/postgres-54db6bdb8b-cmrsb to gke-booknotes-pool-2-c1d23e62-r6nb
  Normal   Pulled     28m (x5 over 30m)       kubelet, gke-booknotes-pool-2-c1d23e62-r6nb  Container image "postgres:10.4" already present on machine
  Normal   Created    28m (x5 over 30m)       kubelet, gke-booknotes-pool-2-c1d23e62-r6nb  Created container postgres
  Warning  Failed     28m (x5 over 30m)       kubelet, gke-booknotes-pool-2-c1d23e62-r6nb  Error: failed to start container "postgres": Error response from daemon: error while creating mount source path '/mnt/data': mkdir /mnt/data: read-only file system
  Warning  BackOff    27m (x10 over 29m)      kubelet, gke-booknotes-pool-2-c1d23e62-r6nb  Back-off restarting failed container
  Warning  Failed     23m (x4 over 25m)       kubelet, gke-booknotes-pool-2-c1d23e62-r6nb  Error: failed to start container "postgres": Error response from daemon: error while creating mount source path '/mnt/data': mkdir /mnt/data: read-only file system
  Warning  BackOff    22m (x11 over 25m)      kubelet, gke-booknotes-pool-2-c1d23e62-r6nb  Back-off restarting failed container
  Normal   Pulled     22m (x5 over 25m)       kubelet, gke-booknotes-pool-2-c1d23e62-r6nb  Container image "postgres:10.4" already present on machine
  Normal   Created    22m (x5 over 25m)       kubelet, gke-booknotes-pool-2-c1d23e62-r6nb  Created container postgres
  Normal   Pulled     19m (x4 over 20m)       kubelet, gke-booknotes-pool-2-c1d23e62-r6nb  Container image "postgres:10.4" already present on machine
  Normal   Created    19m (x4 over 20m)       kubelet, gke-booknotes-pool-2-c1d23e62-r6nb  Created container postgres
  Warning  Failed     19m (x4 over 20m)       kubelet, gke-booknotes-pool-2-c1d23e62-r6nb  Error: failed to start container "postgres": Error response from daemon: error while creating mount source path '/mnt/data': mkdir /mnt/data: read-only file system
  Warning  BackOff    18m (x11 over 20m)      kubelet, gke-booknotes-pool-2-c1d23e62-r6nb  Back-off restarting failed container
  Normal   Created    15m (x4 over 17m)       kubelet, gke-booknotes-pool-2-c1d23e62-r6nb  Created container postgres
  Warning  Failed     15m (x4 over 17m)       kubelet, gke-booknotes-pool-2-c1d23e62-r6nb  Error: failed to start container "postgres": Error response from daemon: error while creating mount source path '/mnt/data': mkdir /mnt/data: read-only file system
  Normal   Pulled     14m (x5 over 17m)       kubelet, gke-booknotes-pool-2-c1d23e62-r6nb  Container image "postgres:10.4" already present on machine
  Warning  BackOff    12m (x19 over 17m)      kubelet, gke-booknotes-pool-2-c1d23e62-r6nb  Back-off restarting failed container
  Normal   Pulled     5m38s (x5 over 8m29s)   kubelet, gke-booknotes-pool-2-c1d23e62-r6nb  Container image "postgres:10.4" already present on machine
  Normal   Created    5m38s (x5 over 8m27s)   kubelet, gke-booknotes-pool-2-c1d23e62-r6nb  Created container postgres
  Warning  Failed     5m37s (x5 over 8m24s)   kubelet, gke-booknotes-pool-2-c1d23e62-r6nb  Error: failed to start container "postgres": Error response from daemon: error while creating mount source path '/mnt/data': mkdir /mnt/data: read-only file system
  Warning  BackOff    5m24s (x10 over 7m58s)  kubelet, gke-booknotes-pool-2-c1d23e62-r6nb  Back-off restarting failed container

Here is also my yaml files:

deployment.yaml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: postgres
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
        - name: postgres
          image: postgres:10.4
          imagePullPolicy: "IfNotPresent"
          ports:
            - containerPort: 5432
          envFrom:
            - configMapRef:
                name: postgres-config
          volumeMounts:
            - mountPath: /var/lib/postgresql/data
              name: postgredb
      volumes:
        - name: postgredb
          persistentVolumeClaim:
            claimName: postgres-pv-claim

postgres-configmap.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: postgres-config
  labels:
    app: postgres
data:
  POSTGRES_DB: postgresdb
  POSTGRES_USER: postgresadmin
  POSTGRES_PASSWORD: some_password

postgres-service.yaml

apiVersion: v1
kind: Service
metadata:
  name: postgres
  labels:
    app: postgres
spec:
  type: NodePort
  ports:
   - port: 5432
  selector:
   app: postgres

postgres-storage.yaml

kind: PersistentVolume
apiVersion: v1
metadata:
  name: postgres-pv-volume
  labels:
    type: local
    app: postgres
spec:
  storageClassName: manual
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/mnt/data"
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: postgres-pv-claim
  labels:
    app: postgres
spec:
  storageClassName: manual
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi

After I've changed RWM to RWO - then I this (I've deleted old instances and have created new one):

Name:           postgres-54db6bdb8b-wgvr2
Namespace:      default
Priority:       0
Node:           gke-booknotes-pool-1-3e566443-dc08/10.128.15.236
Start Time:     Sun, 15 Dec 2019 04:56:57 +0700
Labels:         app=postgres
                pod-template-hash=54db6bdb8b
Annotations:    kubernetes.io/limit-ranger: LimitRanger plugin set: cpu request for container postgres
Status:         Running
IP:             10.56.6.13
IPs:            <none>
Controlled By:  ReplicaSet/postgres-54db6bdb8b
Containers:
  postgres:
    Container ID:   docker://1070018c2a670cc7e0248e6269c271c3cba022fdd2c9cc5099a8eb4da44f7d65
    Image:          postgres:10.4
    Image ID:       docker-pullable://postgres@sha256:9625c2fb34986a49cbf2f5aa225d8eb07346f89f7312f7c0ea19d82c3829fdaa
    Port:           5432/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       ContainerCannotRun
      Message:      error while creating mount source path '/mnt/data': mkdir /mnt/data: read-only file system
      Exit Code:    128
      Started:      Sun, 15 Dec 2019 10:56:21 +0700
      Finished:     Sun, 15 Dec 2019 10:56:21 +0700
    Ready:          False
    Restart Count:  76
    Requests:
      cpu:  100m
    Environment Variables from:
      postgres-config  ConfigMap  Optional: false
    Environment:       <none>
    Mounts:
      /var/lib/postgresql/data from postgredb (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-t48dw (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  postgredb:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  postgres-pv-claim
    ReadOnly:   false
  default-token-t48dw:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-t48dw
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason   Age                  From                                         Message
  ----     ------   ----                 ----                                         -------
  Warning  BackOff  81s (x1629 over 6h)  kubelet, gke-booknotes-pool-1-3e566443-dc08  Back-off restarting failed container

Answer 1

1. How to fix CrashLoopBackOff (postgres) - GCP

The issue is here: error while creating mount source path '/mnt/data': mkdir /mnt/data: read-only file system.

You need to make sure postgres-pv-claim is writable. You need to recreate the pv and pv claim with RWO access (you must have mistyped it to RO instead which is why you ran into the issue) then try to deploy postgres pod which should fix the issue.

2. Fixing FailedScheduling 69s (x10 over 7m35s) default-scheduler pod has unbound immediate PersistentVolumeClaims (repeated 4 times)

Provisioning persistent volume in GKE you don't need to create PersistentVolume objects they are dynamically created by GKE. So solve the Warning FailedScheduling 69s (x10 over 7m35s) issue

1. remove storageClassName property from your pvc and
2. delete pv

which should fix the issue. Please see below revised postgres-storage.yaml.

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: postgres-pv-claim
  labels:
    app: postgres
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi