Postgres running on kubernetes looses data upon Pod recreation or cluster reboot

Question

I have the postgres container running in a Pod on GKE and a PersistentVolume set up to store the data. However, all of the data in the database is lost if the cluster reboots or if the Pod is deleted.

If I run kubectl delete <postgres_pod> to delete the existing Pod and check the newly created (by kubernetes) Pod to replace the deleted one, the respective database has not the data that it had before the Pod being deleted.

Here are the yaml files I used to deploy postgres.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: custom-storage
parameters:
  type: pd-standard
provisioner: kubernetes.io/gce-pd
reclaimPolicy: Retain
volumeBindingMode: Immediate
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: postgres-volume-claim
spec:
  storageClassName: custom-storage
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi

deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: postgres
spec:
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
      - name: postgres
        image: postgres:11.5
        resources: {}
        ports:
        - containerPort: 5432
        env:
          - name: POSTGRES_DB
            value: "dbname"
          - name: POSTGRES_USER
            value: "user"
          - name: POSTGRES_PASSWORD
            value: "password"
        volumeMounts:
          - mountPath: /var/lib/postgresql/
            name: postgresdb
      volumes:
        - name: postgresdb
          persistentVolumeClaim:
            claimName: postgres-volume-claim

I double checked that the persistentVolumeReclaimPolicy has value Retain.

What am I missing?

Answer 1

Is the cluster creating a new volume each time you delete a pod? Check with kubectl get pv.

Is this a multi-zone cluster? Your storage class is not provisionig regional disks, so you might be getting a new disk when the pod moves from one zone to another.

Possibly related to your problem, the postgres container reference recommends mounting at /var/lib/postgresql/data/pgdata and setting the PGDATA env variable: https://hub.docker.com/_/postgres#pgdata