Postgres / K8S : PANIC could not locate a valid checkpoint record / CrashLoopBackOff

Question

Postgres can't start giving the error:

PANIC could not locate a valid checkpoint record

On Google, there is a lot of solution, but all of them need to connect the pod to execute some pg commands.

But, as I use K8S, my pod falls into status: CrashLoopBackOff, so I can't connect anymore to my pod.

How should I do to fix my postgres DB ?

EDIT:

I have tried to run the command:

pg_resetwal /var/lib/postgresql/data

with:

...
spec:
      containers:
      - args:
        - pg_resetwal
        - /var/lib/postgresql/data

But I get:

pg_resetwal: cannot be executed by "root"
You must run pg_resetwal as the PostgreSQL superuser.

Can go further...

EDIT2:

I tried to run a new pod with the same volumes attached, and the same postgres container, but changing the command to : pg_resetwal /var/lib/postgresql/data

I also added:

securityContext:
              runAsUser: 0

Here is the yaml for deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    reloader.stakater.com/auto: "true"
  labels:
    app: metadata-postgres-fix
  name: metadata-postgres-fix
  namespace: metadata
spec:
  selector:
    matchLabels:
      app: metadata-postgres-fix
  replicas: 1
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: metadata-postgres-fix
    spec:
      containers:
      - args:
        - pg_resetwal
        - /var/lib/postgresql/data
        envFrom:
          - secretRef:
              name: metadata-env
        image: postgres:11.3
        name: metadata-postgres-fix
        securityContext:
          runAsUser: 0
        ports:
        - containerPort: 5432
        imagePullPolicy: Always
        volumeMounts:
        - mountPath: /etc/postgresql/postgresql.conf
          name: metadata-postgres-data
          subPath: postgres.conf
        - mountPath: /docker-entrypoint-initdb.d/init.sh
          name: metadata-postgres-data
          subPath: init.sh
        - mountPath: /var/lib/postgresql/data
          name: metadata-postgres-claim
          subPath: postgres
      restartPolicy: Always
      volumes:
      - name: metadata-postgres-data
        configMap:
          name: cfgmap-metadata-postgres
      - name: metadata-postgres-claim
        persistentVolumeClaim:
          claimName: metadata-postgres-claim
      nodeSelector:
        kops.k8s.io/instancegroup: nodes

Answer 1

I solved it changing

- args:
    - pg_resetwal
    - /var/lib/postgresql/data

with a pause to be able to get UID of postgres:

- args:
    - sleep
    - 1000

with

cat /etc/passwd

I could find posgres UID is 999

and finally change runAsUser: 0 with runAsUser: 999