Kubernetes persistent volume mount for PgAdmin

Question

I'm attempting to create a persistent volume claim for my pgadmin deployment so I can persist my settings, servers, etc. when I rollout updates after each deployment from CD pipeline.

In my logs I'm getting the following errors:

...
[2020-10-05 00:54:56 +0000] [91] [INFO] Worker exiting (pid: 91)
WARNING: Failed to set ACL on the directory containing the configuration database:
           [Errno 1] Operation not permitted: '/var/lib/pgadmin'
HINT   : You may need to manually set the permissions on
         /var/lib/pgadmin to allow pgadmin to write to it.
ERROR  : Failed to create the directory /var/lib/pgadmin/sessions:
           [Errno 13] Permission denied: '/var/lib/pgadmin/sessions'
HINT   : Create the directory /var/lib/pgadmin/sessions, ensure it is writeable by
         'pgadmin', and try again, or, create a config_local.py file
         and override the SESSION_DB_PATH setting per
         https://www.pgadmin.org/docs/pgadmin4/4.26/config_py.html

Just a bunch of permission failures for writing:

PGAdmin deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: pgadmin
spec:
  selector:
   matchLabels:
    app: pgadmin
  replicas: 1
  template:
    metadata:
      labels:
        app: pgadmin
    spec:
      containers:
        - name: pgadmin4
          image: dpage/pgadmin4
          volumeMounts:
            - mountPath: /var/lib/pgadmin
              name: pgadminstorage
          env:
           - name: PGADMIN_DEFAULT_EMAIL
             valueFrom:
               secretKeyRef:
                 name: un
                 key: un
           - name: PGADMIN_DEFAULT_PASSWORD
             valueFrom:
               secretKeyRef:
                 name: pw 
                 key: pw
           - name: PGADMIN_PORT
             value: "80"
          ports:
            - containerPort: 80
              name: pgadminport
      volumes:
        - name: pgadminstorage
          persistentVolumeClaim:
            claimName: pgadmin-persistent-volume-claims-cfg

Volumes

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pgadmin-persistent-volume-claims-cfg
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi

What could I be missing here?

Update:

This might be a problem specific to digitalocean and inability to set permissions. PVC will set perms to root, but writing as pgadmin is causing issues at startup Adding this to my pgadmin deployment fixed everything

      initContainers:
        - name: pgadmin-data-permission-fix
          image: busybox
          command: ["/bin/chown", "-R", "5050:5050", "/var/lib/pgadmin"]
          volumeMounts:
          - name: pgadminstorage
            mountPath: /var/lib/pgadmin

You could probably also chmod recursive on the dir as well and also be fine.

Answer 1

I've replicated your issue. Root cause is with PgAdmin issue, not Kubernetes. Pods will be deployed without issue. You will receive error as container won't be able to create folder inside folder /var/lib. If you will check pgadmin pod logs - kubectl logs <pgadmin-pod> you will see errors like:

$ kubectl logs pgadmin-d569b67fd-8rnkc
WARNING: Failed to set ACL on the directory containing the configuration database:
           [Errno 1] Operation not permitted: '/var/lib/pgadmin'
HINT   : You may need to manually set the permissions on
         /var/lib/pgadmin to allow pgadmin to write to it.
ERROR  : Failed to create the directory /var/lib/pgadmin/sessions:
           [Errno 13] Permission denied: '/var/lib/pgadmin/sessions'
HINT   : Create the directory /var/lib/pgadmin/sessions, ensure it is writeable by
         'pgadmin', and try again, or, create a config_local.py file
         and override the SESSION_DB_PATH setting per
         https://www.pgadmin.org/docs/pgadmin4/4.26/config_py.html
sudo: setrlimit(RLIMIT_CORE): Operation not permitted

If you will check /var/lib/ folder permissions you will se that you can only Read and Execute, so you won't be able to create in this folder anything (as default, you will be logged as pgadmin user).

drwxr-xr-x    1 root     root          4096 Sep  5 14:01 lib

Depends on your needs you can resolve it in a few ways. As fastest workaround you can just change path to folder which allows Write, like tmp.

drwxrwxrwt    1 root     root          4096 Oct  5 14:28 tmp

In YAML it would looks like:

  containers:
    - name: pgadmin4
      image: dpage/pgadmin4
      volumeMounts:
        - mountPath: /var/tmp/pgadmin
          name: pgadminstorage

When you will check logs, there won't be any issues.

$ kubectl logs pgadmin-6bb74cffb8-6q9tr
NOTE: Configuring authentication for SERVER mode.

sudo: setrlimit(RLIMIT_CORE): Operation not permitted
[2020-10-05 14:28:15 +0000] [1] [INFO] Starting gunicorn 19.9.0
[2020-10-05 14:28:15 +0000] [1] [INFO] Listening at: http://[::]:80 (1)
[2020-10-05 14:28:15 +0000] [1] [INFO] Using worker: threads
/usr/local/lib/python3.8/os.py:1023: RuntimeWarning: line buffering (buffering=1) isn't supported in binary mode, the default buffer size will be used
  return io.open(fd, *args, **kwargs)
[2020-10-05 14:28:15 +0000] [89] [INFO] Booting worker with pid: 89
user@cloudshell:~/pgadmin (project)$

Regarding PgAdmin permissions issue there was already a few topics on StackOverflow or Github like: OSError: [Errno 13] Permission denied: '/var/lib/pgadmin'

pgadmin exit code 3 PermissionError: [Errno 13] Permission denied: '/var/lib/pgadmin/sessions'

[stable/pgadmin] files in /var/lib/pgadmin/sessions crash the pod

In short, you could try to manually change permissions or use specific user.

In addition, if you are using Cloud environment, you could think about using CloudSQL, instead of trying to put database into cloud. For example PostgreSQL with GKE

EDIT

As per @Ryan comment below this answer, you can also use Init Containers to change /var/lib/ permissions. Each init container must complete successfully before the next one starts and it runs before app containers in a pod.

specialized containers that run before app containers in a Pod. Init containers can contain utilities or setup scripts not present in an app image.