Docker-Compose postgres upgrade initdb: error: directory "/var/lib/postgresql/data" exists but is not empty

Question

I had postgres 11 installed using docker-compose. I wanted to upgrade it to 12 but even though I have removed the container and its volume but the status of the container says "Restarting".

Here is my docker-compose file

version: '3.5'
services:
  postgres:
    image: postgres:12
    environment:
      POSTGRES_HOST_AUTH_METHOD: "trust"
    ports:
    - "5432"
    restart: always
    volumes:
    - /etc/postgresql/12/postgresql.conf:/var/lib/postgresql/data/postgresql.conf
    - db_data:/var/lib/postgresql/data
volumes:
  db_data:

However it is not working and the logs has the following issue

2020-07-02T12:54:47.012973448Z The files belonging to this database system will be owned by user "postgres".
2020-07-02T12:54:47.013030445Z This user must also own the server process.
2020-07-02T12:54:47.013068962Z 
2020-07-02T12:54:47.013222608Z The database cluster will be initialized with locale "en_US.utf8".
2020-07-02T12:54:47.013261425Z The default database encoding has accordingly been set to "UTF8".
2020-07-02T12:54:47.013281815Z The default text search configuration will be set to "english".
2020-07-02T12:54:47.013293326Z 
2020-07-02T12:54:47.013303793Z Data page checksums are disabled.
2020-07-02T12:54:47.013313919Z 
2020-07-02T12:54:47.013450079Z initdb: error: directory "/var/lib/postgresql/data" exists but is not empty
2020-07-02T12:54:47.013487706Z If you want to create a new database system, either remove or empty
2020-07-02T12:54:47.013501126Z the directory "/var/lib/postgresql/data" or run initdb
2020-07-02T12:54:47.013512379Z with an argument other than "/var/lib/postgresql/data".

How could I remove or empty this /var/lib/postgresql/data when the container is constantly restarting?

Thanks in advance

Answer 1

I got this issue because the /var/lib/postgresql/data/postgresql.conf and the /var/lib/postgresql/data overlap in the docker container at /var/lib/postgresql/.

An example of a broken config is:

version: "3.8"
services:
  db:
    image: "postgres:10"
    ports:
      - "5432:5432"
    volumes:
      - ./postgresql.conf:/var/lib/postgresql/data/postgresql.conf
      - ./pg-data:/var/lib/postgresql/data

To avoid this I tell PostgreSQL to find it's config in /etc/postgresql.conf instead so the overlapping volumes don't occur like this:

version: "3.8"
services:
  db:
    image: "postgres:10"
    command: ["postgres", "-c", "config_file=/etc/postgresql.conf"]
    ports:
      - "5432:5432"
    volumes:
      - ./postgresql.conf:/etc/postgresql.conf
      - ./pg-data:/var/lib/postgresql/data

This is similar to what pmsoltani suggests, but I move the location of the postgresql.conf file instead of the data directory.

Answer 2

Quoting @yosifkit from this issue

The volume needs to be empty or a valid already initialized postgres database with the file PG_VERSION in there so the init can be skipped.

... If there are any files or folders in there like lost+found it will probably fail to initialize. If there are files that you want to keep in the volume (or have no control over) you could adjust the PGDATA environment variable to point to a sub-directory in there like -e PGDATA=/var/lib/postgresql/data/db-files/.

So I added PGDATA to the environment section of the compose file to solve the issue (notice the some_name at the end):

services:
  postgres:
    image: postgres:12
    environment:
      PGDATA: /var/lib/postgresql/data/some_name/

Answer 3

I had the same issue today, I fixed it by removing the content of the volume db_data in your case

docker volume ls
docker volume inspect db_data <-- will show you the mountpoint

I went to the directory (mountpoint) ex: /path/data

cp data data.backup
cd data
rm -R *

And start services:

docker-compose up -d