Docker PostgreSQL service: could not bind IPv6 socket: Cannot assign requested address

Question

I am running a PostgreSQL service with Docker. For some reason, PostgreSQL wants to bind to IPV6 - although I haven't specified that anywhere (at least as far as I'm aware).

As a result of this, I am unable to connect to PG. Relevant details follow below:

Dockerfile

FROM postgres:9.6

RUN apt-get update \
 && apt-get -y install apt-utils \
 && apt-get -y install python3 \
 && apt-get -y install postgresql-plpython3-9.6

COPY sql /docker-entrypoint-initdb.d/ 

EXPOSE 5432

# Add VOLUMEs to allow backup of config, logs and databases
VOLUME  ["/etc/postgresql", "/var/log/postgresql", "/var/lib/postgresql"]

PostgreSQL log file contents

LOG:  received fast shutdown request
LOG:  aborting any active transactions
waiting for server to shut down....LOG:  autovacuum launcher shutting down
LOG:  shutting down
LOG:  database system is shut down
 done
server stopped

PostgreSQL init process complete; ready for start up.

LOG:  could not bind IPv6 socket: Cannot assign requested address
HINT:  Is another postmaster already running on port 5432? If not, wait a few seconds and retry.
LOG:  database system was shut down at 2017-10-09 21:22:22 UTC
LOG:  MultiXact member wraparound protections are now enabled
LOG:  database system is ready to accept connections
LOG:  autovacuum launcher started

I run the container using the following command: docker run --name my_db_service_cntnr image_tag

When I run the following command: docker container port my_db_service_cntnr, I get nothing returned:

me@yourbox:~/path/to/pgdb$ docker container port my_db_service_cntnr 
me@yourbox:~/path/to/pgdb$ 

I know PostgreSQL is running in the container:

me@yourbox:~/path/to/pgdb$ docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS               NAMES
824ffe17c5b9        df:pg               "docker-entrypoint..."   16 hours ago        Up 5 minutes        5432/tcp            my_db_service_cntnr


me@yourbox:/path/to/pgdb$ docker container inspect my_db_service_cntnr | grep Address
            "LinkLocalIPv6Address": "",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "GlobalIPv6Address": "",
            "IPAddress": "172.17.0.2",
            "MacAddress": "02:42:ac:11:00:02",
                    "IPAddress": "172.17.0.2",
                    "GlobalIPv6Address": "",
                    "MacAddress": "02:42:ac:11:00:02"

Yet when I attempt to connect to PostgreSQL (using default port of 5432), it fails to connect to the database:

Attempting to connect via psql

me@yourbox:~/path/to/pgdb$ psql -h 172.17.0.2 -U postgres -p 5432
psql: could not connect to server: Connection refused
        Is the server running on host "172.17.0.2" and accepting
        TCP/IP connections on port 5432?
me@yourbox:~/path/to/pgdb$ 

It seems port 5432 is not being listened to on my machine, despite me specifying that the PG image EXPOSE port 5432:

me@yourbox:~/path/to/pgdb$ sudo lsof -i -P | grep -i "listen"
lighttpd  1477  www-data    4u  IPv4  22342      0t0  TCP *:80 (LISTEN)
dnsmasq   1645    nobody    5u  IPv4  26954      0t0  TCP CEBERUS:53 (LISTEN)
master    2182      root   12u  IPv4  28720      0t0  TCP localhost:25 (LISTEN)
master    2182      root   13u  IPv6  28721      0t0  TCP ip6-localhost:25 (LISTEN)
rhythmbox 3149        me   17u  IPv4  33925      0t0  TCP *:3689 (LISTEN)
rhythmbox 3149        me   18u  IPv6  33926      0t0  TCP *:3689 (LISTEN)
cupsd     8432      root   10u  IPv6  87004      0t0  TCP ip6-localhost:631 (LISTEN)
cupsd     8432      root   11u  IPv4  87005      0t0  TCP localhost:631 (LISTEN)

What is causing this error, and how do I resolve it?

Answer 1

This isn't a complete answer, but should get you closer to an answer. It covers the dockerism's that are required for your debug steps in the question.

Run a postgres container

$ CID=$(docker run -d postgres)
$ echo $CID
48024dc71aa446...

Get the PID of the container

$ PID=$(docker inspect -f {{.State.Pid}} $CID)
$ echo $PID
7994

Get the logs from the container, check for errors.

$ docker logs $CID

Process list from the container

$ docker exec -ti $CID ps -ef
UID        PID  PPID  C STIME TTY          TIME CMD
postgres     1     0  0 23:19 ?        00:00:00 postgres
postgres    49     1  0 23:19 ?        00:00:00 postgres: checkpointer process  
postgres    50     1  0 23:19 ?        00:00:00 postgres: writer process  
postgres    51     1  0 23:19 ?        00:00:00 postgres: wal writer process  
postgres    52     1  0 23:19 ?        00:00:00 postgres: autovacuum launcher pr
postgres    53     1  0 23:19 ?        00:00:00 postgres: stats collector proces
postgres    54     1  0 23:19 ?        00:00:00 postgres: bgworker: logical repl
root        66     0  0 23:26 ?        00:00:00 ps -ef

Run ss in the container looking for listening tcp processes (like lsof)

$ docker exec -ti $CID ss -lntp
State      Recv-Q Send-Q Local Address:Port               Peer Address:Port              
LISTEN     0      128          *:5432                     *:*                  
LISTEN     0      128         :::5432                    :::*

Outside the container won't report on ports in container namespaces

$ ss -lntp
State      Recv-Q Send-Q                         Local Address:Port                                        Peer Address:Port              
LISTEN     0      128                                        *:22                                                     *:*                   users:(("sshd",pid=592,fd=3))
LISTEN     0      128                                       :::22                                                    :::*                   users:(("sshd",pid=592,fd=4))

From the host, you can use nsenter to enter the containers namespace and run commands.

$ nsenter -t $PID -n ss -lntp
State      Recv-Q Send-Q                         Local Address:Port                                        Peer Address:Port              
LISTEN     0      128                                        *:5432                                                   *:*                   users:(("postgres",pid=7994,fd=3))
LISTEN     0      128                                       :::5432                                                  :::*                   users:(("postgres",pid=7994,fd=4))

$ nsenter -t $PID -n ip address show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
52: eth0@if53: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:42:ac:11:00:03 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 172.17.0.3/16 scope global eth0
       valid_lft forever preferred_lft forever

Get the IP of the container from Docker

$ IP=$(docker inspect -f '{{.NetworkSettings.Networks.bridge.IPAddress}}' $CID)
$ echo $IP
172.17.0.3

Test the connection

$ psql -h $IP -U postgres -p 5432

Mapped ports

With a mapped port, the ports on the host changes slightly

$ CID=$(docker run -d -p 5432:5432 postgres)
$ echo $CID
020f72394fcd...

Now the container has a port configured

$ docker container port $CID
5432/tcp -> 0.0.0.0:5432

$ docker inspect -f {{.NetworkSettings.Ports}} $CID
map[5432/tcp:[{0.0.0.0 5432}]]

The port will also be listening on the host

$ ss -lntp
State      Recv-Q Send-Q                         Local Address:Port                                        Peer Address:Port              
LISTEN     0      128                                        *:22                                                     *:*                   users:(("sshd",pid=592,fd=3))
LISTEN     0      128                                       :::22                                                    :::*                   users:(("sshd",pid=592,fd=4))
LISTEN     0      128                                       :::5432                                                  :::*                   users:(("docker-proxy",pid=8571,fd=4))

And you can connect to localhost or your hosts IP now

$ psql -h 127.0.0.1 -U postgres -p 5432