Cannot login to private docker registry

Question

i've set up a private docker registry (v2) via the following:

docker run -d -p 4000:5000 --restart=always --name registry \
  -v `pwd`/auth:/auth \
  -e "REGISTRY_AUTH=htpasswd" \
  -e "REGISTRY_AUTH_HTPASSWD_REALM=Registry Realm" \
  -e REGISTRY_AUTH_HTPASSWD_PATH=/auth/htpasswd \
  -v `pwd`/certs:/certs \
  -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt \
  -e REGISTRY_HTTP_TLS_KEY=/certs/domain.key \
  registry:2

i ensure that the registry dns is manually overridden on all machines:

# cat /etc/hosts | grep myregistrydomain.com
172.23.67.28 myregistrydomain.com

and i created a self signed cert under certs for the fake hostname myregistrydomain.com and added a simple auth using:

mkdir auth
sudo docker run --entrypoint htpasswd registry:2 -Bbn kolla kolla-pass > auth/htpasswd

i then copy the .crt to both /etc/docker/certs.d/myregistrydomain.com\:4000/ca.crt and /etc/pki/ca-trust/source/anchors/myregistrydomain.com.crt and run update-ca-trust on all machines and restart docker (centos7).

on host A, i get:

# docker login --username=kolla --password=kolla-pass myregistrydomain.com:4000
Login Succeeded

however, on host B, i get:

# docker login --username=kolla --password=kolla-pass https://myregistrydomain.com:4000
Error response from daemon: Get https://myregistrydomain.com:4000/v1/users/: Forbidden

i can however (on host B) successfully run:

# curl -k https://kolla:[email protected]:4000/v2/_catalog

to make things even stranger, on the registry node itself, i run:

# docker login --username=kolla --password=kolla-pass myregistrydomain.com:4000
Error response from daemon: Get https://myregistrydomain.com:4000/v1/users/: Forbidden

# docker login --username=kolla --password=kolla-pass localhost:4000
Login Succeeded

to make it even stranger, i run tcpdump on the registry node and when i run docker login from host B, i see no packets! (i, of course, do see pings etc. from host B)

i've been trying to work out what i did right on host A that i haven't been able to do on host B - with very little success! can anyone put me out of my misery?

# docker --version # same across all servers
Docker version 1.12.1, build 23cf638

Answer 1

grrr... answering my own question....

so what happened was that i was using the systemctl drop-in's... and node B and the registry node were both in RFC1918 space... so in order to allow docker to download from the official docker registry, it was set to use:

[Service]
Environment="HTTP_PROXY=http://<proxy>:3128/"

which of course is rather restrictive and hence not forwarding the packets to my private registry.

removing this dropin under /etc/systemd/system/docker.service.d fixed the problem!