Why does pushing to private, secured docker registry fail?

Question

I would like to run a private, secure, authenticated docker registry on a self healing AWS ECS cluster. The cluster setup is done and works properly, but I struggled getting the registry:latest running. The problem was, that every time I push an image, pushing the blobs fails, and goes into a retry cycle unless I get a timeout.

To make sure, that my ECS setup isn’t the blocker, I tried to setup everything locally using Docker4Mac 1.12.0-a.

First, the very basic setup works. I created my own version of the registry image, where there I put my TLS certificate bundle and key as well as the necessary htpasswd file directly into the image. [I know, this is insecure, I just do that for the testing purpose]. So here is my Dockerfile:

FROM registry:latest

COPY htpasswd /etc/docker
COPY server_bundle.pem /etc/docker
COPY server_key.pem /etc/docker

server_bundle.pem has a wildcard certificate for my domain mydomain.com (CN=*.mydomain.com) as the first one, followed by the intermediate CA certificates, so clients should be happy. My htpasswd file was created using the recommended approach:

docker run --entrypoint htpasswd registry:2 -Bbn heyitsme mysupersecurepassword > htpasswd

I build my image:

docker build -t heyitsme/registry .

and afterwards I run a very basic version w/o TLS and authentication:

docker run --restart=always -p 5000:5000 heyitsme/registry

and I can actually pull, tag and re-push an image:

docker pull alpine
docker tag alpine localhost:5000/alpine
docker push localhost:5000/alpine

This works. Next I make TLS and basic auth work via environment variables:

docker run -h registry.mydomain.com --name registry --restart=always -p 5000:5000 \ 
  -e REGISTRY_HTTP_HOST=http://registry.mydomain.com:5000 \
  -e REGISTRY_HTTP_TLS_CERTIFICATE=/etc/docker/server_bundle.pem \
  -e REGISTRY_HTTP_TLS_KEY=/etc/docker/server_key.pem \
  -e REGISTRY_AUTH=htpasswd \
  -e REGISTRY_AUTH_HTPASSWD_REALM=Registry-Realm \
  -e REGISTRY_AUTH_HTPASSWD_PATH=/etc/docker/htpasswd heyitsme/registry

For the time being I create an entry in /etc/hosts which says:

127.0.0.1 registry.mydomain.com

And then I login:

docker login registry.mydomain.com:5000
Username: heyitsme
Password: ***********
Login Succeeded

So now, let’s tag and push the image here:

docker tag alpine registry.mydomain.com:5000/alpine
docker push registry.mydomain.com:5000/alpine

The push refers to a repository [registry.mydomain.com:5000/alpine]
4fe15f8d0ae6: Retrying in 4 seconds

What happens is, that the docker clients tries to push fragments and fails. Then it retries and fails again until I get a timeout. So next check, whether the V2 API works properly:

curl -i -XGET https://registry.mydomain.com:5000/v2/
HTTP/1.1 401 Unauthorized
Content-Type: application/json; charset=utf-8
Docker-Distribution-Api-Version: registry/2.0
Www-Authenticate: Basic realm="Registry-Realm"
X-Content-Type-Options: nosniff
Date: Thu, 15 Sep 2016 10:06:04 GMT
Content-Length: 87

{"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":null}]}

Ok, as expected. So let’s authenticate next time:

curl -i -XGET https://heyitsme:[email protected]:5000/v2/
HTTP/1.1 200 OK
Content-Length: 2
Content-Type: application/json; charset=utf-8
Docker-Distribution-Api-Version: registry/2.0
X-Content-Type-Options: nosniff
Date: Thu, 15 Sep 2016 10:06:16 GMT

{}%

Works. But pushing still fails.

The logs say:

time="2016-09-15T10:24:34Z" level=warning msg="error authorizing context: basic authentication challenge for realm \"Registry-Realm\": invalid authorization credential" go.version=go1.6.3 http.request.host="registry.mydomain.com:5000" http.request.id=6d2ec080-6824-4bf7-aac2-5af31db44877 http.request.method=GET http.request.remoteaddr="172.17.0.1:40878" http.request.uri="/v2/" http.request.useragent="docker/1.12.0 go/go1.6.3 git-commit/8eab29e kernel/4.4.15-moby os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.0 \\(darwin\\))" instance.id=be3a8877-de64-4574-b47a-70ab036e7b79 version=v2.5.1
172.17.0.1 - - [15/Sep/2016:10:24:34 +0000] "GET /v2/ HTTP/1.1" 401 87 "" "docker/1.12.0 go/go1.6.3 git-commit/8eab29e kernel/4.4.15-moby os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.0 \\(darwin\\))"
time="2016-09-15T10:24:34Z" level=info msg="response completed" go.version=go1.6.3 http.request.host="registry.mydomain.com:5000" http.request.id=8f81b455-d592-431d-b67d-0bc34155ddbf http.request.method=POST http.request.remoteaddr="172.17.0.1:40882" http.request.uri="/v2/alpine/blobs/uploads/" http.request.useragent="docker/1.12.0 go/go1.6.3 git-commit/8eab29e kernel/4.4.15-moby os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.0 \\(darwin\\))" http.response.duration=30.515131ms http.response.status=202 http.response.written=0 instance.id=be3a8877-de64-4574-b47a-70ab036e7b79 version=v2.5.1
172.17.0.1 - - [15/Sep/2016:10:24:34 +0000] "POST /v2/alpine/blobs/uploads/ HTTP/1.1" 202 0 "" "docker/1.12.0 go/go1.6.3 git-commit/8eab29e kernel/4.4.15-moby os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.0 \\(darwin\\))"
2016/09/15 10:24:34 http: TLS handshake error from 172.17.0.1:40886: tls: first record does not look like a TLS handshake

I also tested different versions of the original registry image, especially several versions above 2. All yield the same error. If someone could help me on that issue, that would be awesome.

Answer 1

Solved:

-e REGISTRY_HTTP_HOST=https://registry.mydomain.com:5000 \

as environment variable did the tick. Just the prior use of http instead https made the connection fail.

Answer 2

Thanks a lot, it's working now!

I added the variable in my kubernetes deployment manifest:

(...)
      - env:
        - name: REGISTRY_STORAGE_DELETE_ENABLED
          value: "true"
        - name: REGISTRY_HTTP_HOST
          value: "https://registry.k8sprod.acme.domain:443"
(...)

For those who use nginx ingress in front, you have to set extra attribute in your ingress manifest file if you have push size errors:

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: ingress-registry
  namespace: default
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/proxy-body-size: "8192m"
    nginx.ingress.kubernetes.io/proxy-connect-timeout: "600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
    nginx.ingress.kubernetes.io/proxy-next-upstream-timeout: "600"
    nginx.ingress.kubernetes.io/proxy-next-upstream-tries: "10"
    nginx.ingress.kubernetes.io/proxy-request-buffering: "off"
  labels:
    k8s-app: kube-registry
    app: kube-registry
spec:
  rules:
    - host: registry.k8sprod.acme.domain
      http:
        paths:
        - path: /
          backend:
            serviceName: svc-kube-registry
            servicePort: 5000