Blob Unknown when pushing to custom registry through apache proxy

Question

I have a custom docker registry running in a container bound to localhost like this 127.0.0.1:5010->5000/tcp. When I push an image to it locally it works. When I try to push an image to it remotely it connects and starts to push the image and then I get err.code="blob unknown". The registry is proxied through Apache with ssl enabled in Apache. TLS isn't on in the registry since it's bound to localhost it really doesn't need it. Not sure what's broken here, any suggestions?

Command (after docker login was successful):

ubuntu@ip-172-31-31-137  ~  docker push registry.sniftershifter.com/nginx
The push refers to repository [registry.sniftershifter.com/nginx]
f12c6cf07176: Pushing [==================================================>]  3.584kB
341dde1390a8: Pushing [===================>                               ]  20.87MB/53.68MB
9c46f426bcb7: Pushing [====>                                              ]  5.394MB/55.29MB
unknown blob

docker-compose.yml

registry:
  container_name: registry
  restart: always
  image: registry:2
  ports:
    - 127.0.0.1:5010:5000
  environment:
    REGISTRY_AUTH: htpasswd
    REGISTRY_AUTH_HTPASSWD_PATH: /auth/htpasswd
    REGISTRY_AUTH_HTPASSWD_REALM: Registry Realm
  volumes:
    - /home/jrow/docker_registry/data:/var/lib/registry
    - /home/jrow/docker_registry/auth:/auth

Apache config:

<IfModule mod_ssl.c>
<VirtualHost *:443>
    ServerAdmin [email protected]
    ServerName registry.sniftershifter.com
    ProxyPreserveHost On
    # setup the proxy
    <Proxy *>
        Order allow,deny
        Allow from all
    </Proxy>
    SetEnv proxy-initial-not-pooled 1
    # SSLProxyEngine on
    ProxyPass / http://localhost:5010/ KeepAlive=On Timeout=600
    ProxyPassReverse / http://localhost:5010/
SSLCertificateFile /etc/letsencrypt/live/registry.sniftershifter.com/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/registry.sniftershifter.com/privkey.pem
Include /etc/letsencrypt/options-ssl-apache.conf
</VirtualHost>
</IfModule>

Log of attempt from registry:

172.17.0.1 - - [24/Jul/2018:21:43:18 +0000] "GET /v2/ HTTP/1.1" 401 87 "" "docker/18.03.1-ce go/go1.9.5 git-commit/9ee9f40 kernel/4.4.0-1061-aws os/linux arch/amd64 UpstreamClient(Docker-Client/18.03.1-ce \\(linux\\))"
time="2018-07-24T21:43:19Z" level=error msg="response completed with error" auth.user.name=jeremiah err.code="blob unknown" err.detail=sha256:683abbb4ea60e108164f1d351e7bcf13daf45941137d800086447874df05f48e err.message="blob unknown to registry" go.version=go1.7.6 http.request.host=registry.sniftershifter.com http.request.id=96767d96-68e9-4d53-9f9e-970fabb5f94b http.request.method=HEAD http.request.remoteaddr=52.22.187.80 http.request.uri="/v2/nginx/blobs/sha256:683abbb4ea60e108164f1d351e7bcf13daf45941137d800086447874df05f48e" http.request.useragent="docker/18.03.1-ce go/go1.9.5 git-commit/9ee9f40 kernel/4.4.0-1061-aws os/linux arch/amd64 UpstreamClient(Docker-Client/18.03.1-ce \\(linux\\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=11.698327ms http.response.status=404 http.response.written=157 instance.id=8d9bc71b-b767-46e3-8acd-37cb676bb4d7 vars.digest="sha256:683abbb4ea60e108164f1d351e7bcf13daf45941137d800086447874df05f48e" vars.name=nginx version=v2.6.2

172.17.0.1 - - [24/Jul/2018:21:43:19 +0000] "HEAD /v2/nginx/blobs/sha256:683abbb4ea60e108164f1d351e7bcf13daf45941137d800086447874df05f48e HTTP/1.1" 404 157 "" "docker/18.03.1-ce go/go1.9.5 git-commit/9ee9f40 kernel/4.4.0-1061-aws os/linux arch/amd64 UpstreamClient(Docker-Client/18.03.1-ce \\(linux\\))"
time="2018-07-24T21:43:19Z" level=error msg="response completed with error" auth.user.name=jeremiah err.code="blob unknown" err.detail=sha256:a58abb4a79903b460a6db8a1237fd38c67adfd5a997a818e7b5f70d29032c256 err.message="blob unknown to registry" go.version=go1.7.6 http.request.host=registry.sniftershifter.com http.request.id=7613fa79-8969-4bf6-97b3-78b989306c94 http.request.method=HEAD http.request.remoteaddr=52.22.187.80 http.request.uri="/v2/nginx/blobs/sha256:a58abb4a79903b460a6db8a1237fd38c67adfd5a997a818e7b5f70d29032c256" http.request.useragent="docker/18.03.1-ce go/go1.9.5 git-commit/9ee9f40 kernel/4.4.0-1061-aws os/linux arch/amd64 UpstreamClient(Docker-Client/18.03.1-ce \\(linux\\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=13.077594ms http.response.status=404 http.response.written=157 instance.id=8d9bc71b-b767-46e3-8acd-37cb676bb4d7 vars.digest="sha256:a58abb4a79903b460a6db8a1237fd38c67adfd5a997a818e7b5f70d29032c256" vars.name=nginx version=v2.6.2

172.17.0.1 - - [24/Jul/2018:21:43:19 +0000] "HEAD /v2/nginx/blobs/sha256:a58abb4a79903b460a6db8a1237fd38c67adfd5a997a818e7b5f70d29032c256 HTTP/1.1" 404 157 "" "docker/18.03.1-ce go/go1.9.5 git-commit/9ee9f40 kernel/4.4.0-1061-aws os/linux arch/amd64 UpstreamClient(Docker-Client/18.03.1-ce \\(linux\\))"
time="2018-07-24T21:43:19Z" level=error msg="response completed with error" auth.user.name=jeremiah err.code="blob unknown" err.detail=sha256:b43279c1d51c22a69660908e3df2bff2cb6ccd35a2e07bafd47964517ac01574 err.message="blob unknown to registry" go.version=go1.7.6 http.request.host=registry.sniftershifter.com http.request.id=f3ccafd2-9202-4013-8689-2fdc95b95c6a http.request.method=HEAD http.request.remoteaddr=52.22.187.80 http.request.uri="/v2/nginx/blobs/sha256:b43279c1d51c22a69660908e3df2bff2cb6ccd35a2e07bafd47964517ac01574" http.request.useragent="docker/18.03.1-ce go/go1.9.5 git-commit/9ee9f40 kernel/4.4.0-1061-aws os/linux arch/amd64 UpstreamClient(Docker-Client/18.03.1-ce \\(linux\\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=11.329439ms http.response.status=404 http.response.written=157 instance.id=8d9bc71b-b767-46e3-8acd-37cb676bb4d7 vars.digest="sha256:b43279c1d51c22a69660908e3df2bff2cb6ccd35a2e07bafd47964517ac01574" vars.name=nginx version=v2.6.2

172.17.0.1 - - [24/Jul/2018:21:43:19 +0000] "HEAD /v2/nginx/blobs/sha256:b43279c1d51c22a69660908e3df2bff2cb6ccd35a2e07bafd47964517ac01574 HTTP/1.1" 404 157 "" "docker/18.03.1-ce go/go1.9.5 git-commit/9ee9f40 kernel/4.4.0-1061-aws os/linux arch/amd64 UpstreamClient(Docker-Client/18.03.1-ce \\(linux\\))"
time="2018-07-24T21:43:19Z" level=info msg="response completed" go.version=go1.7.6 http.request.host=registry.sniftershifter.com http.request.id=f76a8ffd-a554-4214-b98e-290031a9dc89 http.request.method=POST http.request.remoteaddr=52.22.187.80 http.request.uri="/v2/nginx/blobs/uploads/" http.request.useragent="docker/18.03.1-ce go/go1.9.5 git-commit/9ee9f40 kernel/4.4.0-1061-aws os/linux arch/amd64 UpstreamClient(Docker-Client/18.03.1-ce \\(linux\\))" http.response.duration=35.031305ms http.response.status=202 http.response.written=0 instance.id=8d9bc71b-b767-46e3-8acd-37cb676bb4d7 version=v2.6.2

172.17.0.1 - - [24/Jul/2018:21:43:19 +0000] "POST /v2/nginx/blobs/uploads/ HTTP/1.1" 202 0 "" "docker/18.03.1-ce go/go1.9.5 git-commit/9ee9f40 kernel/4.4.0-1061-aws os/linux arch/amd64 UpstreamClient(Docker-Client/18.03.1-ce \\(linux\\))"
time="2018-07-24T21:43:19Z" level=info msg="response completed" go.version=go1.7.6 http.request.host=registry.sniftershifter.com http.request.id=047fe1c3-86fe-44d3-87e7-169ca1d481a6 http.request.method=POST http.request.remoteaddr=52.22.187.80 http.request.uri="/v2/nginx/blobs/uploads/" http.request.useragent="docker/18.03.1-ce go/go1.9.5 git-commit/9ee9f40 kernel/4.4.0-1061-aws os/linux arch/amd64 UpstreamClient(Docker-Client/18.03.1-ce \\(linux\\))" http.response.duration=29.598379ms http.response.status=202 http.response.written=0 instance.id=8d9bc71b-b767-46e3-8acd-37cb676bb4d7 version=v2.6.2

172.17.0.1 - - [24/Jul/2018:21:43:19 +0000] "POST /v2/nginx/blobs/uploads/ HTTP/1.1" 202 0 "" "docker/18.03.1-ce go/go1.9.5 git-commit/9ee9f40 kernel/4.4.0-1061-aws os/linux arch/amd64 UpstreamClient(Docker-Client/18.03.1-ce \\(linux\\))"
time="2018-07-24T21:43:19Z" level=info msg="response completed" go.version=go1.7.6 http.request.host=registry.sniftershifter.com http.request.id=18639e66-864a-4e00-b72e-9815e3d0382b http.request.method=POST http.request.remoteaddr=52.22.187.80 http.request.uri="/v2/nginx/blobs/uploads/" http.request.useragent="docker/18.03.1-ce go/go1.9.5 git-commit/9ee9f40 kernel/4.4.0-1061-aws os/linux arch/amd64 UpstreamClient(Docker-Client/18.03.1-ce \\(linux\\))" http.response.duration=36.164985ms http.response.status=202 http.response.written=0 instance.id=8d9bc71b-b767-46e3-8acd-37cb676bb4d7 version=v2.6.2

172.17.0.1 - - [24/Jul/2018:21:43:19 +0000] "POST /v2/nginx/blobs/uploads/ HTTP/1.1" 202 0 "" "docker/18.03.1-ce go/go1.9.5 git-commit/9ee9f40 kernel/4.4.0-1061-aws os/linux arch/amd64 UpstreamClient(Docker-Client/18.03.1-ce \\(linux\\))"
time="2018-07-24T21:43:19Z" level=info msg="response completed" go.version=go1.7.6 http.request.host=registry.sniftershifter.com http.request.id=badceb1d-cc11-4add-928d-c63b50dd669d http.request.method=GET http.request.referer="http://registry.sniftershifter.com/v2/nginx/blobs/uploads/7eb09801-1fb6-4d55-97bd-e64ab6ff0bd8?_state=Ov8UC8vNnZ--PM57I7u54dQ4MqQMrn1gAa-ly_wJloF7Ik5hbWUiOiJuZ2lueCIsIlVVSUQiOiI3ZWIwOTgwMS0xZmI2LTRkNTUtOTdiZC1lNjRhYjZmZjBiZDgiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMTgtMDctMjRUMjE6NDM6MTkuMTU0MzExNzU5WiJ9" http.request.remoteaddr=52.22.187.80 http.request.uri="/v2/nginx/blobs/uploads/7eb09801-1fb6-4d55-97bd-e64ab6ff0bd8?_state=Ov8UC8vNnZ--PM57I7u54dQ4MqQMrn1gAa-ly_wJloF7Ik5hbWUiOiJuZ2lueCIsIlVVSUQiOiI3ZWIwOTgwMS0xZmI2LTRkNTUtOTdiZC1lNjRhYjZmZjBiZDgiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMTgtMDctMjRUMjE6NDM6MTkuMTU0MzExNzU5WiJ9" http.request.useragent="docker/18.03.1-ce go/go1.9.5 git-commit/9ee9f40 kernel/4.4.0-1061-aws os/linux arch/amd64 UpstreamClient(Docker-Client/18.03.1-ce \\(linux\\))" http.response.duration=24.116927ms http.response.status=204 http.response.written=0 instance.id=8d9bc71b-b767-46e3-8acd-37cb676bb4d7 version=v2.6.2

172.17.0.1 - - [24/Jul/2018:21:43:19 +0000] "GET /v2/nginx/blobs/uploads/7eb09801-1fb6-4d55-97bd-e64ab6ff0bd8?_state=Ov8UC8vNnZ--PM57I7u54dQ4MqQMrn1gAa-ly_wJloF7Ik5hbWUiOiJuZ2lueCIsIlVVSUQiOiI3ZWIwOTgwMS0xZmI2LTRkNTUtOTdiZC1lNjRhYjZmZjBiZDgiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMTgtMDctMjRUMjE6NDM6MTkuMTU0MzExNzU5WiJ9 HTTP/1.1" 204 0 "http://registry.sniftershifter.com/v2/nginx/blobs/uploads/7eb09801-1fb6-4d55-97bd-e64ab6ff0bd8?_state=Ov8UC8vNnZ--PM57I7u54dQ4MqQMrn1gAa-ly_wJloF7Ik5hbWUiOiJuZ2lueCIsIlVVSUQiOiI3ZWIwOTgwMS0xZmI2LTRkNTUtOTdiZC1lNjRhYjZmZjBiZDgiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMTgtMDctMjRUMjE6NDM6MTkuMTU0MzExNzU5WiJ9" "docker/18.03.1-ce go/go1.9.5 git-commit/9ee9f40 kernel/4.4.0-1061-aws os/linux arch/amd64 UpstreamClient(Docker-Client/18.03.1-ce \\(linux\\))"
time="2018-07-24T21:43:20Z" level=info msg="response completed" go.version=go1.7.6 http.request.host=registry.sniftershifter.com http.request.id=5b51b755-1794-41e7-a634-4cab9669e5a5 http.request.method=GET http.request.referer="http://registry.sniftershifter.com/v2/nginx/blobs/uploads/7eb09801-1fb6-4d55-97bd-e64ab6ff0bd8?_state=fD-oMCYtp_KhN3xa_8dFxdGgBmz_NS6US-Y8irYgFAx7Ik5hbWUiOiJuZ2lueCIsIlVVSUQiOiI3ZWIwOTgwMS0xZmI2LTRkNTUtOTdiZC1lNjRhYjZmZjBiZDgiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMTgtMDctMjRUMjE6NDM6MTlaIn0%3D&digest=sha256%3Ab43279c1d51c22a69660908e3df2bff2cb6ccd35a2e07bafd47964517ac01574" http.request.remoteaddr=52.22.187.80 http.request.uri="/v2/nginx/blobs/uploads/7eb09801-1fb6-4d55-97bd-e64ab6ff0bd8?_state=fD-oMCYtp_KhN3xa_8dFxdGgBmz_NS6US-Y8irYgFAx7Ik5hbWUiOiJuZ2lueCIsIlVVSUQiOiI3ZWIwOTgwMS0xZmI2LTRkNTUtOTdiZC1lNjRhYjZmZjBiZDgiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMTgtMDctMjRUMjE6NDM6MTlaIn0%3D&digest=sha256%3Ab43279c1d51c22a69660908e3df2bff2cb6ccd35a2e07bafd47964517ac01574" http.request.useragent="docker/18.03.1-ce go/go1.9.5 git-commit/9ee9f40 kernel/4.4.0-1061-aws os/linux arch/amd64 UpstreamClient(Docker-Client/18.03.1-ce \\(linux\\))" http.response.duration=19.471627ms http.response.status=204 http.response.written=0 instance.id=8d9bc71b-b767-46e3-8acd-37cb676bb4d7 version=v2.6.2

172.17.0.1 - - [24/Jul/2018:21:43:20 +0000] "GET /v2/nginx/blobs/uploads/7eb09801-1fb6-4d55-97bd-e64ab6ff0bd8?_state=fD-oMCYtp_KhN3xa_8dFxdGgBmz_NS6US-Y8irYgFAx7Ik5hbWUiOiJuZ2lueCIsIlVVSUQiOiI3ZWIwOTgwMS0xZmI2LTRkNTUtOTdiZC1lNjRhYjZmZjBiZDgiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMTgtMDctMjRUMjE6NDM6MTlaIn0%3D&digest=sha256%3Ab43279c1d51c22a69660908e3df2bff2cb6ccd35a2e07bafd47964517ac01574 HTTP/1.1" 204 0 "http://registry.sniftershifter.com/v2/nginx/blobs/uploads/7eb09801-1fb6-4d55-97bd-e64ab6ff0bd8?_state=fD-oMCYtp_KhN3xa_8dFxdGgBmz_NS6US-Y8irYgFAx7Ik5hbWUiOiJuZ2lueCIsIlVVSUQiOiI3ZWIwOTgwMS0xZmI2LTRkNTUtOTdiZC1lNjRhYjZmZjBiZDgiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMTgtMDctMjRUMjE6NDM6MTlaIn0%3D&digest=sha256%3Ab43279c1d51c22a69660908e3df2bff2cb6ccd35a2e07bafd47964517ac01574" "docker/18.03.1-ce go/go1.9.5 git-commit/9ee9f40 kernel/4.4.0-1061-aws os/linux arch/amd64 UpstreamClient(Docker-Client/18.03.1-ce \\(linux\\))"
time="2018-07-24T21:43:20Z" level=error msg="response completed with error" auth.user.name=jeremiah err.code="blob unknown" err.detail=sha256:b43279c1d51c22a69660908e3df2bff2cb6ccd35a2e07bafd47964517ac01574 err.message="blob unknown to registry" go.version=go1.7.6 http.request.host=registry.sniftershifter.com http.request.id=076e9eaa-2b0a-40c6-820a-3bc3180844c3 http.request.method=HEAD http.request.remoteaddr=52.22.187.80 http.request.uri="/v2/nginx/blobs/sha256:b43279c1d51c22a69660908e3df2bff2cb6ccd35a2e07bafd47964517ac01574" http.request.useragent="docker/18.03.1-ce go/go1.9.5 git-commit/9ee9f40 kernel/4.4.0-1061-aws os/linux arch/amd64 UpstreamClient(Docker-Client/18.03.1-ce \\(linux\\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=12.122095ms http.response.status=404 http.response.written=157 instance.id=8d9bc71b-b767-46e3-8acd-37cb676bb4d7 vars.digest="sha256:b43279c1d51c22a69660908e3df2bff2cb6ccd35a2e07bafd47964517ac01574" vars.name=nginx version=v2.6.2

172.17.0.1 - - [24/Jul/2018:21:43:20 +0000] "HEAD /v2/nginx/blobs/sha256:b43279c1d51c22a69660908e3df2bff2cb6ccd35a2e07bafd47964517ac01574 HTTP/1.1" 404 157 "" "docker/18.03.1-ce go/go1.9.5 git-commit/9ee9f40 kernel/4.4.0-1061-aws os/linux arch/amd64 UpstreamClient(Docker-Client/18.03.1-ce \\(linux\\))"

Answer 1

Adding these lines to the apache config for this vhost fixed it:

Header add X-Forwarded-Proto "https"
RequestHeader add X-Forwarded-Proto "https"