docker pull manifest unknown blob errors

Question

I am currently trying to set up nexus as a private registry for docker images and have been able to login,push,pull,search through the nexus repository .

Currently we are stuck with pulling images from docker if not available within the nexus hosted repository . We are facing errors ranging from blob unknown to manifest unknown.

[root@server1446 ~]$ docker pull server908.int.org.com:6666/centos
Using default tag: latest
Trying to pull repository server908.int.org.com:6666/centos ...
manifest unknown: manifest unknown

Tried following reference [Setup-Docker-Private-Registry-in-Nexus-Repository-OSS-3.0.0][1]

[1]: https://github.com/TerrenceMiao/nexus/wiki/Setup-Docker-Private-Registry-in-Nexus-Repository-OSS-3.0.0 to set up the docker(proxy) and docker(hosted) repo with docker(group) but it throws blob unknown to registry error.

docker(hosted) is configured with http port 4444 & docker(group) with http port 5555 and we have used the same within nginx configuration as below but neither seems to be working .

server {

    listen 6666;
    server_name server908.int.org.com;
    keepalive_timeout 60;
    ssl on;
    ssl_certificate /etc/ssl/certs/orgnexus.crt;
    ssl_certificate_key /etc/ssl/certs/orgnexus.key;
    ssl_ciphers HIGH:!kEDH:!ADH:!MD5:@STRENGTH;
    ssl_session_cache shared:TLSSSL:16m;
    ssl_session_timeout 10m;
    ssl_prefer_server_ciphers on;
    client_max_body_size 1G;
    chunked_transfer_encoding on;

   location / {

      access_log              /var/log/nginx/docker.log;
      proxy_set_header        Host $http_host;
      proxy_set_header        X-Real-IP $remote_addr;
      proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header        X-Forwarded-Proto "https";
      proxy_pass              http://server908.int.org.com:5555;
      proxy_read_timeout      90;

    }

We have commented below entries in "/etc/sysconfig/docker" file .

http_proxy=http://x.x.x.x:3128 https_proxy=http://x.x.x.x:3128

Answer 1

Below are my configurations that got it working .

server {

    proxy_send_timeout 120;
    proxy_read_timeout 300;
    proxy_buffering    off;
    tcp_nodelay        on;

    server_tokens off;
    client_max_body_size 1G;

    listen 80;
    server_name box.company.net;
    location / {
          rewrite ^(.*) https://box.company.net$1 301;
    }
}

server {

    listen 443;
    server_name box.company.net;
    keepalive_timeout 60;
    ssl on;
    ssl_certificate /etc/ssl/certs/ssl.crt;
    ssl_certificate_key /etc/ssl/certs/ssl.key;
    ssl_ciphers HIGH:!kEDH:!ADH:!MD5:@STRENGTH;
    ssl_session_cache shared:TLSSSL:16m;
    ssl_session_timeout 10m;
    ssl_prefer_server_ciphers on;

    location / {

      proxy_set_header        Host $http_host;
      proxy_set_header        X-Real-IP $remote_addr;
      proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header        X-Forwarded-Proto "https";
      proxy_pass              http://box.company.net:8081;
      proxy_read_timeout      90;

    }
}

# correlates to your nexus http connector
server {

    listen 6666;
    server_name box.company.net;
    keepalive_timeout 60;
    ssl on;
    ssl_certificate /etc/ssl/certs/ssl.crt;
    ssl_certificate_key /etc/ssl/certs/ssl.key;
    ssl_ciphers HIGH:!kEDH:!ADH:!MD5:@STRENGTH;
    ssl_session_cache shared:TLSSSL:16m;
    ssl_session_timeout 10m;
    ssl_prefer_server_ciphers on;
    client_max_body_size 1G;
    chunked_transfer_encoding on;

    location / {

      proxy_set_header        Host $http_host;
      proxy_set_header        X-Real-IP $remote_addr;
      proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;

        if ($request_method !~* GET) {
                proxy_pass              http://box.company.net:4444;
        }
        if ($request_method = GET) {
                proxy_pass              http://box.company.net:5555;
        }
      proxy_read_timeout      90;
    }
}

Commented below entries with "/etc/default/docker" file .

http_proxy=http://x.x.x.x:3128
https_proxy=http://x.x.x.x:3128

Restart Nginx.

Perform Login

[test@server ~]$ docker login -u admin -p admin123 box.company.net:6666
Login Succeeded

Post login a file name "config.json" will be created under ".docker" directory

[test@server ~]$ cat ~/.docker/config.json
{
        "auths": {
                "box.company.net:6666": {
                        "auth": "YWRtaW46YWRtaW4xMjM="
                }
        }
}

Search for Images available in docker hub.

[test@server ~]$ docker search box.company.net:6666/ubuntu
INDEX      NAME                                                                               DESCRIPTION                                     STARS     OFFICIAL   AUTOMATED
company.net   box.company.net:6666/ubuntu                                                 Ubuntu is a Debian-based Linux operating s...   6186      [OK] 

Pull the images from docker hub via nexus proxy.

[test@server ~]$ docker pull box.company.net:6666/ubuntu
Using default tag: latest
Trying to pull repository box.company.net:6666/ubuntu ...
sha256:a0ee7647e24c8494f1cf6b94f1a3cd127f423268293c25d924fbe18fd82db5a4: Pulling from box.company.net:6666/ubuntu

75c416ea735c: Pull complete
c6ff40b6d658: Pull complete
a7050fc1f338: Pull complete
f0ffb5cf6ba9: Pull complete
be232718519c: Pull complete
Digest: sha256:a0ee7647e24c8494f1cf6b94f1a3cd127f423268293c25d924fbe18fd82db5a4
Status: Downloaded newer image for box.company.net:6666/ubuntu:latest

Tag The pulled images

docker tag box.company.net:6666/ubuntu:latest box.company.net:6666/ubuntu:1

Push to NexusHostedRepo (Port : 4444)

[test@server ~]$ docker push box.company.net:6666/ubuntu:1
The push refers to a repository [box.company.net:6666/ubuntu]
0566c118947e: Pushed
6f9cf951edf5: Pushed
182d2a55830d: Pushed
latest: digest: sha256:a0ee7647e24c8494f1cf6b94f1a3cd127f423268293c25d924fbe18fd82db5a4 size: 1357

Pull from Nexus Repo (This should be quick than the pull from docker hub )

[test@server ~]$ docker pull box.company.net:6666/ubuntu:1
Trying to pull repository box.company.net:6666/ubuntu ...
sha256:a0ee7647e24c8494f1cf6b94f1a3cd127f423268293c25d924fbe18fd82db5a4: Pulling from server908.int.org.com:6666/ubuntu

75c416ea735c: Pull complete
c6ff40b6d658: Pull complete
a7050fc1f338: Pull complete
Digest: sha256:a0ee7647e24c8494f1cf6b94f1a3cd127f423268293c25d924fbe18fd82db5a4
Status: Downloaded newer image for box.company.net:6666/ubuntu:1

--------------------------------------------------------------------------------

Also make sure you add you proxy server certificate within Nexus SSL Certificate Section .

keytool -J-Dhttps.proxyHost=<proxy_hostname> -J-Dhttps.proxyPort=<proxy_port> -printcert -rfc -sslserver <remote_host_name:remote_ssl_port>

Replace proxy_hostname and proxy_port with the HTTP proxy server that Nexus is configured with under Administration -> Server. Replace remote_host_name:remote_ssl_port with one of the remote host and port having the certification problem. You can omit the port if it is the default 443. For docker it will be registry-1.docker.io:443

You should see at least two entries printed by the above command. Take the last certificate content printed and copy it entirely to your clipboard. This should be your proxy server's certificate, added to the end of the certificate chain.

The copied certificate content should start with -----BEGIN CERTIFICATE----- and end with-----END CERTIFICATE-----.

Then in the Nexus UI, go to Administration -> SSL Certificates and click Add... and choose Paste PEM. Paste the certificate contents to the dialog that opens.

Click Load Certificate. Verify the certificate contents in the next window. Verify the Issuer details listed are from your proxy server certificate. When you are satisfied click Add Certificate.

---

Hope this helps.