Get manifest of a public Docker image hosted on Docker Hub using the Docker Registry API

Question

I'm trying to figure out the correct URL to use for this. As an example, let's say I want to get the manifest for the alpine:3.9 tag. I've tried https://hub.docker.com/v2/repositories/library/alpine/manifests/3.9 but that yields a 404 error.

I've found that Docker Hub's registry implementation doesn't really match their documentation. For example, https://docs.docker.com/registry/spec/api/#tags indicates that the URL for getting the list of tags is v2/<name>/tags/list, but when you query Docker Hub, you actually need to leave off the "list" part of the URL: https://hub.docker.com/v2/repositories/library/alpine/tags/. So that makes me question everything about their documentation now when it comes to querying the Docker Hub registry.

Answer 1

TL;DR

The hub.docker.com REST API is not the docker registry API, rather it's a custom API mainly used for the Dockerhub frontend, but could occasionally be useful for other things. The docker registry API for dockerhub is hosted on registry-1.docker.io, with an alias on registry.docker.io. Even more confusing is that both API's start with the root path /v2.

Full answer

Here are some cURL commands that exercise some of the V2 endpoints. I'm super confused about what the hub.docker.com endpoints are for (https://hub.docker.com/v2/users/login, https://hub.docker.com/v2/repositories/library/, etc.) but I think the /v2/ there is a total red herring and unrelated to the registry V2 API? This article using hub.docker.com can get you tags, but not the manifests.

DOCKERHUB_USERNAME=$(jq -r '.username' < ~/.secrets/docker.json)
DOCKERHUB_PASSWORD=$(jq -r '.password' < ~/.secrets/docker.json)

TARGET_NS_REPO=library/debian

# yes, you need a new token for each repository, maybe you can have multiple scopes though?
PARAMS="service=registry.docker.io&scope=repository:$TARGET_NS_REPO:pull"
TOKEN=$(curl --user "$DOCKERHUB_USERNAME:$DOCKERHUB_PASSWORD" \
    "https://auth.docker.io/token?$PARAMS" \
    | jq -r '.token'
)

curl "https://registry-1.docker.io/v2/$TARGET_NS_REPO/tags/list" \
    -H "Authorization:Bearer $TOKEN" \
    | jq '.tags[:10]'

TAG="10-slim"
curl "https://registry-1.docker.io/v2/$TARGET_NS_REPO/manifests/$TAG" \
    -H "Authorization:Bearer $TOKEN" \
    | jq '.fsLayers'

Output:

[
  "10-slim",
  "10.0-slim",
  "10.0",
  "10",
  "6.0.10",
  "6.0.8",
  "6.0.9",
  "6.0",
  "6",
  "7-slim"
]
[
  {
    "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
  },
  {
    "blobSum": "sha256:1ab2bdfe97783562315f98f94c0769b1897a05f7b0395ca1520ebee08666703b"
  }
]

Reverse engineer it

I basically had to reverse engineer this with mitmproxy. If you want to know how anything else works:

1. Install/run mitmproxy. Check it's working via:

curl -x localhost:8080 http://mitm.it/cert/pem  # should print out a cert

2. Get/install it's certificate (to MITM yourself):

# Ubuntu 18.04, other distros may vary
MITM_CERT_PATH=/usr/local/share/ca-certificates/mitmproxy.crt
sudo cp ~/.mitmproxy/mitmproxy-ca-cert.cer "$MITM_CERT_PATH"
sudo chown root:root "$MITM_CERT_PATH"
sudo chmod 644 "$MITM_CERT_PATH"
sudo update-ca-certificates

# Verify MITM root cert accepted
curl -x localhost:8080 https://sha256.badssl.com/

# Troubleshooting
# - see if installed (https://unix.stackexchange.com/a/97252/42385)
awk -v cmd='openssl x509 -noout -subject' \
    '/BEGIN/{close(cmd)};{print | cmd}' \
    < /etc/ssl/certs/ca-certificates.crt \
    | grep -i mitmproxy

# - print the cert used (OpenSSL 1.1.0+)
openssl s_client -proxy localhost:8080 -showcerts -connect sha256.badssl.com:443 </dev/null

Uninstall the cert later if desired

sudo rm /usr/local/share/ca-certificates/mitmproxy.crt
sudo update-ca-certificates

Check not in the list
awk -v cmd='openssl x509 -noout -subject' \
    '/BEGIN/{close(cmd)};{print | cmd}' \
    < /etc/ssl/certs/ca-certificates.crt \
    | grep -i mitmproxy

# Double-check MITM root cert rejected
curl -x localhost:8080 https://sha256.badssl.com/

3. Run dockerd (stop the service if it already is running) with HTTPS_PROXY set

sudo HTTPS_PROXY=http://localhost:8080/ dockerd  # bash
# sudo env HTTPS_PROXY=http://localhost:8080/ dockerd  # fish

4. Tell the Docker daemon to do something, e.g. docker pull alpine. In mitmproxy you'd see something like

Flows
   GET https://registry-1.docker.io/v2/
       ← 401 application/json 87b 213ms
   GET https://auth.docker.io/token?account=youraccount&scope=repository%3Alibrary%2Fal
       pine%3Apull&service=registry.docker.io
       ← 200 application/json 4.18k 245ms
>> GET https://registry-1.docker.io/v2/library/alpine/manifests/latest
       ← 200 application/vnd.docker.distribution.manifest.list.v2+json 1.6k 294ms
   GET https://registry-1.docker.io/v2/library/alpine/manifests/sha256:57334c50959f26ce
       1ee025d08f136c2292c128f84e7b229d1b0da5dac89e9866
       ← 200 application/vnd.docker.distribution.manifest.v2+json 528b 326ms
   GET https://registry-1.docker.io/v2/library/alpine/blobs/sha256:b7b28af77ffec6054d13
       378df4fdf02725830086c7444d9c278af25312aa39b9
       ← 307 text/html 242b 288ms
   GET https://registry-1.docker.io/v2/library/alpine/blobs/sha256:0503825856099e6adb39
       c8297af09547f69684b7016b7f3680ed801aa310baaa
       ← 307 text/html 242b 322ms
   GET https://production.cloudflare.docker.com/registry-v2/docker/registry/v2/blobs/sh
       a256/b7/b7b28af77ffec6054d13378df4fdf02725830086c7444d9c278af25312aa39b9/data?…
       ← 200 application/octet-stream 1.48k 191ms
   GET https://production.cloudflare.docker.com/registry-v2/docker/registry/v2/blobs/sh
       a256/05/0503825856099e6adb39c8297af09547f69684b7016b7f3680ed801aa310baaa/data?…
       ← 200 application/octet-stream 2.66m 207ms
⇩  [27/32]                                                                     [*:8080]

5. Inspect the requests. Picking the ...manifests/latest request to look at:

Flow Details
2019-08-20 13:43:44 GET https://registry-1.docker.io/v2/library/alpine/manifests/latest
         ← 200 OK application/vnd.docker.distribution.manifest.list.v2+json 1.6k 294ms
       [[ Request ]]             Response                  Detail
Host:             registry-1.docker.io
User-Agent:       docker/19.03.1 go/go1.12.5 git-commit/74b1e89 kernel/4.15.0-55-generic
                  os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.1\\(linux\\))
Accept:           application/vnd.docker.distribution.manifest.v2+json
Accept:           application/vnd.docker.distribution.manifest.list.v2+json
Accept:           application/vnd.oci.image.index.v1+json
Accept:           application/vnd.docker.distribution.manifest.v1+prettyjws
Accept:           application/json
Accept:           application/vnd.oci.image.manifest.v1+json
Authorization:    Bearer eyJhbGci...(a big JWT returned by the auth.docker.io req.)
Accept-Encoding:  gzip
Connection:       close

Answer 2

The registry API is defined by OCI in the distribution-spec.

The complicated part of this is getting auth and headers setup. For an anonymous manifest pull from Docker Hub, that looks like:

#!/bin/sh

ref="${1:-library/ubuntu:latest}"
sha="${ref#*@}"
if [ "$sha" = "$ref" ]; then
  sha=""
fi
wosha="${ref%%@*}"
repo="${wosha%:*}"
tag="${wosha##*:}"
if [ "$tag" = "$wosha" ]; then
  tag="latest"
fi
api="application/vnd.docker.distribution.manifest.v2+json"
apil="application/vnd.docker.distribution.manifest.list.v2+json"
token=$(curl -s "https://auth.docker.io/token?service=registry.docker.io&scope=repository:${repo}:pull" \
        | jq -r '.token')
curl -H "Accept: ${api}" -H "Accept: ${apil}" \
     -H "Authorization: Bearer $token" \
     -s "https://registry-1.docker.io/v2/${repo}/manifests/${sha:-$tag}" | jq .

Note that official images are all within the library repository, e.g. library/alpine. So this script can be called like the following to pull the alpine:3.9 manifest:

$ ./hub-manifest.sh library/alpine:3.9
{
  "manifests": [
    {
      "digest": "sha256:65b3a80ebe7471beecbc090c5b2cdd0aafeaefa0715f8f12e40dc918a3a70e32",
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "platform": {
        "architecture": "amd64",
        "os": "linux"
      },
      "size": 528
    },
    {
      "digest": "sha256:7a3d88cbc7e2d6c0213deaf2d006933c9f5905c4eb7846b703a66fc6504000b7",
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "platform": {
        "architecture": "arm",
        "os": "linux",
        "variant": "v6"
      },
      "size": 528
    },
    {
      "digest": "sha256:cfd8b55d209956f63c8fcc931f5c6874984e5e0ffdcb8f45ba9085f190385d73",
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "platform": {
        "architecture": "arm",
        "os": "linux",
        "variant": "v7"
      },
      "size": 528
    },
    {
      "digest": "sha256:f920ccc826134587fffcf1ddc6b2a554947e0f1a5ae5264bbf3435da5b2e8e61",
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "platform": {
        "architecture": "arm64",
        "os": "linux",
        "variant": "v8"
      },
      "size": 528
    },
    {
      "digest": "sha256:2a41778b4675b9a91bd2ea3a55a2cfdaf4436aa85a476ee8b48993cdd6989a18",
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "platform": {
        "architecture": "386",
        "os": "linux"
      },
      "size": 528
    },
    {
      "digest": "sha256:6ee74256ce03a4280792ddb67cfefee9119349a63e86ca1c4c6407b08fec008e",
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "platform": {
        "architecture": "ppc64le",
        "os": "linux"
      },
      "size": 528
    },
    {
      "digest": "sha256:7e474fa79d2fc816da8fb626ac37d0344c83cfdffad3d55158123d0cc2683b98",
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "platform": {
        "architecture": "s390x",
        "os": "linux"
      },
      "size": 528
    }
  ],
  "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
  "schemaVersion": 2
}

From there you can see it's outputting a manifest list, and you could pull individual manifests from there:

$ hub-manifest.sh library/alpine@sha256:65b3a80ebe7471beecbc090c5b2cdd0aafeaefa0715f8f12e40dc918a3a70e32
{
  "schemaVersion": 2,
  "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
  "config": {
    "mediaType": "application/vnd.docker.container.image.v1+json",
    "size": 1509,
    "digest": "sha256:78a2ce922f8665f5a227dc5cd9fda87221acba8a7a952b9665f99bc771a29963"
  },
  "layers": [
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 2773413,
      "digest": "sha256:31603596830fc7e56753139f9c2c6bd3759e48a850659506ebfb885d1cf3aef5"
    }
  ]
}

Shell scripts only get me so far with this, so I've been writing regclient with regctl. There's also crane from Google and skopeo from RedHat that do similar things:

$ regctl manifest get alpine:3.9 --format '{{jsonPretty .}}'
{
  "manifests": [
    {
      "digest": "sha256:65b3a80ebe7471beecbc090c5b2cdd0aafeaefa0715f8f12e40dc918a3a70e32",
      "mediaType": "application\/vnd.docker.distribution.manifest.v2+json",
      "platform": {
        "architecture": "amd64",
        "os": "linux"
      },
      "size": 528
    },
...

$ regctl manifest get alpine:3.9 --format '{{jsonPretty .}}' --platform linux/amd64
{
  "schemaVersion": 2,
  "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
  "config": {
    "mediaType": "application/vnd.docker.container.image.v1+json",
    "size": 1509,
    "digest": "sha256:78a2ce922f8665f5a227dc5cd9fda87221acba8a7a952b9665f99bc771a29963"
  },
  "layers": [
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 2773413,
      "digest": "sha256:31603596830fc7e56753139f9c2c6bd3759e48a850659506ebfb885d1cf3aef5"
    }
  ]
}

The advantage of these other commands over curl is they handle different types of auth (basic and bearer), can use credential helpers, and they pass headers for a lot more media types, including the old v1 schema from Docker and the newer OCI schemas.