Docker Private Registry Image Upload

Question

I'm trying to upload images to my private docker registry using the HTTP API, unfortunately without success.

Apparently I haven't understood the upload process yet and would like to ask if anyone can explain this in detail or push me in the right direction.

So far I have tried the following with curl. To experiment I only use the empty alpine image. For that i download it to my workstation with docker pull alpine and then I create tar archive with docker save -o alpine.tar alpine

I then unpack this archive into an alpine directory. This is what the content looks like: ls -R alpine

alpine:
    39cb81dcd06e3d4e2b813f56b72da567696fa9a59b652bd477615b31af969239
    e7d92cdc71feacf90708cb59182d0df1b911f8ae022d29e8e95d75ca6a99776a.json
    manifest.json

alpine/39cb81dcd06e3d4e2b813f56b72da567696fa9a59b652bd477615b31af969239:
    json
    layer.tar
    VERSION

According to the documentation I should first initiate the upload by sending a POST to the /v2//blobs/uploads/ URL, for this I execute the following command:

curl -X POST -L -D headers $DOCKER_HOST/v2/alpine/blobs/uploads

The answer is the following:

cat headers

HTTP/1.1 301 Moved Permanently
Docker distribution api version: registry/2.0
location: /v2/alpine/blobs/uploads/
Date: Tu, 21 Jan 2020 12:42:55 GMT
content length: 0

HTTP/1.1 202 Accepted
content length: 0
Docker distribution api version: registry/2.0
Docker upload guide: ed595b3c-1236-46c8-a759-14187fc60e7d
Location: http://<IPADDRESS>/v2/alpine/blobs/uploads/ed595b3c-1236-46c8-a759-14187fc60e7d?_state=GjzU_y-YDQherf4xXO57KyEonSSSwNEM8FiF8rmNfuN7Ik5hbWUiOiJhbHBpbmUiLCJVVUlEIjoiZWQ1OTViM2MtMTIzNi00NmM4LWE3NTktMTQxODdmYzYwZTdkIiwiT2Zmc2V0IjowLCJTdGFydGVkQXQiOiIyMDIwLTAxLTIxVDEyOjQyOjU1LjMxNDYzNTg0NVoifQ%3D%3D
Range: 0-0
X Content Type Options: nosniff
Date: Tu, 21 Jan 2020 12:42:55 GMT

now i would like to do a monolithic upload, as described in the docker-registry documentation.

For this I make the following curl request:

url=http://<IPADDRESS>/v2/alpine/blobs/uploads/ed595b3c-1236-46c8-a759-14187fc60e7d?_state=GjzU_y-YDQherf4xXO57KyEonSSSwNEM8FiF8rmNfuN7Ik5hbWUiOiJhbHBpbmUiLCJVVUlEIjoiZWQ1OTViM2MtMTIzNi00NmM4LWE3NTktMTQxODdmYzYwZTdkIiwiT2Zmc2V0IjowLCJTdGFydGVkQXQiOiIyMDIwLTAxLTIxVDEyOjQyOjU1LjMxNDYzNTg0NVoifQ%3D%3D?digest=sha256:39cb81dcd06e3d4e2b813f56b72da567696fa9a59b652bd477615b31af969239 

layerpath=/home/user/alpine/39cb81dcd06e3d4e2b813f56b72da567696fa9a59b652bd477615b31af969239/layer.tar

curl -X PUT -H "Content-Type=application/octet-stream" --data-binary @"$layerpath"  $url

For this I receive an answer that means nothing to me:

 {"errors":[{"code": "BLOB_UPLOAD_INVALID", "message": "blob upload invalid", "detail":212}]}

I don't know what I'm doing wrong. For any help I am very grateful.

Answer 1

Okay, I figured it out. For all the others who are also dealing with this question here is a little script which should clarify the handling of the HTTP-Api:

#!/bin/bash


# Global Variables

MANIFEST="./Manifest.json"

DOCKER_HOST=$1
REPOSITORY=$2
LAYERPATH=$3
CONFIGPATH=$4

SIZE=
DIGEST=

LOCATION=


CONFIGSIZE=
CONFIGDIGEST=
LAYERSIZE=
LAYERDIGEST=

# Functions

function initiateUpload(){
   LOCATION=$(curl -X POST -siL -v  -H "Connection: close" $DOCKER_HOST/v2/$REPOSITORY/blobs/uploads | grep Location | sed '2q;d' | cut -d: -f2- | tr -d ' ' | tr -d '\r')
}


function patchLayer(){
    layersize=$(stat -c%s "$1")
    LOCATION=$(curl -X PATCH -v -H "Content-Type: application/octet-stream" \
        -H "Content-Length: $layersize" -H "Connection: close" --data-binary @"$1" \
        $LOCATION 2>&1 | grep 'Location' | cut -d: -f2- | tr -d ' ' | tr -d '\r')
    SIZE=$layersize
}

function putLayer(){
    DIGEST=sha256:$(sha256sum $1 | cut -d ' ' -f1)
    url="$LOCATION&digest=$DIGEST"
    curl -X PUT -v -H "Content-Length: 0" -H "Connection: close" $url

}

function uploadManifest(){
    ((size=$(stat -c%s "$MANIFEST")-1))
    curl -X PUT -vvv -H "Content-Type:   application/vnd.docker.distribution.manifest.v2+json"\
    -H "Content-Length: $size" -H "Connection: close" \
    -d "$(cat "$MANIFEST")" $DOCKER_HOST/v2/$REPOSITORY/manifests/latest
}

# Check Parameters

if [ $# -lt 4 ] 
    then
        echo "Error: No arguments supplied."
        echo "Usage: upload.sh <DOCKER_HOST> <REPOSITORY> <LAYER> <CONFIG>"
       exit 1
fi

#upload Layer
initiateUpload 
patchLayer $LAYERPATH
LAYERSIZE=$SIZE
putLayer $LAYERPATH
LAYERDIGEST=$DIGEST

#upload Config

initiateUpload
patchLayer $CONFIGPATH
CONFIGSIZE=$SIZE
putLayer $CONFIGPATH
CONFIGDIGEST=$DIGEST


cat > $MANIFEST << EOF
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
   "config": {
       "mediaType": "application/vnd.docker.container.image.v1+json",
       "size": $CONFIGSIZE,
       "digest": "$CONFIGDIGEST"
   },
   "layers": [
        {
            "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
            "size":$LAYERSIZE,
            "digest": "$LAYERDIGEST"
        }
    ]
 }
 EOF

 uploadManifest

It's not pretty, but it works for that.

If you need more information, I will gladly extend this answer