Docker Garbage collector not finding anything to delete

Question

Ive been using reg to mark the blobs I want to delete from the registry by doing:

docker run --rm reg/delete -u username -p password --registry 
myregistry.com rm image

I didn't get any output back so I thought I marked the image to delete it.

Then I ran the command in my server to my docker registry container to run the garbage collector like:

     sudo docker exec -it registry bin/registry garbage-
collect --dry-run /config/config.yml

And I got a lot of output back like:

DEBU[0007] filesystem.Stat("/docker/registry/v2/blobs/sha256/17/17954313e0f8926875ce0743fe85693e0e9f695908c8f1544360d067a66b5eee/data")  environment=development go.version=go1.7.3 instance.id=3296f17d-2ba7-4b3f-8530-991cb2ff8e04 service=registry trace.duration=15.631837ms trace.file=/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go trace.func=github.com/docker/distribution/registry/storage/driver/base.(*Base).Stat trace.id=6977da41-fd94-4926-99af-059821e18046 trace.line=137
account-portal: marking manifest sha256:17954313e0f8926875ce0743fe85693e0e9f695908c8f1544360d067a66b5eee
DEBU[0007] (*manifestStore).Get                          environment=development go.version=go1.7.3 instance.id=3296f17d-2ba7-4b3f-8530-991cb2ff8e04 service=registry
DEBU[0007] filesystem.GetContent("/docker/registry/v2/repositories/account-portal/_manifests/revisions/sha256/17954313e0f8926875ce0743fe85693e0e9f695908c8f1544360d067a66b5eee/link")  environment=development go.version=go1.7.3 instance.id=3296f17d-2ba7-4b3f-8530-991cb2ff8e04 service=registry trace.duration=86.1µs trace.file=/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go trace.func=github.com/docker/distribution/registry/storage/driver/base.(*Base).GetContent trace.id=ebd03857-56b6-4957-b791-9c48fdb152e9 trace.line=82
DEBU[0007] filesystem.Stat("/docker/registry/v2/blobs/sha256/17/17954313e0f8926875ce0743fe85693e0e9f695908c8f1544360d067a66b5eee/data")  environment=development go.version=go1.7.3 instance.id=3296f17d-2ba7-4b3f-8530-991cb2ff8e04 service=registry trace.duration=74.19µs trace.file=/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go trace.func=github.com/docker/distribution/registry/storage/driver/base.(*Base).Stat trace.id=96520ca1-6ba1-470c-869e-cf0a49846390 trace.line=137
DEBU[0007] filesystem.GetContent("/docker/registry/v2/blobs/sha256/17/17954313e0f8926875ce0743fe85693e0e9f695908c8f1544360d067a66b5eee/data")  environment=development go.version=go1.7.3 instance.id=3296f17d-2ba7-4b3f-8530-991cb2ff8e04 service=registry trace.duration=20.820776ms trace.file=/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go trace.func=github.com/docker/distribution/registry/storage/driver/base.(*Base).GetContent trace.id=6752e130-1b77-49c2-adda-3b9195aac1df trace.line=82
DEBU[0007] (*schema2ManifestHandler).Unmarshal           environment=development go.version=go1.7.3 instance.id=3296f17d-2ba7-4b3f-8530-991cb2ff8e04 service=registry
account-portal: marking blob sha256:90769d06ffb7dd1f8d1bc4f7b18e52fb94b651191640b680f028806622b99236
account-portal: marking blob sha256:3b8d91fe75724cafe136236e31d5212cb322d68aa3bfc4bfda97a0a53a55e234
account-portal: marking blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
account-portal: marking blob sha256:dfc6505cfad4d185913ccd772541a0d32e64d4e04d9c249728dd7e194c77340f
account-portal: marking blob sha256:47b142d56ce3cc7766143d5fcd836541dac5006fa66076d4261e9e3d4a24a0cd
account-portal: marking blob sha256:6397456b1db08c3b5fb104497d15eb849a317c35ffeb18b608e6013593b82300
account-portal: marking blob sha256:f1074061d22a0681d732d72a8d11abf0a2f77ee3652ee827c62d696c19697a93
account-portal: marking blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
account-portal: marking blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
account-portal: marking blob sha256:7b60f5a7551c3bb15dcd5ecd46a2ec18118cc652f6bdbff23bf16c895a8a1586
account-portal: marking blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
account-portal: marking blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
account-portal: marking blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
account-portal: marking blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
account-portal: marking blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
account-portal: marking blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
account-portal: marking blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
account-portal: marking blob sha256:3b36e36d988def39cf035c9a82076bbff95861bd9092701516d937f7dff24a73
account-portal: marking blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
account-portal: marking blob sha256:283fbeeb0d74113ed38891d145bcbff04a96178d2f1e19bb9dc5564ebd1959ab
account-portal: marking blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
account-portal: marking blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
account-portal: marking blob sha256:68ad5d9d0e66645895611177d50dfc46bf1bbf9cd1275e43da416af0d2b2c30a
account-portal: marking blob sha256:6fae118190fe88ba8487a3a2cfac7b1f2d922d313963cd81329fd37b2516ca3d
account-portal: marking blob sha256:9e14db6259dc2696e6fe816ddb106ec1810a4f39075695c66cbeccd58f54aad1

and at the end I get the recap like this:

12530 blobs marked, 0 blobs eligible for deletion

So according to the output no images were removed from my registry. I confirmed by running df -h and yes I still using the same amount of disk space.

So, I don't know what I'm doing wrong, I've followed all the instructions in the docker documentation about the Garbage Collector I don't know what do to do get space back.

Answer 1

Same problem here using registry v2.

Finally I was able to free out about 500GB using docker-distribution-pruner.

Just look for registry configuration inside the container in

/etc/docker/registry/config.yml

Make a copy of that file in your host and change the rootdirectory path property to the volume path where registry stores images data. That should look something like this:

version: 0.1
log:
  fields:
    service: registry
storage:
  cache:
    blobdescriptor: inmemory
  filesystem:
    rootdirectory: /data/docker-registry-data/

Then (obviously after installing the tool) run the pruner

$ sudo /home/user/bin/docker-distribution-pruner \
    -config=/home/user/registry-config.yml 

By default it runs in dry run mode (no changes). When run with -delete it will soft delete all data by moving them to docker-backup folder. In case of any problems you can move data back and restore previous state of registry.

If you run -delete -soft-delete=false you will remove data forever.