Garbage collecting Docker registry using S3 is very slow

Question

Garbage collecting a Docker registry on S3 is very slow. It took over 24 hours to GC a 500GB registry. My main docker registry is over thirty times larger.

I'm thinking that the GC is slow because the Docker garbage collection routine is not efficiently using S3 API calls. Does anyone know of a Docker registry garbage collector written specifically for S3 or some way to speed up the garbage collection process?

Thanks, Ed

Answer 1

I had a 5Tb registry (s3 backend) to GC, I couldn't make the gc work in a reasonable timeframe (less than 2 days).

As a workaround I found https://gitlab.com/gitlab-org/docker-distribution-pruner

Works a lot faster. I was able to run the garbage collect in less than 3 hours. Longest part was to delete all the blobs.

Gitlab has a lot of issues in their backlog around docker registry garbage collect. They even created their own fork. Maybe we will see some improvements in the future