Gitlab pipeline failed : ERROR: Preparation failed: Error response from daemon: toomanyrequests

Question

I have Harbor local docker registry and all needed images are there and connected GitLab to the Harbor and all the images are received from the Harbor but after November 2, Docker put a limit on the number of pulls and it seems dind service pulls from Docker hub. Is it possible to use dind service to pull from Harbor?

Pipeline output:

Running with gitlab-runner 12.10.1 (ce065b93)
  on docker_runner_7 WykGNjC6
Preparing the "docker" executor
30:20
Using Docker executor with image **harbor**.XXX.XXXX.net/library/docker_maven_jvm14 ...
Starting service docker:**dind** ...
**Pulling docker image docker:dind** ...
**ERROR**: Preparation failed: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit (docker.go:198:2s)
Will be retried in 3s ...
Using Docker executor with image harbor.XXX.XXX.net/library/docker_maven_jvm14 ...
Starting service docker:dind ...
Pulling docker image docker:dind ...
ERROR: Preparation failed: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit (docker.go:198:4s)
Will be retried in 3s ...
Using Docker executor with image harbor.XXX.XXX.net/library/docker_maven_jvm14 ...
Starting service docker:dind ...
Pulling docker image docker:dind ...
ERROR: Preparation failed: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit (docker.go:198:3s)
Will be retried in 3s ...
ERROR: Job failed (system failure): Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit (docker.go:198:3s)

Answer 1

I can't found the solution for Gitlab but you can tell the docker to ignore the docker hub registry and go to the local registry.

Add daemon.json in /etc/docker/daemon.json , if doesn't exist you can simply add in the path.

daemon.json

{
  "registry-mirrors": ["https://harbor.XXX.com"]
}

sudo systemctl restart docker

Answer 2

I too faced the same issue while deploying some micro-services to kube cluster, here is a blog that I wrote that provides a workaround to optimize the deployment workflow: https://mailazy.com/blog/optimize-docker-pull-gitlab-pipelines/