How to mirror dockerhub with artifactory

Question

Since dockerhub has started limiting downloads for non-paid accounts, I am frequently getting this error

ERROR: Preparation failed: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit (executor_docker.go:188:15s)

https://docs.docker.com/docker-hub/download-rate-limit/

I have an artifactory server, how can I setup docker to cache/mirror from artifactory first?

Answer 1

First setup an artifactory remote that points to docker hub at https://registry-1.docker.io

Then reconfigure all docker agents to use the registry_mirror. There are multiple ways to do this, but adding --registry-mirror to the docker start up options is most likely the easiest. See the docker documentation for more information

https://docs.docker.com/registry/recipes/mirror/

Method 1

Add --registry-mirror to the OPTIONS variable in /etc/default/docker

cat /etc/default/docker
OPTIONS=" -H unix:///var/run/docker.sock --ip-forward=true --iptables=true --ip-masq=true --registry-mirror=https://docker.artifactory.example.com -G docker"

Method 2

Edit /etc/docker/registry/config.yml

proxy:
  remoteurl: https://registry-1.docker.io
  username: [username]
  password: [password]

Method 3

If using puppet the config looks like this

  class { '::docker':
    use_upstream_package_source => false,
    manage_package              => false,
    registry_mirror             => 'https://docker.artifactory.example.com',
  }

Answer 2

Steps to avoid running into Dockerhub rate limits:

1. Sign up for a Dockerhub free account if you do not already have one. Dockerhub enables credential pulls of up to 200 per 6 hours vs 100 per 6 hours for anonymous pulls.

2. Use Artifactory as a cache between Dockerhub by setting up a remote repo to Dockerhub and a local repo to push and pull images that are not on dockerhub.

3. Avoid using Dockerhub for personal images. Only pull official images as necessary.

4. Setup your docker clients to always pull through Artifactory by using the docker repo path of the virtual repo ex:

   docker.artifactory.example.com/docker-virtual/myimage:1.0.0

5. Pull official images the same way by using the path w/ the virtual repo in it.

   docker.artifactory.example.com/docker-virtual/ubuntu:latest

6. Monitor your Dockerhub rate limits through the use of analytics JFrog has provided integrations into Splunk, Elastic, and Prometheus to monitor your rolling 6 hour window of dockerhub pull requests and cache hit ratio.