How to use a caching proxy as a dockerhub mirror in EKS?

Question

I want my kubernetes cluster to retrieve container images from dockerhub via some kind of caching proxy, to mitigate Docker Hub's rate-limiting (which causes pod startup/scaling failures) and hopefully improve performance.

How do I accomplish this in AWS, and in particular, how can I reconfigure the EKS nodes to pull images via this cache or mirror (instead of directly from the default registry docker.io)?

---

Note, ideally I'd like to avoid having to: customise all of the charts and image specs to explicitly point to a private alternative registry; manually select (in advance) which dockerhub repos to mirror; or maintain bespoke tooling (to sync registries).

Answer 1

There are two parts to this:

Standing up your container registry

Ideally the cloud provider would have already facilitated some sort of caching by default in managed kubernetes clusters, but unfortunately AWS impedes interoperability with Docker Hub (presumably part of the vendor lock-in and proprietary jargon strategy). ECR Public is positioned as a competitor to Docker Hub, and only Docker's official images get sync'd from dockerhub by AWS. Private ECR registries can be configured as pull-through caches, but only of public registries other than Docker Hub, and even then they only sync updates once per day.

Instead, the official docker Registry image can be deployed as a pull-through cache (such as with this helm chart) in an EKS cluster. This should check for upstream updates on every request, and can be used directly as a mirror (without requiring customisation of charts and image specs).

Configuring the cluster to use the mirror

This configuration depends on which container runtime the cluster is using, but generally involves modifying a config file on all nodes (and ensuring the runtime has read the updated config version). EKS currently (as of 2022) uses kubernetes ≤1.23 and defaults to the docker runtime. For kubernetes 1.24+, EKS will instead default to the containerd runtime. Dockerd requires changing /etc/docker/daemon.json to contain registry-mirrors: ["foo"], whereas containerd requires /etc/containerd/certs.d/docker.io/hosts.toml be modified to configure host."https://foo".capabilities = ["pull", "resolve"].

There are basically two ways to customise the EKS nodes. One is to customise the AMI's os image (but assuming the management burden of keeping it up to date with security patches, and of running the EKS bootstrap -- incidentally, the EKS bootstrap script also includes an argument to rewrite the docker config file). The other way is the user data field of the launch template, which gets passed to cloud-init as the node starts up. The user data usually consists of a unix script. (It can be base64 encoded, can be gzipped, can give web urls to include from, can be multipart mime, it can also contain cloud-init directives, and on bottlerocket, which has no shell interpreter, the user data can use toml to specify settings such as the registry mirror.) In particular, a user data script may write to the runtime's config file and then restart the runtime daemon.