How do I access a private Docker registry with a self signed certificate using Kubernetes?

Question

Currently, running a private Docker registry (Artifactory) on an internal network that uses a self signed certificate for authentication.

When Kubernetes starts up a new node, it is unable to auth with the private Docker registry because this new node does not have the self signed certificate.

Any help would be much appreciated. Thanks!

Answer 1

The simplest solution I found after an extensive search is suggested in this guide by CoreOS : https://github.com/coreos/tectonic-docs/blob/master/Documentation/admin/add-registry-cert.md

It consists to create a secret that contains your certificate and a DaemonSet to populate it to /etc/docker/certs.d/my-private-insecure-registry.com/ca.crt on all the nodes of your cluster.

I think this answers your question because, when adding a new node, the DaemonSet is automatically executed on it.

I give the detailed solution below but all the credits goes to Kyle Brown (kbrwn) for his very cool guide (cf. link above).

Detailed solution for Kubernetes 1.16+

Lets suppose that your certificate is a file named ca.crt in your working directory. Create a secret from this file content :

kubectl create secret generic registry-ca --namespace kube-system --from-file=registry-ca=./ca.crt

Then, use the following DaemonSet that mounts the certificate as the file /home/core/registry-ca and copy it to the desired location : /etc/docker/certs.d/reg.example.com/ca.crt.

Simply replace my-private-insecure-registry.com with the hostname of your container registry.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: registry-ca
  namespace: kube-system
  labels:
    k8s-app: registry-ca
spec:
  selector:
    matchLabels:
      name: registry-ca
  template:
    metadata:
      labels:
        name: registry-ca
    spec:
      containers:
      - name: registry-ca
        image: busybox
        command: [ 'sh' ]
        args: [ '-c', 'cp /home/core/registry-ca /etc/docker/certs.d/my-private-insecure-registry.com/ca.crt && exec tail -f /dev/null' ]
        volumeMounts:
        - name: etc-docker
          mountPath: /etc/docker/certs.d/my-private-insecure-registry.com
        - name: ca-cert
          mountPath: /home/core
      terminationGracePeriodSeconds: 30
      volumes:
      - name: etc-docker
        hostPath:
          path: /etc/docker/certs.d/my-private-insecure-registry.com
      - name: ca-cert
        secret:
          secretName: registry-ca

Save the file as registry-ca-ds.yaml and then create the DaemonSet :

kubectl create -f registry-ca-ds.yaml

You can now check that your application correctly pulls from your private self-signed registry.

As mentioned, the certificate will be added to new nodes' docker in an automatic fashion by the registry-ca DaemonSet. If you want to avoid this, simply delete the DaemonSet :

kubectl delete ds registry-ca --namespace kube-system

I think this is more secure than setting the insecure-registries flag of the docker daemon. Also, it is resilient to new nodes.

Original solution for Kubernetes prior to 1.16

As suggested by @MarcusMaxwell, this answer has been edited to take into account the deprecation of the API extensions/v1beta1 for Kubernetes 1.16+ clusters. If you still run a Kubernetes cluster with version prior to 1.16, you should adapt this code instead:

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: registry-ca
  namespace: kube-system
  labels:
    k8s-app: registry-ca
spec:
  template:
    metadata:
      labels:
        name: registry-ca
    spec:
      containers:
      - name: registry-ca
        image: busybox
        command: [ 'sh' ]
        args: [ '-c', 'cp /home/core/registry-ca /etc/docker/certs.d/my-private-insecure-registry.com/ca.crt && exec tail -f /dev/null' ]
        volumeMounts:
        - name: etc-docker
          mountPath: /etc/docker/certs.d/my-private-insecure-registry.com
        - name: ca-cert
          mountPath: /home/core
      terminationGracePeriodSeconds: 30
      volumes:
      - name: etc-docker
        hostPath:
          path: /etc/docker/certs.d/my-private-insecure-registry.com
      - name: ca-cert
        secret:
          secretName: registry-ca

Known limitations (edit 2021)

According to these Kubernetes answers (here and here) to related github issues, a Kubernetes volume path cannot contain a colon. Therefore, this solution is not valid for registries that communicate securely with a self-signed certificate on specific ports (for example 5000).

In such case, please see Gary Plattenburg's answer to create the directory in the shell command instead of using Kubernetes to handle it during the mount.

Answer 2

The accepted answer from Bichon fro 1.16+ has in the comments that it does not work for URLs on a port due to not being able to mount the path with a colon (:) in it. This can be fixed by mounting the parent directory and changing the command arguments to make it.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: registry-ca
  namespace: kube-system
  labels:
    k8s-app: registry-ca
spec:
  selector:
    matchLabels:
      name: registry-ca
  template:
    metadata:
      labels:
        name: registry-ca
    spec:
      containers:
      - name: registry-ca
        image: busybox
        command: [ 'sh' ]
        args: [ '-c', 'mkdir /etc/docker/certs.d/my-private-insecure-registry.com:5000 && cp /home/core/registry-ca /etc/docker/certs.d/my-private-insecure-registry.com:5000/ca.crt && exec tail -f /dev/null' ]
        volumeMounts:
        - name: etc-docker
          mountPath: /etc/docker/certs.d
        - name: ca-cert
          mountPath: /home/core
      terminationGracePeriodSeconds: 30
      volumes:
      - name: etc-docker
        hostPath:
          path: /etc/docker/certs.d
      - name: ca-cert
        secret:
          secretName: registry-ca