How can I allow a private insecure registry to be used inside a minikube node?

Question

I know there are about a thousand answers to various permutations of this question but none of the fifteen or so that I've tried have worked.

I'm running on Mac OS Sierra and using Minikube 0.17.1 as well as kubectl 1.5.3.

We run our own private Docker registry that is insecure as it is not open to the internet. (This is not my choice or in my control so it's not open for discussion). This is my first foray into Kubernetes and actually container orchestration altogether. I also have a very intermediate level of knowledge about Docker in general so I'm drowning in terminology/platform soup here.

When I execute

kubectl run perf-ui --image=X.X.X.X/performance/perf-ui:master

I see

image pull failed for X.X.X.X/performance/perf-ui:master, this may be because there are no credentials on this request. details: (Error response from daemon: Get https://X.X.X.X/v1/_ping: dial tcp X.X.X.X:443: getsockopt: connection refused)

We have an Ubuntu box that accesses the same registry (not using Kubernetes, just Docker) that works just fine. This is likely due to

DOCKER_OPTS="--insecure-registry X.X.X.X"

being in /etc/default/docker.

I made a similar change using the UI of Docker for Mac. I don't know where this change persisted in a config file. After this change a docker pull worked on my laptop!!! Again, this is just using Docker not Kubernetes. The interesting part is I got the same "Connection refused error" (as it tries to access via HTTPS) on my Mac as I get in the Minikube VM and after the change the pull worked. I feel like I'm on to something there.

After sshing into minikube (the VM created my minikube start) using

minikube ssh

I added the following content to /var/lib/boot2docker/profile

export EXTRA_ARGS="$EXTRA_ARGS --insecure-registry 10.129.100.3
export DOCKER_OPTS="$DOCKER_OPTS --insecure-registry 10.129.100.3

As you can infer, nothing has worked. I know I've tried other things but they too have failed.

I know this isn't the most comprehensive explanation but I've been digging into this for the past 4 hours.

The bottom line is docker pulls work from our Ubuntu box with the config file setup correctly and from my Mac with the setting configured properly.

How can I enable the same setting in my "Linux 2.6" VM that was created by Minikube?

If someone knows the answer I would be forever grateful.

Thank you in advance!

Answer 1

Thank you to Janos for your alternative solution. I'm confident that is the right choice for some use cases.

It turns out that what I needed was a good night sleep and the following command to start Minikube in the first place:

minikube start --insecure-registry="X.X.X.X"

@intelfx says that adding a port won't be necessary. I'm inclined to believe them but if your registry is on a non-standard port just keep it in mind in case things still aren't working.

In the end it was, in fact, a matter of telling Docker to use an insecure registry but it was not clear how to tell this to Docker when I was not controlling it directly.

I know that seems simple but after you've tried a hundred things you're almost hallucinating so you're not in a great state to make rational decisions. I'm sorry for the dumb post but I'm willing to bet this will help at least one person one day, which makes it worth it.

Thanks SO!

Answer 2

The flag --insecure-registry doesn't work on the existing cluster on MacOS. You need to do minikube delete, it's not enough just to stop the cluster with kubectl stop.

I spent plenty of time to figure this out and then I found this comment at https://github.com/kubernetes/minikube/issues/604:

the --insecure-registry flag is ignored if the machine already existed (even if it is stopped). You must first minikube delete if you want new flags to be respected.