Trying to pull/run docker images from docker hub on Minikube fails

Question

I am very new to Kuberetes and I have done some work with docker previously. I am trying to accomplish following:

1. Spin up Minikube
2. Use Kube-ctl to spin up a docker image from docker hub.

I started minikube and things look like they are up and running. Then I pass following command

kubectl run nginx --image=nginx (Please note I do not have this image anywhere on my machine and I am expecting k8 to fetch it for me)

Now, when I do that, it spins up the pod but the status is ImagePullBackOff. So I ran kubectl describe pod command on it and the results look like following:

Events:
  Type     Reason     Age              From               Message
  ----     ------     ----             ----               -------
  Normal   Scheduled  8m               default-scheduler  Successfully assigned default/ngix-67c6755c86-qm5mv to minikube
  Warning  Failed     8m               kubelet, minikube  Failed to pull image "nginx": rpc error: code = Unknown desc = Error response from daemon: Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io on 192.168.64.1:53: read udp 192.168.64.2:52133->192.168.64.1:53: read: connection refused
  Normal   Pulling    8m (x2 over 8m)  kubelet, minikube  Pulling image "nginx"
  Warning  Failed     8m (x2 over 8m)  kubelet, minikube  Error: ErrImagePull
  Warning  Failed     8m               kubelet, minikube  Failed to pull image "nginx": rpc error: code = Unknown desc = Error response from daemon: Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io on 192.168.64.1:53: read udp 192.168.64.2:40073->192.168.64.1:53: read: connection refused
  Normal   BackOff    8m (x3 over 8m)  kubelet, minikube  Back-off pulling image "nginx"
  Warning  Failed     8m (x3 over 8m)  kubelet, minikube  Error: ImagePullBackOff

Then I searched around to see if anyone has faced similar issues and it turned out that some people have and they did resolve it by restarting minikube using some more flags which look like below:

minikube start --vm-driver="xhyve" --insecure-registry="$REG_IP":80

when I do nslookup inside Minikube, it does resolve with following information:

Server:         10.12.192.22
Address:        10.12.192.22#53

Non-authoritative answer:
hub.docker.com  canonical name = elb-default.us-east-1.aws.dckr.io.
elb-default.us-east-1.aws.dckr.io       canonical name = us-east-1-elbdefau-1nlhaqqbnj2z8-140214243.us-east-1.elb.amazonaws.com.
Name:   us-east-1-elbdefau-1nlhaqqbnj2z8-140214243.us-east-1.elb.amazonaws.com
Address: 52.205.36.130
Name:   us-east-1-elbdefau-1nlhaqqbnj2z8-140214243.us-east-1.elb.amazonaws.com
Address: 3.217.62.246
Name:   us-east-1-elbdefau-1nlhaqqbnj2z8-140214243.us-east-1.elb.amazonaws.com
Address: 35.169.212.184

still no luck. Is there anything that I am doing wrong here?

Answer 1

There error message suggests that the Docker daemon running in the minikube VM can't resolve the registry-1.docker.io hostname because the DNS nameserver it's configured to use for DNS resolution (192.168.64.1:53) is refusing connection. It's strange to me that the Docker deamon is trying to resolve registry-1.docker.io via a nameserver at 192.168.64.1 but when you nslookup on the VM it's using a nameserver at 10.12.192.22. I did an Internet search for "minkube Get registry-1.docker.io/v2: dial tcp: lookup registry-1.docker.io on 192.168.64.1:53" and found an issue where someone made this comment, seems identical to your problem, and seems specific to xhyve.

In that comment the person says:

This issue does look like an xhyve issue not seen with virtualbox.

and

Switching to virtualbox fixed this issue for me.

I stopped minikube, deleted it, started it without --vm-driver=xhyve (minikube uses virtualbox driver by default), and then docker build -t hello-node:v1 . worked fine without errors

Answer 2

In my case it was caused by running dnsmasq, a dns server, on my Mac using Homebrew, which caused the DNS requests to fail inside minikube. After stopping dnsmasq, everything worked.