microk8s pulling image, stuck in ContainerCreating state

Question

As described in the doc (based on).

I am on Ubuntu as a host OS.

docker --version

Docker version 18.09.6, build 481bc77

microk8s 1.14/beta

---

Enable local registry for microk2s:

microk8s.enable registry

Checking:

watch microk8s.kubectl get all --all-namespaces

container-registry pod/registry-577986746b-v8xqc 1/1 Running 0 36m

Then:

Edit:

sudo vim /etc/docker/daemon.json

Add this content:

{
  "insecure-registries" : ["127.0.0.1:32000"]
}

Restart:

sudo systemctl restart docker

Double checking, see if insecure is applied:

docker info | grep -A 2 Insecure

Insecure Registries:
 127.0.0.1:32000
 127.0.0.0/8
WARNING: No swap limit support

Tag:

docker tag my-registry/my-services/my-service:0.0.1-SNAPSHOT 127.0.0.1:32000/my-service

Checking:

docker images

127.0.0.1:32000/my-service latest e68f8a7e4675 19 hours ago 540MB

Pushing:

docker push 127.0.01:32000/my-service

I see my image here: http://127.0.0.1:32000/v2/_catalog

In deployment-local.yml I have, now the proper image:

...spec:
  containers:
    - name: my-service-backend
      image: 127.0.0.1:32000/my-service
      imagePullPolicy: Always ...

Then applying:

envsubst < ./.local/deployment-local.yml | microk8s.kubectl apply -f -

I see: ContainerCreating

By: microk8s.kubectl describe pods my-service-deployment-f85d5dcd5-cmd5

In the Events section:

Events:
  Type     Reason     Age                 From                     Message
  ----     ------     ----                ----                     -------
  Normal   Scheduled  107s                default-scheduler        Successfully assigned default/my-service-deployment-f85d5dcd5-z75tr to my-desktop
  Normal   Pulling    25s (x4 over 106s)  kubelet, my-desktop  Pulling image "127.0.0.1:32000/my-service"
  Warning  Failed     25s (x4 over 106s)  kubelet, my-desktop  Failed to pull image "127.0.0.1:32000/my-service": rpc error: code = Unknown desc = failed to resolve image "127.0.0.1:32000/my-service:latest": no available registry endpoint: failed to do request: Head https://127.0.0.1:32000/v2/my-service/manifests/latest: http: server gave HTTP response to HTTPS client
  Warning  Failed     25s (x4 over 106s)  kubelet, my-desktop  Error: ErrImagePull
  Normal   BackOff    10s (x5 over 105s)  kubelet, my-desktop  Back-off pulling image "127.0.0.1:32000/my-service"
  Warning  Failed     10s (x5 over 105s)  kubelet, my-desktop  Error: ImagePullBackOff

Seems like my-service is stuck there.

Q: What could be the reason?

UPDATE: changing all to localhost helped, meaning I could see now in the browser: http://localhost:32000/v2/_catalog .

{"repositories":["my-service"]}

But it worked only in Firefox.. weird. In chrome is pending..

Still:

Events:
  Type     Reason          Age                    From                    Message
  ----     ------          ----                   ----                    -------
  Normal   Scheduled       161m                   default-scheduler       Successfully assigned default/my-service-deployment-6d4c5587df-72xvd to my-laptop
  Normal   Pulling         160m (x4 over 161m)    kubelet, my-laptop  Pulling image "localhost/my-service"
  Warning  Failed          160m (x4 over 161m)    kubelet, my-laptop  Failed to pull image "localhost/my-service": rpc error: code = Unknown desc = failed to resolve image "localhost/my-service:latest": no available registry endpoint: failed to do request: Head https://localhost/v2/my-service/manifests/latest: dial tcp 127.0.0.1:443: connect: connection refused
  Warning  Failed          160m (x4 over 161m)    kubelet, my-laptop  Error: ErrImagePull
  Warning  Failed          159m (x6 over 161m)    kubelet, my-laptop  Error: ImagePullBackOff
  Normal   BackOff         131m (x132 over 161m)  kubelet, my-laptop  Back-off pulling image "localhost/my-service"
  Normal   SandboxChanged  22m                    kubelet, my-laptop  Pod sandbox changed, it will be killed and re-created.
  Normal   Pulling         21m (x4 over 22m)      kubelet, my-laptop  Pulling image "localhost/my-service"
  Warning  Failed          21m (x4 over 22m)      kubelet, my-laptop  Failed to pull image "localhost/my-service": rpc error: code = Unknown desc = failed to resolve image "localhost/my-service:latest": no available registry endpoint: failed to do request: Head https://localhost/v2/my-service/manifests/latest: dial tcp 127.0.0.1:443: connect: connection refused
  Warning  Failed          21m (x4 over 22m)      kubelet, my-laptop  Error: ErrImagePull
  Warning  Failed          20m (x6 over 22m)      kubelet, my-laptop  Error: ImagePullBackOff
  Normal   BackOff         2m22s (x87 over 22m)   kubelet, my-laptop  Back-off pulling image "localhost/my-service"

Seems it tries to connect by https..

--

Ok.. had to add the port for the yml image: part. will accept the answer below. Thanks.

Answer 1

In microk8s is a big different between localhost and 127.0.0.1. As you mentioned, you were based on this Stack case and there user were using localhost.

The main issue here is error no available registry endpoint: failed to do request:. It means microk8s cannot find endpoint.

If you will run commend below, you will have the same output

$ cat /var/snap/microk8s/current/args/containerd-template.toml | grep endpoint

endpoint = ["https://registry-1.docker.io"]
endpoint = ["http://localhost:32000"]

As you see in microk8s endpoint is localhost not 127.0.0.1

Please change image in deployment-local.yaml

...spec:
  containers:
    - name: my-service-backend
      image: localhost/my-service
      imagePullPolicy: Always ...

Let me know if you will occur any other issues.

In my case I did not need tag and push image again with localhost, only change in YAML was required.

Answer 2

I also needed to comment out following line in the /etc/hosts (Ubuntu 18.04)

::1 localhost ip6-localhost

It seems that microk8s isn't capable of IPv6.