microk8s.enable dns gets stuck in ContainerCreating

Question

I have installed microk8s snap on Ubuntu 19 in a VBox. When I run microk8s.enable dns, the pod for the deployment does not get past ContainerCreating state.

I used to work in before. I have also re-installed microk8s, this helped in the passed, but not anymore.

n.a.

Output from microk8s.kubectl get all --all-namespaces shows that something is wrong with the volume for the secrets. I don't know how I can investigate further, so any help is appreciated.

Cheers

NAMESPACE     NAME                          READY   STATUS              RESTARTS   AGE
kube-system   pod/coredns-9b8997588-z88lz   0/1     ContainerCreating   0          16m

NAMESPACE     NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                  AGE
default       service/kubernetes   ClusterIP   10.152.183.1    <none>        443/TCP                  20m
kube-system   service/kube-dns     ClusterIP   10.152.183.10   <none>        53/UDP,53/TCP,9153/TCP   16m

NAMESPACE     NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
kube-system   deployment.apps/coredns   0/1     1            0           16m

NAMESPACE     NAME                                DESIRED   CURRENT   READY   AGE
kube-system   replicaset.apps/coredns-9b8997588   1         1         0       16m

Output from microk8s.kubectl describe pod/coredns-9b8997588-z88lz -n kube-system

Name:                 coredns-9b8997588-z88lz
Namespace:            kube-system
Priority:             2000000000
Priority Class Name:  system-cluster-critical
Node:                 peza-ubuntu-19/10.0.2.15
Start Time:           Sun, 29 Sep 2019 15:49:27 +0200
Labels:               k8s-app=kube-dns
                      pod-template-hash=9b8997588
Annotations:          scheduler.alpha.kubernetes.io/critical-pod: 
Status:               Pending
IP:                   
IPs:                  <none>
Controlled By:        ReplicaSet/coredns-9b8997588
Containers:
  coredns:
    Container ID:  
    Image:         coredns/coredns:1.5.0
    Image ID:      
    Ports:         53/UDP, 53/TCP, 9153/TCP
    Host Ports:    0/UDP, 0/TCP, 0/TCP
    Args:
      -conf
      /etc/coredns/Corefile
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Limits:
      memory:  170Mi
    Requests:
      cpu:        100m
      memory:     70Mi
    Liveness:     http-get http://:8080/health delay=60s timeout=5s period=10s #success=1 #failure=5
    Readiness:    http-get http://:8181/ready delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:  <none>
    Mounts:
      /etc/coredns from config-volume (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from coredns-token-h6qlm (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  config-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      coredns
    Optional:  false
  coredns-token-h6qlm:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  coredns-token-h6qlm
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     CriticalAddonsOnly
                 node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason       Age                    From                     Message
  ----     ------       ----                   ----                     -------
  Normal   Scheduled    <unknown>              default-scheduler        Successfully assigned kube-system/coredns-9b8997588-z88lz to peza-ubuntu-19
  Warning  FailedMount  5m59s                  kubelet, peza-ubuntu-19  Unable to attach or mount volumes: unmounted volumes=[coredns-token-h6qlm config-volume], unattached volumes=[coredns-token-h6qlm config-volume]: timed out waiting for the condition
  Warning  FailedMount  3m56s (x11 over 10m)   kubelet, peza-ubuntu-19  MountVolume.SetUp failed for volume "coredns-token-h6qlm" : failed to sync secret cache: timed out waiting for the condition
  Warning  FailedMount  3m44s (x2 over 8m16s)  kubelet, peza-ubuntu-19  Unable to attach or mount volumes: unmounted volumes=[config-volume coredns-token-h6qlm], unattached volumes=[config-volume coredns-token-h6qlm]: timed out waiting for the condition
  Warning  FailedMount  113s (x12 over 10m)    kubelet, peza-ubuntu-19  MountVolume.SetUp failed for volume "config-volume" : failed to sync configmap cache: timed out waiting for the condition

Answer 1

I spent my morning fighting with this on ubuntu 19.04. None of the microk8s add-ons worked. Their containers got stuck in "ContainerCreating" status having something like "MountVolume.SetUp failed for volume "kubernetes-dashboard-token-764ml" : failed to sync secret cache: timed out waiting for the condition" in their descriptions.

I tried to start/stop/reset/reinstall microk8s a few times. Nothing worked. Once I downgraded it to the prev version the problem went away.

sudo snap install microk8s --classic --channel=1.15/stable