Kubernetes- error uploading crisocket: timed out waiting for the condition

Question

I am trying to create a template for a Kubernetes cluster having 1 master and 2 worker nodes. I have installed all the pre-req software and have run the kubeadmn init on my master node. But when i try to run the kubeadmn join which i get as an output of the init command i am getting an error.

[discovery] Created cluster-info discovery client, requesting info
from "https://10.31.2.33:6443" [discovery] Requesting info from
"https://10.31.2.33:6443" again to validate TLS against the pinned
public key [discovery] Cluster info signature and contents are valid
and TLS certificate validates against pinned roots, will use API
Server "10.31.2.33:6443" [discovery] Successfully established
connection with API Server "10.31.2.33:6443" [kubelet] Downloading
configuration for the kubelet from the "kubelet-config-1.12" ConfigMap
in the kube-system namespace [kubelet] Writing kubelet configuration
to file "/var/lib/kubelet/config.yaml" [kubelet] Writing kubelet
environment file with flags to file
"/var/lib/kubelet/kubeadm-flags.env" [preflight] Activating the
kubelet service [tlsbootstrap] Waiting for the kubelet to perform the
TLS Bootstrap... [patchnode] Uploading the CRI Socket information
"/var/run/dockershim.sock" to the Node API object "<workernode2>" as
an annotation error uploading crisocket: timed out waiting for the
condition```

I have done a swapoff -a before running this on the workdernode2

I was able to run the join once but after that, as a part of a script, I ran the kubeadmn reset followed by init and join few times where this has started showing up.

Not able to figure out what or where I am doing a mistake.

My main intent is to put all the commands in the form of a shell script (on master node) so that it can be run on a cluster to create a network.

Answer 1

I had the encountered the following issue after node was rebooted:

[kubelet] Creating a ConfigMap "kubelet-config-1.13" in namespace kube-system with the configuration for the kubelets in the cluster
[patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "k8smaster" as an annotation
[kubelet-check] Initial timeout of 40s passed.
error execution phase upload-config/kubelet: Error writing Crisocket information for the control-plane node: timed out waiting for the condition

Steps to get rid of this issue:

1. Check the hostname again, after reboot it might have changed.

   sudo vi /etc/hostname 
   sudo vi /etc/hosts
   

2. Perform the following clean-up actions

   Code:

   sudo kubeadm reset
   rm -rf /var/lib/cni/
   sudo rm -rf /var/lib/cni/
   
   systemctl daemon-reload
   
   systemctl restart kubelet
   
   sudo iptables -F && sudo iptables -t nat -F && sudo iptables -t mangle -F && sudo iptables -X
   

3. Execute the init action with the special tag as below

   Code:

   sudo kubeadm init --pod-network-cidr=192.168.0.0/16 --apiserver-advertise-address=10.10.10.2 --ignore-preflight-errors=all    
   

   (where 10.10.10.2 is the IP of master node and 192.168.0.0/16 is the private subnet assigned for Pods)

Answer 2

I’ve had the same problem on Ubuntu 16.04 amd64, fixed it with these commands:

swapoff -a    # will turn off the swap 
kubeadm reset
systemctl daemon-reload
systemctl restart kubelet
iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X  # will reset iptables

Also, look at that issue in kubeadm GitHub kubeadm swap on the related issue where people still report having the problem after turning swap off.

You may also try to add --fail-swap-on=false flag in /etc/default/kubelet file, it didn't help in my case though.

It seems to be fixed in the latest k8 version because after upgrading the cluster, I haven't experienced the issue.