Kube-state-metrics error: Failed to create client: ... i/o timeout

Question

I'm running Kubernetes in virtual machines and going through the basic tutorials, currently Add logging and metrics to the PHP / Redis Guestbook example. I'm trying to install kube-state-metrics:

git clone https://github.com/kubernetes/kube-state-metrics.git kube-state-metrics
kubectl create -f kube-state-metrics/kubernetes

but it fails.

kubectl describe pod --namespace kube-system kube-state-metrics-7d84474f4d-d5dg7

...

Warning Unhealthy 28m (x8 over 30m) kubelet, kubernetes-node1 Readiness probe failed: Get http://192.168.129.102:8080/healthz: dial tcp 192.168.129.102:8080: connect: connection refused

kubectl logs --namespace kube-system kube-state-metrics-7d84474f4d-d5dg7 -c kube-state-metrics

I0514 17:29:26.980707 1 main.go:85] Using default collectors
I0514 17:29:26.980774 1 main.go:93] Using all namespace
I0514 17:29:26.980780 1 main.go:129] metric white-blacklisting: blacklisting the following items:
W0514 17:29:26.980800 1 client_config.go:549] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
I0514 17:29:26.983504 1 main.go:169] Testing communication with server
F0514 17:29:56.984025 1 main.go:137] Failed to create client: ERROR communicating with apiserver: Get https://10.96.0.1:443/version?timeout=32s: dial tcp 10.96.0.1:443: i/o timeout

I'm unsure if this 10.96.0.1 IP is correct. My virtual machines are in a bridged network 10.10.10.0/24 and a host-only network 192.168.59.0/24. When initializing Kubernetes I used the argument --pod-network-cidr=192.168.0.0/16 so that's one more IP range that I'd expect. But 10.96.0.1 looks unfamiliar.

I'm new to Kubernetes, just doing the basic tutorials, so I don't know what to do now. How to fix it or investigate further?

---

EDIT - additonal info:

kubectl get nodes -o wide

NAME                STATUS   ROLES    AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
kubernetes-master   Ready    master   15d   v1.14.1   10.10.10.11   <none>        Ubuntu 18.04.2 LTS   4.15.0-48-generic   docker://18.9.2
kubernetes-node1    Ready    <none>   15d   v1.14.1   10.10.10.5    <none>        Ubuntu 18.04.2 LTS   4.15.0-48-generic   docker://18.9.2
kubernetes-node2    Ready    <none>   15d   v1.14.1   10.10.10.98   <none>        Ubuntu 18.04.2 LTS   4.15.0-48-generic   docker://18.9.2

The command I used to initialize the cluster:

sudo kubeadm init --apiserver-advertise-address=192.168.59.20 --pod-network-cidr=192.168.0.0/16

Answer 1

The reason for this is probably overlapping of Pod network with Node network - you set Pod network CIDR to 192.168.0.0/16 which your host-only network will be included into as its address is 192.168.59.0/24.

To solve this you can either change the pod network CIDR to 192.168.0.0/24 (it is not recommended as this will give you only 255 addresses for your pod networking)

You can also use different range for your Calico. If you want to do it on a running cluster here is an instruction.

Also other way I tried:

edit Calico manifest to different range (for example 10.0.0.0/8) - sudo kubeadm init --apiserver-advertise-address=192.168.59.20 --pod-network-cidr=10.0.0.0/8) and apply it after the init.

Another way would be using different CNI like Flannel (which uses 10.244.0.0/16).

You can find more information about ranges of CNI plugins here.