Kubernetes pods can't ping each other using ClusterIP

Question

I'm trying to ping the kube-dns service from a dnstools pod using the cluster IP assigned to the kube-dns service. The ping request times out. From the same dnstools pod, I tried to curl the kube-dns service using the exposed port, but that timed out as well.

Following is the output of kubectl get pods --all-namespaces -o wide

NAMESPACE       NAME                                      READY   STATUS    RESTARTS   AGE     IP             NODE          NOMINATED NODE
default         pod/busybox                               1/1     Running   62         2d14h   192.168.1.37   kubenode      <none>
default         pod/dnstools                              1/1     Running   0          2d13h   192.168.1.45   kubenode      <none>
default         pod/nginx-deploy-7c45b84548-ckqzb         1/1     Running   0          6d11h   192.168.1.5    kubenode      <none>
default         pod/nginx-deploy-7c45b84548-vl4kh         1/1     Running   0          6d11h   192.168.1.4    kubenode      <none>
dmi             pod/elastic-deploy-5d7c85b8c-btptq        1/1     Running   0          2d14h   192.168.1.39   kubenode      <none>
kube-system     pod/calico-node-68lc7                     2/2     Running   0          6d11h   10.62.194.5    kubenode      <none>
kube-system     pod/calico-node-9c2jz                     2/2     Running   0          6d12h   10.62.194.4    kubemaster    <none>
kube-system     pod/coredns-5c98db65d4-5nprd              1/1     Running   0          6d12h   192.168.0.2    kubemaster    <none>
kube-system     pod/coredns-5c98db65d4-5vw95              1/1     Running   0          6d12h   192.168.0.3    kubemaster    <none>
kube-system     pod/etcd-kubemaster                       1/1     Running   0          6d12h   10.62.194.4    kubemaster    <none>
kube-system     pod/kube-apiserver-kubemaster             1/1     Running   0          6d12h   10.62.194.4    kubemaster    <none>
kube-system     pod/kube-controller-manager-kubemaster    1/1     Running   1          6d12h   10.62.194.4    kubemaster    <none>
kube-system     pod/kube-proxy-9hcgv                      1/1     Running   0          6d11h   10.62.194.5    kubenode      <none>
kube-system     pod/kube-proxy-bxw9s                      1/1     Running   0          6d12h   10.62.194.4    kubemaster    <none>
kube-system     pod/kube-scheduler-kubemaster             1/1     Running   1          6d12h   10.62.194.4    kubemaster    <none>
kube-system     pod/tiller-deploy-767d9b9584-5k95j        1/1     Running   0          3d9h    192.168.1.8    kubenode      <none>
nginx-ingress   pod/nginx-ingress-66wts                   1/1     Running   0          5d17h   192.168.1.6    kubenode      <none>

In the above output, why do some pods have an IP assigned in the 192.168.0.0/24 subnet whereas others have an IP that is equal to the IP address of my node/master? (10.62.194.4 is the IP of my master, 10.62.194.5 is the IP of my node)

This is the config.yml I used to initialize the cluster using kubeadm init --config=config.yml

apiServer:
certSANs:
- 10.62.194.4
extraArgs:
    authorization-mode: Node,RBAC
timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta2
certificatesDir: /etc/kubernetes/pki
clusterName: dev-cluster
controllerManager: {}
dns:
type: CoreDNS
etcd:
local:
    dataDir: /var/lib/etcd
imageRepository: k8s.gcr.io
kind: ClusterConfiguration
kubernetesVersion: v1.15.1
networking:
dnsDomain: cluster.local
podSubnet: 192.168.0.0/16
serviceSubnet: 10.96.0.0/12
scheduler: {}

Result of kubectl get svc --all-namespaces -o wide

NAMESPACE     NAME                    TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                  AGE     SELECTOR
default       service/kubernetes      ClusterIP   10.96.0.1       <none>        443/TCP                  6d12h   <none>
default       service/nginx-deploy    ClusterIP   10.97.5.194     <none>        80/TCP                   5d17h   run=nginx
dmi           service/elasticsearch   ClusterIP   10.107.84.159   <none>        9200/TCP,9300/TCP        2d14h   app=dmi,component=elasticse
dmi           service/metric-server   ClusterIP   10.106.117.2    <none>        8098/TCP                 2d14h   app=dmi,component=metric-se
kube-system   service/calico-typha    ClusterIP   10.97.201.232   <none>        5473/TCP                 6d12h   k8s-app=calico-typha
kube-system   service/kube-dns        ClusterIP   10.96.0.10      <none>        53/UDP,53/TCP,9153/TCP   6d12h   k8s-app=kube-dns
kube-system   service/tiller-deploy   ClusterIP   10.98.133.94    <none>        44134/TCP                3d9h    app=helm,name=tiller

The command I ran was kubectl exec -ti dnstools -- curl 10.96.0.10:53

EDIT:

I raised this question because I got this error when trying to resolve service names from within the cluster. I was under the impression that I got this error because I cannot ping the DNS server from a pod.

Output of kubectl exec -ti dnstools -- nslookup kubernetes.default

;; connection timed out; no servers could be reached

command terminated with exit code 1

Output of kubectl exec dnstools cat /etc/resolv.conf

nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local reddog.microsoft.com
options ndots:5

Result of kubectl get ep kube-dns --namespace=kube-system

NAME       ENDPOINTS                                                  AGE
kube-dns   192.168.0.2:53,192.168.0.3:53,192.168.0.2:53 + 3 more...   6d13h

EDIT: Ping-ing the CoreDNS pod directly using its Pod IP times out as well:

/ # ping 192.168.0.2
PING 192.168.0.2 (192.168.0.2): 56 data bytes
^C
--- 192.168.0.2 ping statistics ---
24 packets transmitted, 0 packets received, 100% packet loss

EDIT: I think something has gone wrong when I was setting up the cluster. Below are the steps I took when setting up the cluster:

1. Edit host files on master and worker to include the IP's and hostnames of the nodes
2. Disabled swap using swapoff -a and disabled swap permanantly by editing /etc/fstab
3. Install docker prerequisites using apt-get install apt-transport-https ca-certificates curl software-properties-common -y
4. Added Docker GPG key using curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
5. Added Docker repo using add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
6. Install Docker using apt-get update -y; -get install docker-ce -y
7. Install Kubernetes prerequisites using curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
8. Added Kubernetes repo using echo 'deb http://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list
9. Update repo and install Kubernetes components using apt-get update -y; apt-get install kubelet kubeadm kubectl -y

Configure master node:

1. kubeadm init --apiserver-advertise-address=10.62.194.4 --apiserver-cert-extra-sans=10.62.194.4 --pod-network-cidr=192.168.0.0/16
2. Copy Kube config to $HOME: mkdir -p $HOME/.kube; sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config; sudo chown $(id -u):$(id -g) $HOME/.kube/config
3. Installed Calico using kubectl apply -f https://docs.projectcalico.org/v3.1/getting-started/kubernetes/installation/hosted/rbac-kdd.yaml; kubectl apply -f https://docs.projectcalico.org/v3.1/getting-started/kubernetes/installation/hosted/kubernetes-datastore/calico-networking/1.7/calico.yaml

On node: On the node I did the kubeadm join command using the command printed out from kubeadm token create --print-join-command on the master

Answer 1

The kubernetes system pods get assigned the host ip since they provide low level services that are not dependant on an overlay network (or in case of calico even provide the overlay network). They have the ip of the node where they run.

A common pod uses the overlay network and gets assigned an ip from the calico range, not from the metal node they run on.

You can't access DNS (port 53) with HTTP using curl. You can use dig to query a DNS resolver. A service IP is not reachable by ping since it is a virtual IP just used as a routing handle for the iptables rules setup by kube-proxy, therefore a TCP connection works, but ICMP not. You can ping a pod IP though, since it is assigned from the overlay network.