Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to debug when Kubernetes nodes are in 'Not Ready' state

Tags:

kubernetes

I initialized the master node and add 2 worker nodes, but only master and one of the worker node show up when I run the following command:

kubectl get nodes 

also, both these nodes are in 'Not Ready' state. What are the steps should I take to understand what the problem could be?

  • I can ping all the nodes from each of the other nodes.
  • The version of Kubernetes is 1.8.
  • OS is Cent OS 7
  • I used the following repo to install Kubernetes:

    cat <<EOF > /etc/yum.repos.d/kubernetes.repo [kubernetes] name=Kubernetes baseurl=http://yum.kubernetes.io/repos/kubernetes-el7-x86_64 enabled=1 gpgcheck=0 repo_gpgcheck=0 EOF  yum install kubelet kubeadm kubectl kubernetes-cni 
like image 252
lex Avatar asked Nov 04 '17 03:11

lex


People also ask

Why my deployment is not ready in Kubernetes?

If a Pod is Running but not Ready it means that the Readiness probe is failing. When the Readiness probe is failing, the Pod isn't attached to the Service, and no traffic is forwarded to that instance.

How can I check my Kubernetes node status?

The kubectl command is used to show the detailed status of the Kubernetes pods deployed to run the PowerAI Vision application. When the application is running correctly, each of the pods should have: A value of 1/1 in the READY column. A value of Running in the STATUS column.

What happens if Kubernetes node fails?

Irrespective of deployments (StatefuleSet or Deployment), Kubernetes will automatically evict the pod on the failed node and then try to recreate a new one with old volumes. If the node is back online within 5 – 6 minutes of the failure, Kubernetes will restart pods, unmount, and re-mount volumes.


1 Answers

First, describe nodes and see if it reports anything:

$ kubectl describe nodes

Look for conditions, capacity and allocatable:

Conditions:   Type              Status   ----              ------   OutOfDisk         False   MemoryPressure    False   DiskPressure      False   Ready             True Capacity:  cpu:       2  memory:    2052588Ki  pods:      110 Allocatable:  cpu:       2  memory:    1950188Ki  pods:      110 

If everything is alright here, SSH into the node and observe kubelet logs to see if it reports anything. Like certificate erros, authentication errors etc.

If kubelet is running as a systemd service, you can use

$ journalctl -u kubelet

like image 78
Shahidh Avatar answered Oct 20 '22 04:10

Shahidh