1 node(s) had taints that the pod didn't tolerate in kubernetes cluster

Question

Today my kubernetes cluster v1.15.2 give me this error: 1 node(s) had taints that the pod didn't tolerate and the pods could not start.

It tells me one nodes have taints and I check the node status and works fine, how to know it exactly have taints?

I am searching from internet and all tells me that master node could not allocate for pods running by default. But now my kubernetes pods is not running a master node.

• What may cause my node have taints(for example this node have not enough resource)?
• What should I do to find out the taints of the node and fix it?

Answer 1

By default master node is tainted (means no pod or workload will be scheduled on master node. and this is best practices because master node meant to run cluster component like ETCD, kubeapi-server etc. and all other application related pods should go onto worker nodes ) so that's why by default taint applied on master node. Taints and toleration work together to ensure that pods are not scheduled onto inappropriate nodes. One or more taints are applied to a node.

To check if node has taint or not

kubectl describe node <nodename> | grep Taints

and you will get something like this if any taint present on node

node-role.kubernetes.io/master:NoSchedule

If you want to keep the taint on node as it is and still you want to schedule particular pod on that node then include this in your pod/deployment.yaml file.

tolerations:
- key: "key"
  operator: "Exists"
  effect: "NoSchedule"

To get more info about this check this section https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/

and If you want to remove taint from that node then follow these steps

First check the taint present or not with nodename

kubectl describe node <nodename> | grep Taints

and you will get something like this (master or worker_node)

node-role.kubernetes.io/master:NoSchedule

To remove taint from node just run like this (here in my case it is master node)

kubectl taint node master node-role.kubernetes.io/master:NoSchedule-

Make sure you add - in front of NoSchedule

Answer 2

You can use kubectl describe node <nodename> to check taints.

kubectl describe node masternode
Name:               masternode
Roles:              master
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/os=linux
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=ip-10-0-0-115
                    kubernetes.io/os=linux
                    node-role.kubernetes.io/master=
Annotations:        kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock
                    node.alpha.kubernetes.io/ttl: 0
                    projectcalico.org/IPv4Address: 10.0.0.115/24
                    projectcalico.org/IPv4IPIPTunnelAddr: 192.168.217.0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Thu, 18 Jun 2020 10:21:48 +0530
Taints:             node-role.kubernetes.io/master:NoSchedule

The node controller automatically taints a Node when certain conditions are true. The following taints are built in:

node.kubernetes.io/not-ready: Node is not ready. This corresponds to the NodeCondition Ready being "False".

node.kubernetes.io/unreachable: Node is unreachable from the node controller. This corresponds to the NodeCondition Ready being "Unknown".

node.kubernetes.io/out-of-disk: Node becomes out of disk.

node.kubernetes.io/memory-pressure: Node has memory pressure.

node.kubernetes.io/disk-pressure: Node has disk pressure.

node.kubernetes.io/network-unavailable: Node's network is unavailable.

node.kubernetes.io/unschedulable: Node is unschedulable.

node.cloudprovider.kubernetes.io/uninitialized: When the kubelet is started with "external" cloud provider, this taint is set on a node to mark it as unusable. After a controller from the cloud-controller-manager initializes this node, the kubelet removes this taint.

Along with above a special taint node-role.kubernetes.io/master:NoSchedule is added by default to master nodes.

The error typically comes if there is a taint on nodes for which you don't have corresponding toleration in pod spec.

Below is an example pod with toleration.

apiVersion: v1
kind: Pod
metadata:
  name: nginx
  labels:
    env: test
spec:
  containers:
  - name: nginx
    image: nginx
    imagePullPolicy: IfNotPresent
  tolerations:
  - key: "example-key"
    operator: "Exists"
    effect: "NoSchedule"

Answer 3

kubectl describe node nodename | grep Taints

kubectl taint node master node-role.kubernetes.io/master:NoSchedule-

This one works fine