How to restart master node in kubernetes

Question

I have a kubernetes cluster with 3 masters and 3 workers, I want to restart one of the masters to update the system of the master machine.

So can I just reboot the machine directly on the console with reboot, or some steps need to be done before the reboot to void the risk of out of service and data loss?

Answer 1

If you need to reboot a node (such as for a kernel upgrade, libc upgrade, hardware repair, etc.), and the downtime is brief, then when the Kubelet restarts, it will attempt to restart the pods scheduled to it. If the reboot takes longer (the default time is 5 minutes, controlled by --pod-eviction-timeout on the controller-manager), then the node controller will terminate the pods that are bound to the unavailable node. If there is a corresponding replica set (or replication controller), then a new copy of the pod will be started on a different node. So, in the case where all pods are replicated, upgrades can be done without special coordination, assuming that not all nodes will go down at the same time

If you want more control over the upgrading process, you may use the following workflow:

Use kubectl drain to gracefully terminate all pods on the node while marking the node as unschedulable:

kubectl drain $NODENAME

This keeps new pods from landing on the node while you are trying to get them off. For pods with a replica set, the pod will be replaced by a new pod which will be scheduled to a new node. Additionally, if the pod is part of a service, then clients will automatically be redirected to the new pod. For pods with no replica set, you need to bring up a new copy of the pod, and assuming it is not part of a service, redirect clients to it. Perform maintenance work on the node. Make the node schedulable again:

kubectl uncordon $NODENAME

Additionally if the node is hosting ETCD then you need to be extra careful in terms of rolling upgrade of ETCD and backing up the data