Renew kubernetes pki after expired

Question

My kubernetes PKI expired (API server to be exact) and I can't find a way to renew it. The error I get is

May 27 08:43:51 node1 kubelet[8751]: I0527 08:43:51.922595    8751 server.go:417] Version: v1.14.2
May 27 08:43:51 node1 kubelet[8751]: I0527 08:43:51.922784    8751 plugins.go:103] No cloud provider specified.
May 27 08:43:51 node1 kubelet[8751]: I0527 08:43:51.922800    8751 server.go:754] Client rotation is on, will bootstrap in background
May 27 08:43:51 node1 kubelet[8751]: E0527 08:43:51.925859    8751 bootstrap.go:264] Part of the existing bootstrap client certificate is expired: 2019-05-24 13:24:42 +0000 UTC
May 27 08:43:51 node1 kubelet[8751]: F0527 08:43:51.925894    8751 server.go:265] failed to run Kubelet: unable to load bootstrap
kubeconfig: stat /etc/kubernetes/bootstrap-kubelet.conf: no such file or directory

The documentation on https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/ describes how to renew but it only works if the API server is not expired. I have tried to do a

kubeadm alpha cert renew all

and do a reboot but that just made the entire cluster fail so I did a rollback to a snapshot (my cluster is running on VMware).

The cluster is running and all containers seem to work but I can't access it via kubectl so I can't really deploy or query.

My kubernetes version is 1.14.2.

Answer 1

So the solution was to (first a backup)

$ cd /etc/kubernetes/pki/
$ mv {apiserver.crt,apiserver-etcd-client.key,apiserver-kubelet-client.crt,front-proxy-ca.crt,front-proxy-client.crt,front-proxy-client.key,front-proxy-ca.key,apiserver-kubelet-client.key,apiserver.key,apiserver-etcd-client.crt} ~/
$ kubeadm init phase certs all --apiserver-advertise-address <IP>
$ cd /etc/kubernetes/
$ mv {admin.conf,controller-manager.conf,kubelet.conf,scheduler.conf} ~/
$ kubeadm init phase kubeconfig all
$ reboot

then

$ cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

that did the job for me and thanks for your hints :)

Answer 2

This topic is also discussed in:

• https://github.com/kubernetes/kubeadm/issues/581
  • after 1.15 kubeadm upgrade automatically will renewal the certificates for you!
  • also 1.15 added a command to check cert expiration in kubeadm
• Kubernetes: expired certificate

---

Kubernetes v1.15 provides docs for "Certificate Management with kubeadm":

• https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/
• Check certificate expiration:

kubeadm alpha certs check-expiration

• Automatic certificate renewal:
  • kubeadm renews all the certificates during control plane upgrade.
• Manual certificate renewal:
  • You can renew your certificates manually at any time with the kubeadm alpha certs renew command.
  • This command performs the renewal using CA (or front-proxy-CA) certificate and key stored in /etc/kubernetes/pki.

---

Overall for Kubernetes v1.14 I find this procedure the most helpful:

• https://stackoverflow.com/a/56334732/1147487

Answer 3

Try to do cert renewal via kubeadm init phase certs command.

You can check certs expiration via the following command:

openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -text

openssl x509 -in /etc/kubernetes/pki/apiserver-kubelet-client.crt -noout -text

First, ensure that you have most recent backup of k8s certificates inventory /etc/kubernetes/pki/*.

Delete apiserver.* and apiserver-kubelet-client.* cert files in /etc/kubernetes/pki/ directory.

Spawn a new certificates via kubeadm init phase certs command:

sudo kubeadm init phase certs apiserver

sudo kubeadm init phase certs apiserver-kubelet-client

Restart kubelet and docker daemons:

sudo systemctl restart docker; sudo systemctl restart kubelet

You can find more related information in the official K8s documentation.