Kubernetes: expired certificate

Question

Our Kubernetes 1.6 cluster had certificates generated when the cluster was built on April 13th, 2017.

On December 13th, 2017, our cluster was upgraded to version 1.8, and new certificates were generated [apparently, an incomplete set of certificates].

On April 13th, 2018, we started seeing this message within our Kubernetes dashboard for api-server:

[authentication.go:64] Unable to authenticate the request due to an error: [x509: certificate has expired or is not yet valid, x509: certificate has expired or is not yet valid]

Tried pointing client-certificate & client-key within /etc/kubernetes/kubelet.conf at the certificates generated on Dec 13th [apiserver-kubelet-client.crt and apiserver-kubelet-client.crt], but continue to see the above error.

Tried pointing client-certificate & client-key within /etc/kubernetes/kubelet.conf at different certificates generated on Dec 13th [apiserver.crt and apiserver.crt] (I honestly don't understand the difference between these 2 sets of certs/keys), but continue to see the above error.

Tried pointing client-certificate & client-key within /etc/kubernetes/kubelet.conf at non-existent files, and none of the kube* services would start, with /var/log/syslog complaining about this:

Apr 17 17:50:08 kuber01 kubelet[2422]: W0417 17:50:08.181326 2422 server.go:381] invalid kubeconfig: invalid configuration: [unable to read client-cert /tmp/this/cert/does/not/exist.crt for system:node:node01 due to open /tmp/this/cert/does/not/exist.crt: no such file or directory, unable to read client-key /tmp/this/key/does/not/exist.key for system:node:node01 due to open /tmp/this/key/does/not/exist.key: no such file or directory]

Any advice on how to overcome this error, or even troubleshoot it at a more granular level? Was considering regenerating certificates for api-server (kubeadm alpha phase certs apiserver), based on instructions within https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-alpha/#cmd-phase-certs ... but not sure if I'd be doing more damage.

Relatively new to Kubernetes, and the gentleman who set this up is not available for consult ... any help is appreciated. Thanks.

Answer 1

I think you need re-generate the apiserver certificate /etc/kubernetes/pki/apiserver.crt you can view current expire date like this.

openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -text |grep ' Not '             Not Before: Dec 20 14:32:00 2017 GMT             Not After : Dec 20 14:32:00 2018 GMT 

Here is the steps I used to regenerate the certificates on v1.11.5 cluster. compiled steps from here https://github.com/kubernetes/kubeadm/issues/581

---

to check all certificate expire date:

find /etc/kubernetes/pki/ -type f -name "*.crt" -print|egrep -v 'ca.crt$'|xargs -L 1 -t  -i bash -c 'openssl x509  -noout -text -in {}|grep After' 

Renew certificate on Master node.

*) Renew certificate

mv /etc/kubernetes/pki/apiserver.key /etc/kubernetes/pki/apiserver.key.old mv /etc/kubernetes/pki/apiserver.crt /etc/kubernetes/pki/apiserver.crt.old mv /etc/kubernetes/pki/apiserver-kubelet-client.crt /etc/kubernetes/pki/apiserver-kubelet-client.crt.old mv /etc/kubernetes/pki/apiserver-kubelet-client.key /etc/kubernetes/pki/apiserver-kubelet-client.key.old mv /etc/kubernetes/pki/front-proxy-client.crt /etc/kubernetes/pki/front-proxy-client.crt.old mv /etc/kubernetes/pki/front-proxy-client.key /etc/kubernetes/pki/front-proxy-client.key.old   kubeadm alpha phase certs apiserver  --config /root/kubeadm-kubetest.yaml kubeadm alpha phase certs apiserver-kubelet-client kubeadm alpha phase certs front-proxy-client   mv /etc/kubernetes/pki/apiserver-etcd-client.crt /etc/kubernetes/pki/apiserver-etcd-client.crt.old mv /etc/kubernetes/pki/apiserver-etcd-client.key /etc/kubernetes/pki/apiserver-etcd-client.key.old kubeadm alpha phase certs  apiserver-etcd-client   mv /etc/kubernetes/pki/etcd/server.crt /etc/kubernetes/pki/etcd/server.crt.old mv /etc/kubernetes/pki/etcd/server.key /etc/kubernetes/pki/etcd/server.key.old kubeadm alpha phase certs  etcd-server --config /root/kubeadm-kubetest.yaml  mv /etc/kubernetes/pki/etcd/healthcheck-client.crt /etc/kubernetes/pki/etcd/healthcheck-client.crt.old mv /etc/kubernetes/pki/etcd/healthcheck-client.key /etc/kubernetes/pki/etcd/healthcheck-client.key.old kubeadm alpha phase certs  etcd-healthcheck-client --config /root/kubeadm-kubetest.yaml   mv /etc/kubernetes/pki/etcd/peer.crt /etc/kubernetes/pki/etcd/peer.crt.old mv /etc/kubernetes/pki/etcd/peer.key /etc/kubernetes/pki/etcd/peer.key.old kubeadm alpha phase certs  etcd-peer --config /root/kubeadm-kubetest.yaml  *)  Backup old configuration files mv /etc/kubernetes/admin.conf /etc/kubernetes/admin.conf.old mv /etc/kubernetes/kubelet.conf /etc/kubernetes/kubelet.conf.old mv /etc/kubernetes/controller-manager.conf /etc/kubernetes/controller-manager.conf.old mv /etc/kubernetes/scheduler.conf /etc/kubernetes/scheduler.conf.old  kubeadm alpha phase kubeconfig all  --config /root/kubeadm-kubetest.yaml  mv $HOME/.kube/config .$HOMEkube/config.old cp -i /etc/kubernetes/admin.conf $HOME/.kube/config chown $(id -u):$(id -g) $HOME/.kube/config chmod 777 $HOME/.kube/config export KUBECONFIG=.kube/config 

Reboot the node and check the logs for etcd, kubeapi and kubelet.

Note: Remember to update your CI/CD job kubeconfig file. If you’re using helm command test that also.

Answer 2

This topic is also discussed in:

• https://github.com/kubernetes/kubeadm/issues/581
  • after 1.15 kubeadm upgrade automatically will renewal the certificates for you!
  • also 1.15 added a command to check cert expiration in kubeadm
• Renew kubernetes pki after expired

---

Kubernetes v1.15 provides docs for "Certificate Management with kubeadm":

• https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/
• Check certificate expiration:

kubeadm alpha certs check-expiration 

• Automatic certificate renewal:
  • kubeadm renews all the certificates during control plane upgrade.
• Manual certificate renewal:
  • You can renew your certificates manually at any time with the kubeadm alpha certs renew command.
  • This command performs the renewal using CA (or front-proxy-CA) certificate and key stored in /etc/kubernetes/pki.

---

For Kubernetes v1.14 I find this procedure the most helpful:

• https://stackoverflow.com/a/56334732/1147487
• backup and re-generate all certs:

$ cd /etc/kubernetes/pki/ $ mv {apiserver.crt,apiserver-etcd-client.key,apiserver-kubelet-client.crt,front-proxy-ca.crt,front-proxy-client.crt,front-proxy-client.key,front-proxy-ca.key,apiserver-kubelet-client.key,apiserver.key,apiserver-etcd-client.crt} ~/ $ kubeadm init phase certs all --apiserver-advertise-address <IP> 

• backup and re-generate all kubeconfig files:

$ cd /etc/kubernetes/ $ mv {admin.conf,controller-manager.conf,kubelet.conf,scheduler.conf} ~/ $ kubeadm init phase kubeconfig all $ reboot 

• copy new admin.conf:

$ cp -i /etc/kubernetes/admin.conf $HOME/.kube/config