Rancher Server v2.x expired certificates

Question

My certificates for rancher server expired and now I can not log in to UI anymore to manage my k8s clusters.

Error:

2021-05-26 00:57:52.437334 I | http: TLS handshake error from 127.0.0.1:43238: remote error: tls: bad certificate 
2021/05/26 00:57:52 [INFO] Waiting for server to become available: Get https://127.0.0.1:6443/version?timeout=30s: x509: certificate has expired or is not yet valid

So what I did was rolling back the date on the RancherOS machine that is running Rancher Server container. After that I restarted the container and it refreshed the certificates. I checked with:

for i in `ls /var/lib/rancher/k3s/server/tls/*.crt`; do echo $i; openssl x509 -enddate -noout -in $i; done

Since now I was able to log into the UI I forced a certificate rotation on the k8s cluster.

But I still get the same error once the date is reset to current and I can not log in to the Rancher Server UI.

What am I missing here?

Answer 1

This was the missing piece: https://github.com/rancher/rancher/issues/26984#issuecomment-818770519

Deleting the dynamic-cert.json and running kubectl delete secret