How can I fix 'failed calling webhook "webhook.cert-manager.io"'?

Question

I'm trying to set up a K3s cluster. When I had a single master and agent setup cert-manager had no issues. Now I'm trying a 2 master setup with embedded etcd. I opened TCP ports 6443 and 2379-2380 for both VMs and did the following:

VM1: curl -sfL https://get.k3s.io | sh -s server --token TOKEN --cluster-init
VM2: curl -sfL https://get.k3s.io | sh -s server --token TOKEN --server https://MASTER_IP:6443

# k3s kubectl get nodes
NAME  STATUS   ROLES                       AGE    VERSION
VM1   Ready    control-plane,etcd,master   130m   v1.22.7+k3s1
VM2   Ready    control-plane,etcd,master   128m   v1.22.7+k3s1

Installing cert-manager works fine:

# k3s kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.8.0/cert-manager.yaml
# k3s kubectl get pods --namespace cert-manager
NAME                                       READY   STATUS
cert-manager-b4d6fd99b-c6fpc               1/1     Running
cert-manager-cainjector-74bfccdfdf-gtmrd   1/1     Running
cert-manager-webhook-65b766b5f8-brb76      1/1     Running

My manifest has the following definition:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: [email protected]
    privateKeySecretRef:
      name: letsencrypt-account-key
    solvers:
    - selector: {}
      http01:
        ingress: {}

Which results in the following error:

# k3s kubectl apply -f manifest.yaml
Error from server (InternalError): error when creating "manifest.yaml": Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s": context deadline exceeded

I tried disabling both firewalls, waiting a day, reset and re-setup, but the error persists. Google hasn't been much help either. The little info I can find goes over my head for the most part and no tutorial seems to do any extra steps.

Answer 1

A good starting point for troubleshooting issues with the webhook can be found int the docs, e.g. there is a section for problems on GKE private clusters.

In my case, however, this didn't really solve the problem. For me the issue was that when I played around with cert-manager I happen to install and uninstall it multiple times. It turned out that just removing the namespace, e.g. kubectl delete namespace cert-manager didn't remove the webhooks and other non-obvious resources.

Following the official guide for uninstalling cert-manager and applying the manifests again solved the issue.