Kubernetes not getting Letsencrypt Cert

Question

I have installed microk8s, traefik and cert-manager. When I try to receive a letsencrypt certificate, a new pod for answering the challenge is created, but the request from the letsencryt server does not reach this pod. Instead, the request is forwarded to the pod that serves the website.

It looks like the ingressroute routing the traffic to the web pod has higher priority then the ingress that routes the /.well-known/acme-challenge/... requests to the correct pod. What am I missing?

kubectl edit clusterissuer letsencrypt-prod:

kind: ClusterIssuer
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"cert-manager.io/v1","kind":"ClusterIssuer","metadata":{"annotations":{},"name":"letsencrypt-prod"},"spec":{"acme":{"email":"[email protected]","privateKeySecretRef":{"name":"letsencrypt-prod"},"server":"https://acme-v02.api.letsencrypt.org/directory","solvers":[{"http01":{"ingress":{"class":"traefik"}}}]}}}
  creationTimestamp: "2022-07-11T14:32:15Z"
  generation: 11
  name: letsencrypt-prod
  resourceVersion: "49979842"
  uid: 40c4e26d-9c94-4cda-aa3a-357491bdb25a
spec:
  acme:
    email: [email protected]
    preferredChain: ""
    privateKeySecretRef:
      name: letsencrypt-prod
    server: https://acme-v02.api.letsencrypt.org/directory
    solvers:
    - http01:
        ingress: {}
status:
  acme:
    lastRegisteredEmail: [email protected]
    uri: https://acme-v02.api.letsencrypt.org/acme/acct/627190636
  conditions:
  - lastTransitionTime: "2022-07-11T14:32:17Z"
    message: The ACME account was registered with the ACME server
    observedGeneration: 11
    reason: ACMEAccountRegistered
    status: "True"
    type: Ready

kubectl edit ingressroute webspace1-tls:

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"traefik.containo.us/v1alpha1","kind":"IngressRoute","metadata":{"annotations":{},"name":"w271a19-tls","namespace":"default"},"spec":{"entryPoints":["websecure"],"routes":[{"kind":"Rule","match":"Host(`test1.mydomain.com`)","middlewares":[{"name":"test-compress"}],"priority":10,"services":[{"name":"w271a19","port":80}]}],"tls":{"secretName":"test1.mydomain.com-tls"}}}
  creationTimestamp: "2022-10-05T20:01:38Z"
  generation: 7
  name: w271a19-tls
  namespace: default
  resourceVersion: "45151920"
  uid: 77e9b7ac-33e7-4810-9baf-579f00e2db6b
spec:
  entryPoints:
  - websecure
  routes:
  - kind: Rule
    match: Host(`test1.mydomain.com`)
    middlewares:
    - name: test-compress
    priority: 10
    services:
    - name: w271a19
      port: 80
  tls:
    secretName: test1.mydomain.com-tls

kubectl edit ingress cm-acme-http-solver-rz9mm:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/whitelist-source-range: 0.0.0.0/0,::/0
  creationTimestamp: "2023-03-22T13:00:18Z"
  generateName: cm-acme-http-solver-
  generation: 1
  labels:
    acme.cert-manager.io/http-domain: "2306410973"
    acme.cert-manager.io/http-token: "1038683769"
    acme.cert-manager.io/http01-solver: "true"
  name: cm-acme-http-solver-rz9mm
  namespace: default
  ownerReferences:
  - apiVersion: acme.cert-manager.io/v1
    blockOwnerDeletion: true
    controller: true
    kind: Challenge
    name: test1.mydomain.com-glnrn-2096762198-4162956557
    uid: db8b5c78-8549-4f13-b43d-c6c7bba7468d
  resourceVersion: "52806119"
  uid: 6b27e02a-ee65-4809-b391-95c03f9ebb36
spec:
  ingressClassName: traefik
  rules:
  - host: test1.mydomain.com
    http:
      paths:
      - backend:
          service:
            name: cm-acme-http-solver-ll2zr
            port:
              number: 8089
        path: /.well-known/acme-challenge/9qtVY8FjfMIWd_wBNhP3PEPJZo4lFTw8WfWLMucRqAQ
        pathType: ImplementationSpecific
status:
  loadBalancer: {}

get_cert.yaml:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: test1.mydomain.com
  namespace: default
spec:
  secretName: test1.mydomain.com-tls
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
  commonName: test1.mydomain.com
  dnsNames:
  - test1.mydomain.com

In the webserver log of the web pod I see the reqests to /.well-known... coming in.

Answer 1

Shouldn't this annotation be added to ingress?

cert-manager.io/cluster-issuer=letsencrypt-production