can't get certs working with cert-manager

Question

I can't seem to get cert-manager working:

$ kubectl get certificates -o wide
NAME              READY   SECRET            ISSUER             STATUS                                                                    AGE
example-ingress   False   example-ingress   letsencrypt-prod   Waiting for CertificateRequest "example-ingress-2556707613" to complete   6m23s
$ kubectl get CertificateRequest -o wide
NAME                         READY   ISSUER             STATUS                                                                               AGE
example-ingress-2556707613   False   letsencrypt-prod   Referenced "Issuer" not found: issuer.cert-manager.io "letsencrypt-prod" not found   7m7s

and in the logs i see:

I1025 06:22:00.117292       1 sync.go:163] cert-manager/controller/ingress-shim "level"=0 "msg"="certificate already exists for ingress resource, ensuring it is up to date" "related_resource_kind"="Certificate" "related_resource_name"="example-ingress" "related_resource_namespace"="default" "resource_kind"="Ingress" "resource_name"="example-ingress" "resource_namespace"="default"
I1025 06:22:00.117341       1 sync.go:176] cert-manager/controller/ingress-shim "level"=0 "msg"="certificate resource is already up to date for ingress" "related_resource_kind"="Certificate" "related_resource_name"="example-ingress" "related_resource_namespace"="default" "resource_kind"="Ingress" "resource_name"="example-ingress" "resource_namespace"="default"
I1025 06:22:00.117382       1 controller.go:135] cert-manager/controller/ingress-shim "level"=0 "msg"="finished processing work item" "key"="default/example-ingress"
I1025 06:22:00.118026       1 sync.go:361] cert-manager/controller/certificates "level"=0 "msg"="no existing CertificateRequest resource exists, creating new request..." "related_resource_kind"="Secret" "related_resource_name"="example-ingress" "related_resource_namespace"="default" "resource_kind"="Certificate" "resource_name"="example-ingress" "resource_namespace"="default"
I1025 06:22:00.147147       1 controller.go:129] cert-manager/controller/certificaterequests-issuer-venafi "level"=0 "msg"="syncing item" "key"="default/example-ingress-2556707613"
I1025 06:22:00.147267       1 sync.go:373] cert-manager/controller/certificates "level"=0 "msg"="created certificate request" "related_resource_kind"="Secret" "related_resource_name"="example-ingress" "related_resource_namespace"="default" "resource_kind"="Certificate" "resource_name"="example-ingress" "resource_namespace"="default" "request_name"="example-ingress-2556707613"
I1025 06:22:00.147284       1 controller.go:129] cert-manager/controller/certificaterequests-issuer-acme "level"=0 "msg"="syncing item" "key"="default/example-ingress-2556707613"
I1025 06:22:00.147273       1 conditions.go:200] Setting lastTransitionTime for CertificateRequest "example-ingress-2556707613" condition "Ready" to 2019-10-25 06:22:00.147254385 +0000 UTC m=+603.871617341
I1025 06:22:00.147392       1 conditions.go:200] Setting lastTransitionTime for CertificateRequest "example-ingress-2556707613" condition "Ready" to 2019-10-25 06:22:00.147380513 +0000 UTC m=+603.871743521
E1025 06:22:00.147560       1 pki.go:128] cert-manager/controller/certificates "msg"="error decoding x509 certificate" "error"="error decoding cert PEM block" "related_resource_kind"="Secret" "related_resource_name"="example-ingress" "related_resource_namespace"="default" "resource_kind"="Certificate" "resource_name"="example-ingress" "resource_namespace"="default" "secret_key"="tls.crt"
I1025 06:22:00.147620       1 conditions.go:155] Setting lastTransitionTime for Certificate "example-ingress" condition "Ready" to 2019-10-25 06:22:00.147613112 +0000 UTC m=+603.871976083
I1025 06:22:00.147731       1 controller.go:129] cert-manager/controller/certificaterequests-issuer-ca "level"=0 "msg"="syncing item" "key"="default/example-ingress-2556707613"
I1025 06:22:00.147765       1 conditions.go:200] Setting lastTransitionTime for CertificateRequest "example-ingress-2556707613" condition "Ready" to 2019-10-25 06:22:00.14776244 +0000 UTC m=+603.872125380
I1025 06:22:00.147912       1 controller.go:129] cert-manager/controller/certificaterequests-issuer-selfsigned "level"=0 "msg"="syncing item" "key"="default/example-ingress-2556707613"
I1025 06:22:00.147942       1 conditions.go:200] Setting lastTransitionTime for CertificateRequest "example-ingress-2556707613" condition "Ready" to 2019-10-25 06:22:00.147938966 +0000 UTC m=+603.872301909
I1025 06:22:00.147968       1 controller.go:129] cert-manager/controller/certificaterequests-issuer-vault "level"=0 "msg"="syncing item" "key"="default/example-ingress-2556707613"
I1025 06:22:00.148023       1 conditions.go:200] Setting lastTransitionTime for CertificateRequest "example-ingress-2556707613" condition "Ready" to 2019-10-25 06:22:00.148017945 +0000 UTC m=+603.872380906

i deployed cert-manager via the manifest:

https://github.com/jetstack/cert-manager/releases/download/v0.11.0/cert-manager.yaml

$ kubectl get clusterissuer letsencrypt-prod -o yaml
apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"cert-manager.io/v1alpha2","kind":"ClusterIssuer","metadata":{"annotations":{},"name":"letsencrypt-prod"},"spec":{"acme":{"email":"[email protected]","privateKeySecretRef":{"name":"letsencrypt-prod"},"server":"https://acme-staging-v02.api.letsencrypt.org/directory","solvers":[{"http01":{"ingress":{"class":"nginx"}},"selector":{}}]}}}
  creationTimestamp: "2019-10-25T06:27:06Z"
  generation: 1
  name: letsencrypt-prod
  resourceVersion: "1759784"
  selfLink: /apis/cert-manager.io/v1alpha2/clusterissuers/letsencrypt-prod
  uid: 05831417-b359-42de-8298-60da553575f2
spec:
  acme:
    email: [email protected]
    privateKeySecretRef:
      name: letsencrypt-prod
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    solvers:
    - http01:
        ingress:
          class: nginx
      selector: {}
status:
  acme:
    lastRegisteredEmail: [email protected]
    uri: https://acme-staging-v02.api.letsencrypt.org/acme/acct/11410425
  conditions:
  - lastTransitionTime: "2019-10-25T06:27:07Z"
    message: The ACME account was registered with the ACME server
    reason: ACMEAccountRegistered
    status: "True"
    type: Ready

and my ingress is:

$ kubectl get ingress example-ingress -o yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/issuer: letsencrypt-prod
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"cert-manager.io/issuer":"letsencrypt-prod","kubernetes.io/ingress.class":"nginx","kubernetes.io/tls-acme":"true"},"name":"example-ingress","namespace":"default"},"spec":{"rules":[{"host":"example-ingress.example.com","http":{"paths":[{"backend":{"serviceName":"apple-service","servicePort":5678},"path":"/apple"},{"backend":{"serviceName":"banana-service","servicePort":5678},"path":"/banana"}]}}],"tls":[{"hosts":["example-ingress.example.com"],"secretName":"example-ingress"}]}}
    kubernetes.io/ingress.class: nginx
    kubernetes.io/tls-acme: "true"
  creationTimestamp: "2019-10-25T06:22:00Z"
  generation: 1
  name: example-ingress
  namespace: default
  resourceVersion: "1758822"
  selfLink: /apis/extensions/v1beta1/namespaces/default/ingresses/example-ingress
  uid: 921b2e91-9101-4c3c-a0d8-3f871dafdd30
spec:
  rules:
  - host: example-ingress.example.com
    http:
      paths:
      - backend:
          serviceName: apple-service
          servicePort: 5678
        path: /apple
      - backend:
          serviceName: banana-service
          servicePort: 5678
        path: /banana
  tls:
  - hosts:
    - example-ingress.example.com
    secretName: example-ingress
status:
  loadBalancer:
    ingress:
    - ip: x.y.z.a

any idea whats wrong? cheers,

Answer 1

Your ingress is referring to an issuer, but the issuer is a ClusterIssuer. Could that be the reason? I have a similar setup with Issuer instead of a ClusterIssuer and it is working.

Answer 2

I have done this implementation, you can follow this way -

1. Install jetstack from here

2. Then follow these steps from this stackoverflow post

Make one clusterIssuer or you can make individual issuer too, once you patch the hostname to ingress, then the tls-certificate in that namespace will be autogenerated by Jetstack after the acme-challenge validation

Kindly make sure to map the IP of loadbalancer nginx/traefik etc to DNS/hostname