Issuing certificate as Secret does not exist

Question

Below is the describe output for both my clusterissuer and certificate reource. I am brand new to cert-manager so not 100% sure this is set up properly - we need to use http01 validation however we are not using an nginx controller. Right now we only have 2 microservices so the public-facing IP address simply belongs to a k8s service (type loadbalancer) which routes traffic to a pod where an Extensible Service Proxy container sits in front of the container running the application code. Using this set up I haven't been able to get anything beyond the errors below, however as I mentioned I'm brand new to cert-manager & ESP so this could be configured incorrectly...

Name:         clusterissuer-dev
Namespace:    
Labels:       <none>
Annotations:  kubectl.kubernetes.io/last-applied-configuration:
API Version:  cert-manager.io/v1beta1
Kind:         ClusterIssuer
Metadata:
  Creation Timestamp:  2020-08-07T18:46:29Z
  Generation:          1
  Resource Version:    4550439
  Self Link:           /apis/cert-manager.io/v1beta1/clusterissuers/clusterissuer-dev
  UID:                 65933d87-1893-49af-b90e-172919a18534
Spec:
  Acme:
    Email:  [email protected]
    Private Key Secret Ref:
      Name:  letsencrypt-dev
    Server:  https://acme-staging-v02.api.letsencrypt.org/directory
    Solvers:
      http01:
        Ingress:
          Class:  nginx
Status:
  Acme:
    Last Registered Email:  [email protected]
    Uri:                    https://acme-staging-v02.api.letsencrypt.org/acme/acct/15057658
  Conditions:
    Last Transition Time:  2020-08-07T18:46:30Z
    Message:               The ACME account was registered with the ACME server
    Reason:                ACMEAccountRegistered
    Status:                True
    Type:                  Ready
Events:                    <none>


Name:         test-cert-default-ns
Namespace:    default
Labels:       <none>
Annotations:  kubectl.kubernetes.io/last-applied-configuration:
API Version:  cert-manager.io/v1beta1
Kind:         Certificate
Metadata:
  Creation Timestamp:  2020-08-10T15:05:31Z
  Generation:          2
  Resource Version:    5961064
  Self Link:           /apis/cert-manager.io/v1beta1/namespaces/default/certificates/test-cert-default-ns
  UID:                 259f62e0-b272-47d6-b70e-dbcb7b4ed21b
Spec:
  Dns Names:
    dev.test.com
  Issuer Ref:
    Name:       clusterissuer-dev
  Secret Name:  clusterissuer-dev-tls
Status:
  Conditions:
    Last Transition Time:        2020-08-10T15:05:31Z
    Message:                     Issuing certificate as Secret does not exist
    Reason:                      DoesNotExist
    Status:                      False
    Type:                        Ready
    Last Transition Time:        2020-08-10T15:05:31Z
    Message:                     Issuing certificate as Secret does not exist
    Reason:                      DoesNotExist
    Status:                      True
    Type:                        Issuing
  Next Private Key Secret Name:  test-cert-default-ns-rrl7j
Events:
  Type    Reason     Age    From          Message
  ----    ------     ----   ----          -------
  Normal  Requested  2m51s  cert-manager  Created new CertificateRequest resource "test-cert-default-ns-c4wxd"

One last item - if I run the command kubectl get certificate -o wide I get the following output.

  NAME                           READY   SECRET                         ISSUER                     STATUS                                         AGE
  test-cert-default-ns           False   clusterissuer-dev-tls          clusterissuer-dev          Issuing certificate as Secret does not exist   2d23h

Answer 1

I had the same issue and I followed the advice given in the comments by @Popopame suggesting to check out the troubleshooting guide of cert-manager to find out how to troubleshoot cert-manager. or [cert-managers troubleshooting guide for acme issues] to find out which part of the acme process breaks the setup.

It seems that often it is the acme-challenge where letsencrypt verifies the domain ownership by requesting a certain code be offered at port 80 at a certain path. For example: http://example.com/.well-known/acme-challenge/M8iYs4tG6gM-B8NHuraXRL31oRtcE4MtUxRFuH8qJmY. Notice the http:// that shows letsencrypt will try to validate domain ownership on port 80 of your desired domain.

So one of the common errors is, that cert-manager could not put the correct challenge in the correct path behind port 80. For example due to a firewall blocking port 80 on a bare metal server or a loadbalancer that only forwards port 443 to the kubernetes cluster and redirects to 443 directly.

Also be aware of the fact, that cert-manager tries to validate the ACME challenge as well so you should configure the firewalls to allow requests coming from your servers as well.

If you have trouble getting your certificate to a different namespace, this would be a good point to start with.

In your specific case I would guess at a problem with the ACME challenge as the CSR (Certificate Signing Request) was created as indicated in the bottom most describe line but nothing else happened.

Answer 2

1. Setup Using Helm

By far the easiest method I've found was to use helm v3 to install cert-manager. I was able to set it up on a k3s cluster as follows:

$   helm repo add jetstack https://charts.jetstack.io
$   helm repo update
$   helm install \
        cert-manager jetstack/cert-manager \
        --namespace cert-manager \
        --version v1.2.0 \
        --create-namespace \
        --set installCRDs=true

2. Setup ClusterIssuer

Once it's installed you need to create a ClusterIssuer which can then be used when requesting certificates from let's encrypt.

$ more cert-clusterissuer.yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-stg
  namespace: cert-manager
spec:
  acme:
    email: [email protected]
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      # Secret resource that will be used to store the account's private key.
      name: le-issuer-acct-key
    solvers:
    - dns01:
        cloudflare:
          email: [email protected]
          apiTokenSecretRef:
            name: cloudflare-api-token-secret
            key: api-token
      selector:
        dnsZones:
        - 'mydomdom.org'
        - '*.mydomdom.org'

Deploy that, notice it'll get deployed into the same namespaces as cert-manager:

$ kubectl apply -f cert-clusterissuer.yaml

$ kubectl -n cert-manager get clusterissuers
NAME              READY   AGE
letsencrypt-stg   True    53m

3. Setup Cloudflare API Token Secret

Deploy your Cloudflare API token into a secret, again put it into the cert-manager namespace:

$ more cloudflare-api-token.yaml
apiVersion: v1
kind: Secret
metadata:
  name: cloudflare-api-token-secret
  namespace: cert-manager
type: Opaque
stringData:
  api-token: <my cloudflare api token key>

$ kubectl -n cert-manager -f cloudflare-api-token.yaml

4. Create a test Certificate

Now attempt to request the generation of a certificate from let's encrypt:

$ more test-certificate.yaml
apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
  name: le-test-mydomdom-org
  namespace: cert-manager
spec:
  secretName: le-test-mydomdom-org
  issuerRef:
    name: letsencrypt-stg
    kind: ClusterIssuer
  commonName: 'le-test.mydomdom.org'
  dnsNames:
  - "le-test.mydomdom.org"

$ kubectl -n cert-manager apply -f test-certificate.yaml

5. Debugging Certificate Creation

You can then watch the request as it flows through the various stages. I believe the flow is certificates -> certificaterequests -> orders -> challenges.

NOTE: Knowing this general flow was hugely helpful for me in terms of understanding where a request was failing within kubernetes as I was attempting to debug it.

When debugging you'll typically want to do kubectl -n cert-manager <stage> -A to see a list of all the outstanding resources within that stage. Keep in mind that after a challenge is fulfilled it'll no longer show up within the output of kubectl -n cert-manager challenges.

Also keep in mind that any DNS entries created to fulfill the challenge stage will typically have their TTL set to ~2min so if you go looking in your Cloudflare UI and do not see them, they likely already timed out and rolled off.

For e.g.:

References

• https://cert-manager.io/docs/configuration/acme/
• Installing and using cert-manager with k3s
• Install and Setup Cert-Manager for Automated SSL Certificates
• Make SSL certs easy with k3s
• Installing cert-manager on Kubernetes with CloudFlare DNS - Update

Answer 3

I had this problem on DigitalOcean, for me disabling proxy protocol and tls-passthrough fixed the problem.

These configs should be commented on ingress-nginx service:

# Enable proxy protocol
service.beta.kubernetes.io/do-loadbalancer-enable-proxy-protocol: "true"
# Specify whether the DigitalOcean Load Balancer should pass encrypted data to backend droplets
service.beta.kubernetes.io/do-loadbalancer-tls-passthrough: "true"