Kubernetes Let's Encrypt cert-manager Error secret not found

Question

I followed this tutorial to get let's encrypt in kubernetes : https://github.com/ahmetb/gke-letsencrypt/blob/master/

I encountered some problems, cert-manager don't create the needed secret. Could you help me please to resolve this problem ?

Cert-manager ERRORS :

Found status change for Certificate "mydomain.fr" condition "Ready": "False" -> "False"; setting lastTransitionTime to 2018-11-06 17:37:20.683089649 +0000 UTC m=+5887.364224968
Error preparing issuer for certificate coffeer-ci/mydomain.fr: http-01 self check failed for domain "mydomain.fr"
[coffeer-ci/mydomain.fr] Error getting certificate 'domain-tls': secret "domain-tls" not found

Here is my kubernetes objects :

kubectl -n kube-system describe pod cert-manager

Name:           cert-manager-7bb46cc6b-scqrp
Namespace:      kube-system
Node:           gke-inkubator-default-pool-68c0309d-b86b/10.132.0.3
Start Time:     Tue, 06 Nov 2018 16:59:10 +0100
Labels:         app=cert-manager
                pod-template-hash=366027726
                release=cert-manager
Annotations:    <none>
Status:         Running
IP:             10.16.1.132
Controlled By:  ReplicaSet/cert-manager-7bb46cc6b
Containers:
  cert-manager:
    Container ID:  docker://d4795cfa85aacd2cbd0c5fd51246c436e3cf953632f4ca4a26e683c5867bf113
    Image:         quay.io/jetstack/cert-manager-controller:v0.5.0
    Image ID:      docker-pullable://quay.io/jetstack/cert-manager-controller@sha256:fd89c3c33fd89ffe0a9f91df2f54423397058d4180eccfe90b831859ba46b6e5
    Port:          <none>
    Host Port:     <none>
    Args:
      --cluster-resource-namespace=$(POD_NAMESPACE)
      --leader-election-namespace=$(POD_NAMESPACE)
    State:          Running
      Started:      Tue, 06 Nov 2018 16:59:13 +0100
    Ready:          True
    Restart Count:  0
    Environment:
      POD_NAMESPACE:  kube-system (v1:metadata.namespace)
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from cert-manager-token-9ck7b (ro)
Conditions:
  Type           Status
  Initialized    True 
  Ready          True 
  PodScheduled   True 
Volumes:
  cert-manager-token-9ck7b:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  cert-manager-token-9ck7b
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:          <none>

kubectl describe clusterissuer

Name:         letsencrypt-staging
Namespace:    
Labels:       <none>
Annotations:  <none>
API Version:  certmanager.k8s.io/v1alpha1
Kind:         ClusterIssuer
Metadata:
  Cluster Name:        
  Creation Timestamp:  2018-11-06T16:00:23Z
  Generation:          1
  Resource Version:    10184529
  Self Link:           /apis/certmanager.k8s.io/v1alpha1/clusterissuers/letsencrypt-staging
  UID:                 11e44fe0-e1dd-11e8-8bc6-42010a840078
Spec:
  Acme:
    Email:  [email protected]
    Http 01:
    Private Key Secret Ref:
      Key:   
      Name:  letsencrypt-staging
    Server:  https://acme-staging-v02.api.letsencrypt.org/directory
Status:
  Acme:
    Uri:  https://acme-staging-v02.api.letsencrypt.org/acme/acct/7297218
  Conditions:
    Last Transition Time:  2018-11-06T16:00:33Z
    Message:               The ACME account was registered with the ACME server
    Reason:                ACMEAccountRegistered
    Status:                True
    Type:                  Ready
Events:                    <none>

kubectl -n coffeer-ci describe certificate

Name:         mydomain.fr
Namespace:    coffeer-ci
Labels:       <none>
Annotations:  <none>
API Version:  certmanager.k8s.io/v1alpha1
Kind:         Certificate
Metadata:
  Cluster Name:        
  Creation Timestamp:  2018-11-06T16:10:57Z
  Generation:          1
  Resource Version:    10197662
  Self Link:           /apis/certmanager.k8s.io/v1alpha1/namespaces/coffeer-ci/certificates/mydomain.fr
  UID:                 8b6d508a-e1de-11e8-8bc6-42010a840078
Spec:
  Acme:
    Config:
      Domains:
        mydomain.fr
      Http 01:
        Ingress:  coffee-ingress
  Common Name:    mydomain.fr
  Issuer Ref:
    Kind:       ClusterIssuer
    Name:       letsencrypt-staging
  Secret Name:  domain-tls
Status:
  Acme:
    Order:
      Challenges:
        Authz URL:  https://acme-staging-v02.api.letsencrypt.org/acme/authz/wm5MvoFA12U37qdXdBCccyIWezpEsLoxHUGVDacmHpI
        Domain:     mydomain.fr
        Http 01:
          Ingress:  coffee-ingress
        Key:        RjHMkquS8Hh4dvJWZp2jLGW-MrSKEba-y8B8PzmVQ-M.4LwovuRj4ZgjrwLuye1cd5ftBRYaGIvtK__igMmDUD8
        Token:      RjHMkquS8Hh4dvJWZp2jLGW-MrSKEba-y8B8PzmVQ-M
        Type:       http-01
        URL:        https://acme-staging-v02.api.letsencrypt.org/acme/challenge/wm5MvoFA12U37qdXdBCccyIWezpEsLoxHUGVDacmHpI/192521366
        Wildcard:   false
      URL:          https://acme-staging-v02.api.letsencrypt.org/acme/order/7297218/12596140
  Conditions:
    Last Transition Time:  2018-11-06T17:47:28Z
    Message:               http-01 self check failed for domain "mydomain.bap.fr"
    Reason:                ValidateError
    Status:                False
    Type:                  Ready
Events:                    <none>

kubectl -n coffeer-ci describe ingress

Name:             coffee-ingress
Namespace:        coffeer-ci
Address:          35.233.8.223
Default backend:  default-http-backend:80 (10.16.1.5:8080)
Rules:
  Host                       Path  Backends
  ----                       ----  --------
  mydomain.fr  
                             /                                                                         coffee-service:80 (<none>)
                             /.well-known/acme-challenge/RjHMkquS8Hh4dvJWZp2jLGW-MrSKEba-y8B8PzmVQ-M   cm-acme-http-solver-kw2w4:8089 (<none>)
Annotations:
  ingress.kubernetes.io/forwarding-rule:        k8s-fw-coffeer-ci-coffee-ingress--4b1e5690f5d3853f
  ingress.kubernetes.io/target-proxy:           k8s-tp-coffeer-ci-coffee-ingress--4b1e5690f5d3853f
  ingress.kubernetes.io/url-map:                k8s-um-coffeer-ci-coffee-ingress--4b1e5690f5d3853f
  kubernetes.io/ingress.global-static-ip-name:  coffeer-ci-static
  kubernetes.io/tls-acme:                       true
  ingress.kubernetes.io/backends:               {"k8s-be-32603--4b1e5690f5d3853f":"HEALTHY"}
Events:
  Type     Reason             Age                 From                      Message
  ----     ------             ----                ----                      -------
  Normal   CREATE             40m                 nginx-ingress-controller  Ingress coffeer-ci/coffee-ingress
  Normal   CreateCertificate  34m                 cert-manager              Successfully created Certificate "domain-tls"
  Warning  Sync               25m (x23 over 59m)  loadbalancer-controller   Could not find TLS certificates. Continuing setup for the load balancer to serve HTTP. Note: this behavior is deprecated and will be removed in a future version of ingress-gce
  Normal   UPDATE             15m (x8 over 39m)   nginx-ingress-controller  Ingress coffeer-ci/coffee-ingress
  Warning  Sync               3m (x49 over 1h)    loadbalancer-controller   Error during sync: googleapi: Error 403: Quota 'BACKEND_SERVICES' exceeded. Limit: 9.0 globally., quotaExceeded

I have also the error Error 403: Quota 'BACKEND_SERVICES' exceeded. Limit: 9.0 globally., quotaExceeded in the ingress.

Thanks

Answer 1

Error preparing issuer for certificate coffeer-ci/mydomain.fr: http-01 self check failed for domain "mydomain.fr"

Means that it can't do the HTTP check that you actually own the domain. Do you own mydomain.fr? If yes, you need to add a DNS entry to make mydomain.fr resolve to the external IP (A record) of the load balancer (or if the load balancer has a name entry it would have to be a CNAME record, in the case of AWS ELBs) This way letsencrypt can use it to verify that you own the domain.

The other error:

Warning Sync 3m (x49 over 1h) loadbalancer-controller Error during sync: googleapi: Error 403: Quota 'BACKEND_SERVICES' exceeded. Limit: 9.0 globally., quotaExceeded

looks like the byproduct of not being able to verify the domain. If you don't specify and Ingress looks like cert-manager creates one for you with a 'LoadBalancer' type of service. It looks like it creates it initially but it keeps trying to sync to create it on GCP (maybe because to check if it can configure port 443) but after a while, the GCP API is throttling you.