How to setup letsencrypt cert issuer for kubernetes on AWS EKS with Terraform

Question

I'm trying to setup letsencrypt cert-issuer on kubernetes cluster. My terraform looks like this:

resource "helm_release" "cert_manager" {
  keyring = ""
  name = "cert-manager"
  chart = "stable/cert-manager"
  namespace = "kube-system"
  depends_on = ["helm_release.ingress"]
  set {
    name  = "webhook.enabled"
    value = "false"
  }

  provisioner "local-exec" {
    command = "kubectl --server=${aws_eks_cluster.demo.endpoint} --insecure-skip-tls-verify=true --token=${data.aws_eks_cluster_auth.demo.token} apply -f https://raw.githubusercontent.com/jetstack/cert-manager/release-0.6/deploy/manifests/00-crds.yaml"
  }

  provisioner "local-exec" {
    command = "kubectl --server=${aws_eks_cluster.demo.endpoint} --insecure-skip-tls-verify=true --token=${data.aws_eks_cluster_auth.demo.token} label namespace kube-system certmanager.k8s.io/disable-validation=\"true\" --overwrite"
  }

  provisioner "local-exec" {
    command = <<EOT
cat <<EOF | kubectl --server=${aws_eks_cluster.demo.endpoint} --insecure-skip-tls-verify=true --token=${data.aws_eks_cluster_auth.demo.token} create -f -
apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
  name: letsencrypt
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: [email protected]
    privateKeySecretRef:
      name: letsencrypt
    http01: {}
EOF
EOT
  }
}

I have simple test pod and service deployed. When I go to http://<cluster-address>/apple it responds with apple. So I try to create ingress for it:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: example-ingress
  annotations:
    certmanager.k8s.io/cluster-issuer: letsencrypt
    kubernetes.io/ingress.class: nginx
  labels:
    app: apple
    heritage: Tiller
    release: apple
spec:
  rules:
    - http:
        paths:
          - path: /apple
            backend:
              serviceName: apple-service
              servicePort: 5678
  tls:
    - hosts:
      - my.domain.alias.to.cluster.address.io
      secretName: my.domain.alias.to.cluster.address.io

But still, when I go to https://my.domain.alias.to.cluster.address.io/apple my browser warns me, and I can see the certificate is Kubernetes Ingress Controller Fake Certificate.

What am I missing? What should I do to have cert created by letsencrypt there?

UPDATE:

Logs from my cert-manager pod:

I0220 16:34:49.071883       1 sync.go:180] Certificate "my.domain.alias.to.cluster.address.io" for ingress "example-ingress" is up to date
I0220 16:34:49.072121       1 controller.go:179] ingress-shim controller: Finished processing work item "default/example-ingress"
I0220 16:34:49.071454       1 controller.go:145] certificates controller: syncing item 'default/my.domain.alias.to.cluster.address.io'
I0220 16:34:49.073892       1 helpers.go:183] Setting lastTransitionTime for Certificate "my.domain.alias.to.cluster.address.io" condition "Ready" to 2019-02-20 16:34:49.073885527 +0000 UTC m=+889.175312552
I0220 16:34:49.074450       1 sync.go:263] Certificate default/my.domain.alias.to.cluster.address.io scheduled for renewal in 1438h47m42.92555861s
I0220 16:34:49.081224       1 controller.go:151] certificates controller: Finished processing work item "default/my.domain.alias.to.cluster.address.io"
I0220 16:34:49.081479       1 controller.go:173] ingress-shim controller: syncing item 'default/example-ingress'
I0220 16:34:49.081567       1 sync.go:177] Certificate "my.domain.alias.to.cluster.address.io" for ingress "example-ingress" already exists
I0220 16:34:49.081631       1 sync.go:180] Certificate "my.domain.alias.to.cluster.address.io" for ingress "example-ingress" is up to date
I0220 16:34:49.081672       1 controller.go:179] ingress-shim controller: Finished processing work item "default/example-ingress"
I0220 16:34:49.081743       1 controller.go:145] certificates controller: syncing item 'default/my.domain.alias.to.cluster.address.io'
I0220 16:34:49.082384       1 sync.go:263] Certificate default/my.domain.alias.to.cluster.address.io scheduled for renewal in 1438h47m42.917624001s
I0220 16:34:49.087552       1 controller.go:151] certificates controller: Finished processing work item "default/my.domain.alias.to.cluster.address.io"
I0220 16:35:04.571789       1 controller.go:173] ingress-shim controller: syncing item 'default/example-ingress'

And this is what kubectl describe certificate my.domain.alias.to.cluster.address.io returns:

Name:         my.domain.alias.to.cluster.address.io
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  certmanager.k8s.io/v1alpha1
Kind:         Certificate
Metadata:
  Creation Timestamp:  2019-02-20T16:34:49Z
  Generation:          1
  Owner References:
    API Version:           extensions/v1beta1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Ingress
    Name:                  example-ingress
    UID:                   709a55df-352d-11e9-bf9d-06ede39599be
  Resource Version:        278211
  Self Link:               /apis/certmanager.k8s.io/v1alpha1/namespaces/default/certificates/my.domain.alias.to.cluster.address.io
  UID:                     709bf1bd-352d-11e9-b941-026486635030
Spec:
  Acme:
    Config:
      Domains:
        my.domain.alias.to.cluster.address.io
      Http 01:
        Ingress:
        Ingress Class:  nginx
  Dns Names:
    my.domain.alias.to.cluster.address.io
  Issuer Ref:
    Kind:       ClusterIssuer
    Name:       letsencrypt
  Secret Name:  my.domain.alias.to.cluster.address.io
Status:
  Conditions:
    Last Transition Time:  2019-02-20T16:34:49Z
    Message:               Certificate is up to date and has not expired
    Reason:                Ready
    Status:                True
    Type:                  Ready
  Not After:               2019-05-21T15:22:32Z
Events:                    <none>

In the logs of ingress controller I can find this:

I0220 16:22:34.428736       8 store.go:446] secret default/my.domain.alias.to.cluster.address.io was updated and it is used in ingress annotations. Parsing...
I0220 16:22:34.429898       8 backend_ssl.go:68] Adding Secret "default/my.domain.alias.to.cluster.address.io" to the local store
I0220 16:22:35.410950       8 controller.go:172] Configuration changes detected, backend reload required.
I0220 16:22:35.522502       8 controller.go:190] Backend successfully reloaded.
[20/Feb/2019:16:22:35 +0000]TCP200000.000
I0220 16:27:39.225810       8 controller.go:172] Configuration changes detected, backend reload required.
I0220 16:27:39.226685       8 event.go:221] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"example-ingress", UID:"f2f0c9bd-345d-11e9-bf9d-06ede39599be", APIVersion:"extensions/v1beta1", ResourceVersion:"277488", FieldPath:""}): type: 'Normal' reason: 'DELETE' Ingress default/example-ingress
I0220 16:27:39.336879       8 controller.go:190] Backend successfully reloaded.
[20/Feb/2019:16:27:39 +0000]TCP200000.001
I0220 16:27:53.090686       8 event.go:221] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"example-ingress", UID:"78ab0815-352c-11e9-bf9d-06ede39599be", APIVersion:"extensions/v1beta1", ResourceVersion:"277520", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress default/example-ingress
I0220 16:27:53.091216       8 controller.go:172] Configuration changes detected, backend reload required.
I0220 16:27:53.212854       8 controller.go:190] Backend successfully reloaded.
[20/Feb/2019:16:27:53 +0000]TCP200000.000
I0220 16:28:04.566342       8 status.go:388] updating Ingress default/example-ingress status from [] to [{34.245.112.11 }]
I0220 16:28:04.576525       8 event.go:221] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"example-ingress", UID:"78ab0815-352c-11e9-bf9d-06ede39599be", APIVersion:"extensions/v1beta1", ResourceVersion:"277542", FieldPath:""}): type: 'Normal' reason: 'UPDATE' Ingress default/example-ingress
I0220 16:28:05.676217       8 event.go:221] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"example-ingress", UID:"78ab0815-352c-11e9-bf9d-06ede39599be", APIVersion:"extensions/v1beta1", ResourceVersion:"277546", FieldPath:""}): type: 'Normal' reason: 'DELETE' Ingress default/example-ingress
I0220 16:28:07.909830       8 controller.go:172] Configuration changes detected, backend reload required.
I0220 16:28:08.019070       8 controller.go:190] Backend successfully reloaded.
[20/Feb/2019:16:28:08 +0000]TCP200000.000
I0220 16:28:22.557334       8 controller.go:172] Configuration changes detected, backend reload required.
I0220 16:28:22.557490       8 event.go:221] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"cm-acme-http-solver-dmnqh", UID:"7f8f4be4-3461-11e9-b941-026486635030", APIVersion:"extensions/v1beta1", ResourceVersion:"277576", FieldPath:""}): type: 'Normal' reason: 'DELETE' Ingress default/cm-acme-http-solver-dmnqh
I0220 16:28:22.662971       8 controller.go:190] Backend successfully reloaded.
[20/Feb/2019:16:28:22 +0000]TCP200000.000
I0220 16:34:49.057385       8 event.go:221] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"example-ingress", UID:"709a55df-352d-11e9-bf9d-06ede39599be", APIVersion:"extensions/v1beta1", ResourceVersion:"278207", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress default/example-ingress
I0220 16:34:49.057688       8 controller.go:172] Configuration changes detected, backend reload required.
I0220 16:34:49.175039       8 controller.go:190] Backend successfully reloaded.
[20/Feb/2019:16:34:49 +0000]TCP200000.000
I0220 16:35:04.565324       8 status.go:388] updating Ingress default/example-ingress status from [] to [{34.245.112.11 }]
I0220 16:35:04.572954       8 event.go:221] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"example-ingress", UID:"709a55df-352d-11e9-bf9d-06ede39599be", APIVersion:"extensions/v1beta1", ResourceVersion:"278236", FieldPath:""}): type: 'Normal' reason: 'UPDATE' Ingress default/example-ingress
10.0.1.114 - [10.0.1.114] - - [20/Feb/2019:18:38:33 +0000] "\x05\x01\x00" 400 157 "-" "-" 0 0.751 [] - - - - e0aec2a9e3e71e136a1c62939e341b49
10.0.1.114 - [10.0.1.114] - - [20/Feb/2019:18:39:50 +0000] "\x04\x01\x00P\x05\xBC\xD2\x0C\x00" 400 157 "-" "-" 0 0.579 [] - - - - 7f825a3ef2f94e200b14fe3691e4fdde
10.0.1.114 - [10.0.1.114] - - [20/Feb/2019:18:41:30 +0000] "GET http://5.188.210.12/echo.php HTTP/1.1" 400 657 "https://www.google.com/" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36" 359 0.000 [] - - - - 1167890a763ddc360051046c84a47d21
10.0.1.114 - [10.0.1.114] - - [20/Feb/2019:19:46:35 +0000] "GET /apple HTTP/1.1" 308 171 "-" "Mozilla/5.0 (Linux; Android 8.0.0; ONEPLUS A3003) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.105 Mobile Safari/537.36" 555 0.000 [default-apple-service-5678] - - - - b1f1bb0da3e465c3a54e963663dffb61
10.0.1.114 - [10.0.1.114] - - [20/Feb/2019:20:38:39 +0000] "\x03\x00\x00/*\xE0\x00\x00\x00\x00\x00Cookie: mstshash=Administr" 400 157 "-" "-" 0 0.065 [] - - - - cd420e70b3f78bee069f8bac97918e36

Answer 1

Basically, letsencrypt is not issuing the certificate for you so it's defaulting to the Fake cert. You need to make sure that my.domain.alias.to.cluster.address.io is publicly resolvable, say through a DNS server like 8.8.8.8 and then it needs to resolve to a publicly accessible IP address. You can debug what's happening by looking at the certmanager pod logs.

$ kubectl logs <certmanagerpod>

You can also see the details about the certificates (and you might be able to see why it didn't get issued).

$ kubectl get certificates
$ kubectl describe <certificate-name>

Another aspect is that you could be being rate-limited by https://acme-v02.api.letsencrypt.org/directory which is their prod environment. You could also try: https://acme-staging-v02.api.letsencrypt.org/directory which is their staging environment.

Answer 2

It turned out I was missing host in ingress rule. path is not enough if I want to use certificate.