Let's Encrypt kubernetes Ingress Controller issuing Fake Certificate

Question

Not Sure why I'm getting Fake certificate, even the certificate is properly issued by Let's Encrypt using certmanager

The setup is running on the Alibaba Cloud ECS console, where one Kube-master and one cube-minion form a Kubernetes cluster.

Service Details

root@kube-master:~# kubectl get svc 
NAME         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.96.0.1        <none>        443/TCP   3h20m
my-nginx     ClusterIP   10.101.150.247   <none>        80/TCP    77m

Pod Details

root@kube-master:~# kubectl get pods --show-labels
NAME                        READY   STATUS    RESTARTS   AGE   LABELS
my-nginx-6cc48cd8db-n6scm   1/1     Running   0          46s   app=my-nginx,pod-template-hash=6cc48cd8db

Helm Cert-manager deployed

root@kube-master:~# helm ls 
NAME            REVISION    UPDATED                     STATUS      CHART               APP VERSION NAMESPACE  
cert-manager    1           Tue Mar 12 15:29:21 2019    DEPLOYED    cert-manager-v0.5.2 v0.5.2      kube-system
kindred-garfish 1           Tue Mar 12 17:03:41 2019    DEPLOYED    nginx-ingress-1.3.1 0.22.0      kube-system

Certificate Issued Properly

root@kube-master:~# kubectl describe certs 
Name:         tls-prod-cert
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  certmanager.k8s.io/v1alpha1
Kind:         Certificate
Metadata:
  Creation Timestamp:  2019-03-12T10:26:58Z
  Generation:          2
  Owner References:
    API Version:           extensions/v1beta1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Ingress
    Name:                  nginx-ingress-prod
    UID:                   5ab11929-44b1-11e9-b431-00163e005d19
  Resource Version:        17687
  Self Link:               /apis/certmanager.k8s.io/v1alpha1/namespaces/default/certificates/tls-prod-cert
  UID:                     5dad4740-44b1-11e9-b431-00163e005d19
Spec:
  Acme:
    Config:
      Domains:
        zariga.com
      Http 01:
        Ingress:        
        Ingress Class:  nginx
  Dns Names:
    zariga.com
  Issuer Ref:
    Kind:       ClusterIssuer
    Name:       letsencrypt-prod
  Secret Name:  tls-prod-cert
Status:
  Acme:
    Order:
      URL:  https://acme-v02.api.letsencrypt.org/acme/order/53135536/352104603
  Conditions:
    Last Transition Time:  2019-03-12T10:27:00Z
    Message:               Order validated
    Reason:                OrderValidated
    Status:                False
    Type:                  ValidateFailed
    Last Transition Time:  <nil>
    Message:               Certificate issued successfully
    Reason:                CertIssued
    Status:                True
    Type:                  Ready
Events:
  Type    Reason        Age   From          Message
  ----    ------        ----  ----          -------
  Normal  CreateOrder   27s   cert-manager  Created new ACME order, attempting validation...
  Normal  IssueCert     27s   cert-manager  Issuing certificate...
  Normal  CertObtained  25s   cert-manager  Obtained certificate from ACME server
  Normal  CertIssued    25s   cert-manager  Certificate issued successfully

Ingress Details

root@kube-master:~# kubectl describe ingress
Name:             nginx-ingress-prod
Namespace:        default
Address:          
Default backend:  my-nginx:80 (192.168.123.202:80)
TLS:
  tls-prod-cert terminates zariga.com
Rules:
  Host  Path  Backends
  ----  ----  --------
  *     *     my-nginx:80 (192.168.123.202:80)
Annotations:
  kubernetes.io/ingress.class:        nginx
  kubernetes.io/tls-acme:             true
  certmanager.k8s.io/cluster-issuer:  letsencrypt-prod
Events:
  Type    Reason             Age    From                      Message
  ----    ------             ----   ----                      -------
  Normal  CREATE             7m13s  nginx-ingress-controller  Ingress default/nginx-ingress-prod
  Normal  CreateCertificate  7m8s   cert-manager              Successfully created Certificate "tls-prod-cert"
  Normal  UPDATE             6m57s  nginx-ingress-controller  Ingress default/nginx-ingress-prod

Letsencrypt Nginx Production Definition

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nginx-ingress-prod
  annotations:
    kubernetes.io/ingress.class: nginx
    certmanager.k8s.io/cluster-issuer: letsencrypt-prod
    kubernetes.io/tls-acme: 'true'
  labels:
    app: 'my-nginx'
spec:
  backend:
    serviceName: my-nginx
    servicePort: 80
  tls:
  - secretName: tls-prod-cert
    hosts:
    - zariga.com

Answer 1

Maybe would be helpful for someone experiencing similar issues. As for me, a forgot to specify hostname in Ingress yaml file for both rules and tls sections. After duplicating the hostname, it started responding with a proper certificate.

Example:

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: test-web-ingress
  annotations:
    kubernetes.io/ingress.class: nginx
spec:
  tls:
  - hosts:
    - my.host.com                # <----
    secretName: tls-secret
  rules:
    - host: my.host.com          # <----
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              serviceName: my-nginx
              servicePort: 80

Answer 2

Sometimes it may happen if you are using the clusterissuer URL as staging URL.

Check the letsencrypt url set in you issuer.yaml or clusterissuer.yaml and change it to production url: https://acme-v02.api.letsencrypt.org/directory

I faced the same issue once and changing the url to production url solved it.

Also check that the ingress tls secrets you are using is right.

Actual cluster issuer should be something like for production :

apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
  name: dev-clusterissuer
spec:
  acme:
    email: [email protected]
    privateKeySecretRef:
      name: dev-clusterissuer
    server: https://acme-v02.api.letsencrypt.org/directory       # <----check this server URL it is for Prod and use this only
    solvers:
    - http01:
        ingress:
          class: nginx

If you are using server: https://acme-staging-v02.api.letsencrypt.org/directory you will face issue better replace it with server: https://acme-v02.api.letsencrypt.org/directory