Kubernetes TLS Ingress route with cert-manager and SelfSigned ClusterIssuer not working

Question

I am trying to work with TLS in our Kubernetes cluster. I've followed MS documentation on "Create an HTTPS ingress controller on Azure Kubernetes Service" (https://learn.microsoft.com/en-us/azure/aks/ingress-tls).

I've deployed a nginx-ingress controller, added the DNS record and installed the cert-manager. I created a CA ClusterIssuer of SelfSigned and also created the 2 demo applications.

When I created the ingress route, the certificate created automatically and with "True" on the Ready status, but the route is not working - I can't access the demo applications with the host name deployed (https://hello-world-ingress.<Ingress_Service_DNS_Name>).

The Self-Signed ClusterIssuer:

apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
  name: selfsigned-issuer
spec:
  selfSigned: {}

The Ingress route:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: hello-world-ingress
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/rewrite-target: /$2
    cert-manager.io/cluster-issuer: selfsigned-issuer
spec:
  tls:
  - hosts:
    - hello-world-ingress.<Ingress_Service_DNS_Name>
    secretName: tls-secret
  rules:
  - host: hello-world-ingress.<Ingress_Service_DNS_Name>
    http:
      paths:
      - backend:
          serviceName: aks-helloworld
          servicePort: 80
        path: /(.*)
      - backend:
          serviceName: aks-helloworld-two
          servicePort: 80
        path: /hello-world-two(/|$)(.*)
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: hello-world-ingress-static
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/rewrite-target: /static/$2
    cert-manager.io/cluster-issuer: selfsigned-issuer
spec:
  tls:
  - hosts:
    - hello-world-ingress.<Ingress_Service_DNS_Name>
    secretName: tls-secret
  rules:
  - host: hello-world-ingress.<Ingress_Service_DNS_Name>
    http:
      paths:
      - backend:
          serviceName: aks-helloworld
          servicePort: 80
        path: /static(/|$)(.*)

I've created a DNS record on GoDaddy in our domain for <Ingress_Service_DNS_Name> (but with the real name) that points to the external ingress controller service IP Address.

The rest of the installations and deployments are the same as the documentation.

Does anyone has any idea why it's not working?

---------------- Edit ----------------------

Ingress-controller logs:

I0330 06:03:16.780788       7 event.go:281] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"ingress-basic", Name:"hello-world-ingress", UID:"488a4c00-7072-11ea-a46c-1a8c7fb34cf9", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"37375594", FieldPath:""}): type: 'Normal' reason: 'UPDATE' Ingress ingress-basic/hello-world-ingressI0330 06:03:46.358414       7 event.go:281] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"ingress-basic", Name:"hello-world-ingress-static", UID:"48b91e0e-7072-11ea-a46c-1a8c7fb34cf9", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"37375687", FieldPath:""}): type: 'Normal' reason: 'UPDATE' Ingress ingress-basic/hello-world-ingress-static
I0330 06:03:46.386930       7 event.go:281] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"ingress-basic", Name:"hello-world-ingress", UID:"488a4c00-7072-11ea-a46c-1a8c7fb34cf9", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"37375688", FieldPath:""}): type: 'Normal' reason: 'UPDATE' Ingress ingress-basic/hello-world-ingress
I0330 06:04:16.783483       7 event.go:281] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"ingress-basic", Name:"hello-world-ingress", UID:"488a4c00-7072-11ea-a46c-1a8c7fb34cf9", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"37375802", FieldPath:""}): type: 'Normal' reason: 'UPDATE' Ingress ingress-basic/hello-world-ingress
I0330 06:04:16.788210       7 event.go:281] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"ingress-basic", Name:"hello-world-ingress-static", UID:"48b91e0e-7072-11ea-a46c-1a8c7fb34cf9", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"37375803", FieldPath:""}): type: 'Normal' reason: 'UPDATE' Ingress ingress-basic/hello-world-ingress-static
I0330 06:04:46.584035       7 event.go:281] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"ingress-basic", Name:"hello-world-ingress", UID:"488a4c00-7072-11ea-a46c-1a8c7fb34cf9", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"37375904", FieldPath:""}): type: 'Normal' reason: 'UPDATE' Ingress ingress-basic/hello-world-ingress
I0330 06:04:46.587677       7 event.go:281] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"ingress-basic", Name:"hello-world-ingress-static", UID:"48b91e0e-7072-11ea-a46c-1a8c7fb34cf9", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"37375905", FieldPath:""}): type: 'Normal' reason: 'UPDATE' Ingress ingress-basic/hello-world-ingress-static
I0330 06:05:16.938952       7 event.go:281] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"ingress-basic", Name:"hello-world-ingress", UID:"488a4c00-7072-11ea-a46c-1a8c7fb34cf9", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"37376008", FieldPath:""}): type: 'Normal' reason: 'UPDATE' Ingress ingress-basic/hello-world-ingress
I0330 06:05:16.938975       7 event.go:281] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"ingress-basic", Name:"hello-world-ingress-static", UID:"48b91e0e-7072-11ea-a46c-1a8c7fb34cf9", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"37376007", FieldPath:""}): type: 'Normal' reason: 'UPDATE' Ingress ingress-basic/hello-world-ingress-static
I0330 06:05:46.337384       7 event.go:281] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"ingress-basic", Name:"hello-world-ingress-static", UID:"48b91e0e-7072-11ea-a46c-1a8c7fb34cf9", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"37376095", FieldPath:""}): type: 'Normal' reason: 'UPDATE' Ingress ingress-basic/hello-world-ingress-static

Cert-manager logs:

I0330 06:16:19.953430       1 reflector.go:432] external/io_k8s_client_go/tools/cache/reflector.go:108: Watch close - *v1alpha2.Order total 0 items received
I0330 06:16:19.989382       1 reflector.go:278] external/io_k8s_client_go/tools/cache/reflector.go:108: forcing resync
I0330 06:16:39.861201       1 metrics.go:304] cert-manager/metrics "msg"="attempting to clean up metrics for recently deleted certificates"  
I0330 06:16:39.861233       1 metrics.go:307] cert-manager/metrics "msg"="active certificates is still uninitialized"  
I0330 06:16:46.353253       1 controller.go:129] cert-manager/controller/ingress-shim "msg"="syncing item" "key"="ingress-basic/hello-world-ingress" 
I0330 06:16:46.354661       1 metrics.go:385] cert-manager/metrics "msg"="incrementing controller sync call count"  "controllerName"="ingress-shim"
I0330 06:16:46.355124       1 sync.go:163] cert-manager/controller/ingress-shim "msg"="certificate already exists for ingress resource, ensuring it is up to date" "related_resource_kind"="Certificate" "related_resource_name"="tls-secret-selfsigned" "related_resource_namespace"="ingress-basic" "resource_kind"="Ingress" "resource_name"="hello-world-ingress" "resource_namespace"="ingress-basic" 
I0330 06:16:46.356804       1 sync.go:176] cert-manager/controller/ingress-shim "msg"="certificate resource is already up to date for ingress" "related_resource_kind"="Certificate" "related_resource_name"="tls-secret-selfsigned" "related_resource_namespace"="ingress-basic" "resource_kind"="Ingress" "resource_name"="hello-world-ingress" "resource_namespace"="ingress-basic" 
I0330 06:16:46.357190       1 controller.go:135] cert-manager/controller/ingress-shim "msg"="finished processing work item" "key"="ingress-basic/hello-world-ingress" 
I0330 06:16:46.358636       1 controller.go:129] cert-manager/controller/ingress-shim "msg"="syncing item" "key"="ingress-basic/hello-world-ingress-static" 
I0330 06:16:46.361782       1 metrics.go:385] cert-manager/metrics "msg"="incrementing controller sync call count"  "controllerName"="ingress-shim"
I0330 06:16:46.367596       1 sync.go:163] cert-manager/controller/ingress-shim "msg"="certificate already exists for ingress resource, ensuring it is up to date" "related_resource_kind"="Certificate" "related_resource_name"="tls-secret-selfsigned" "related_resource_namespace"="ingress-basic" "resource_kind"="Ingress" "resource_name"="hello-world-ingress-static" "resource_namespace"="ingress-basic" 
I0330 06:16:46.368271       1 sync.go:171] cert-manager/controller/ingress-shim "msg"="certificate resource is not owned by this ingress. refusing to update non-owned certificate resource for ingress" "related_resource_kind"="Certificate" "related_resource_name"="tls-secret-selfsigned" "related_resource_namespace"="ingress-basic" "resource_kind"="Ingress" "resource_name"="hello-world-ingress-static" "resource_namespace"="ingress-basic" 
I0330 06:16:46.368424       1 controller.go:135] cert-manager/controller/ingress-shim "msg"="finished processing work item" "key"="ingress-basic/hello-world-ingress-static" 
I0330 06:16:47.581355       1 reflector.go:278] external/io_k8s_client_go/tools/cache/reflector.go:108: forcing resync
I0330 06:16:49.383317       1 reflector.go:278] external/io_k8s_client_go/tools/cache/reflector.go:108: forcing resync

The only thing that looks like it can be a problem is in the cert manager logs:

"certificate resource is not owned by this ingress. refusing to update non-owned certificate resource for ingress" "related_resource_kind"="Certificate" "related_resource_name"="tls-secret-selfsigned" "related_resource_namespace"="ingress-basic" "resource_kind"="Ingress" "resource_name"="hello-world-ingress-static" "resource_namespace"="ingress-basic" "

Thanks,

Afik

Answer 1

Based on the information provided a believe that the problem is two ingresses using the same self-signed certificate.

What you trying to achieve here is that you want to manage your certificate from two different places. As the documentation states:

Deploy a TLS Ingress Resource - “There are two primary ways to do this: using annotations on the ingress with ingress-shim or directly creating a certificate resource.”

So your hello-world-ingress can use the annotation:

cert-manager.io/cluster-issuer: selfsigned-issuer

But the helo-world-ingress-static cant because the certificate has been already created under secretName: tls-secret.

So from the hello-world-ingress-static you should remove the annotation:

cert-manager.io/cluster-issuer: selfsigned-issuer

Because it creates interest conflict since the secretName is already created and managed by other resource. In this case CertificateRequest from another Ingress.

Let me know if this helps.