Kubernetes 'nginx ingress' unable to find service

Question

I have done a sample application to deploy Jenkins on Kubernetes and exposing the same using Ingress. When I access the jenkins pod via NodePort it is working but when I try to access it via Ingress / Nginx setup I am getting 404

I did google around and tried a few workarounds but none have worked so far. Here are the details for the files

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  annotations:
    kompose.cmd: kompose -f ../docker-compose.yml -f ../docker-compose.utils.yml -f
      ../docker-compose.demosite.yml convert
    kompose.version: 1.17.0 (a74acad)
  creationTimestamp: null
  labels:
    io.kompose.service: ci
  name: ci
spec:
  replicas: 1
  strategy:
    type: Recreate
  template:
    metadata:
      creationTimestamp: null
      labels:
        io.kompose.service: ci
    spec:
      containers:
      - image: jenkins/jenkins:lts
        name: almsmart-ci
        ports:
        - containerPort: 8080
        env:
          - name: JENKINS_USER
            value: admin
          - name: JENKINS_PASS
            value: admin
          - name: JAVA_OPTS
            value: -Djenkins.install.runSetupWizard=false
          - name: JENKINS_OPTS
            value: --prefix=/ci
        imagePullPolicy: Always
        resources: {}
      restartPolicy: Always
status: {}
---

apiVersion: v1
kind: Service
metadata:
  annotations:
    kompose.cmd: kompose -f ../docker-compose.yml -f ../docker-compose.utils.yml -f
      ../docker-compose.demosite.yml convert
    kompose.version: 1.17.0 (a74acad)
  creationTimestamp: null
  labels:
    io.kompose.service: ci
  name: ci
spec:
  type : NodePort
  ports:
  - port: 8080
    protocol: TCP
    targetPort: 8080
  selector:
    io.kompose.service: ci
status:
  loadBalancer: {}

Here is my ingress definition

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: demo-ingress
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/cors-allow-credentials: "true"
    nginx.ingress.kubernetes.io/cors-allow-headers: Authorization, origin, accept
    nginx.ingress.kubernetes.io/cors-allow-methods: GET, OPTIONS
    nginx.ingress.kubernetes.io/enable-cors: "true"
    nginx.ingress.kubernetes.io/ssl-redirect: "false"
spec:
  rules:
   - http:
      paths:
      - path: /ci
        backend:
          serviceName: ci
          servicePort: 8080

When I checked the logs in nginx controller I am seeing the following

I0222 19:59:45.826062       6 controller.go:172] Configuration changes detected, backend reload required.
I0222 19:59:45.831627       6 event.go:221] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"almsmart-ingress", UID:"444858e5-36d9-11e9-9e29-080027811fa3", APIVersion:"extensions/v1beta1", ResourceVersion:"198832", FieldPath:""}): type: 'Normal' reason: 'DELETE' Ingress default/almsmart-ingress
I0222 19:59:46.063220       6 controller.go:190] Backend successfully reloaded.
[22/Feb/2019:19:59:46 +0000]TCP200000.000
W0222 20:00:00.870990       6 endpoints.go:76] Error obtaining Endpoints for Service "default/ci": no object matching key "default/ci" in local store
W0222 20:00:00.871023       6 controller.go:842] Service "default/ci" does not have any active Endpoint.
I0222 20:00:00.871103       6 controller.go:172] Configuration changes detected, backend reload required.
I0222 20:00:00.872556       6 event.go:221] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"almsmart-ingress", UID:"6fc5272c-36dc-11e9-9e29-080027811fa3", APIVersion:"extensions/v1beta1", ResourceVersion:"198872", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress default/almsmart-ingress
I0222 20:00:01.060291       6 controller.go:190] Backend successfully reloaded.
[22/Feb/2019:20:00:01 +0000]TCP200000.000
W0222 20:00:04.205398       6 controller.go:842] Service "default/ci" does not have any active Endpoint.
[22/Feb/2019:20:00:09 +0000]TCP200000.000
10.244.0.0 - [10.244.0.0] - - [22/Feb/2019:20:00:36 +0000] "GET /ci/ HTTP/1.1" 404 274 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:65.0) Gecko/20100101 Firefox/65.0" 498 0.101 [default-ci-8080] 10.244.1.97:8080 315 0.104 404 b5b849647749e2b626f00c011c15bc4e
10.244.0.0 - [10.244.0.0] - - [22/Feb/2019:20:00:46 +0000] "GET /ci HTTP/1.1" 404 274 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:65.0) Gecko/20100101 Firefox/65.0" 497 0.003 [default-ci-8080] 10.244.1.97:8080 315 0.004 404 ac8fbe2faa37413f5e533ed3c8d98a7d
10.244.0.0 - [10.244.0.0] - - [22/Feb/2019:20:00:49 +0000] "GET /ci/ HTTP/1.1" 404 274 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:65.0) Gecko/20100101 Firefox/65.0" 498 0.003 [default-ci-8080] 10.244.1.97:8080 315 0.004 404 865cb82af7f570f2144ef27fdea850c9
I0222 20:00:54.871828       6 status.go:388] updating Ingress default/almsmart-ingress status from [] to [{ }]
I0222 20:00:54.877693       6 event.go:221] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"almsmart-ingress", UID:"6fc5272c-36dc-11e9-9e29-080027811fa3", APIVersion:"extensions/v1beta1", ResourceVersion:"198974", FieldPath:""}): type: 'Normal' reason: 'UPDATE' Ingress default/almsmart-ingress

When I try

kubectl get endpoints, I get the following

NAME                            ENDPOINTS                        AGE
ci                              10.244.1.97:8080                 31m

The default 404 page is available so I assume Ingress Controller is working fine but not sure why it is not able to find the service. I have all the objects in default namespace only and working but still unable to access it using nginx ingress.

Answer 1

+1 for this well asked question.

You setup seemed OK to me. One problem is that, you have --prefix=/ci configured for your Jenkins, but you configured nginx.ingress.kubernetes.io/rewrite-target: / for your ingress. This would cause rewrite of your route: xxx/ci => xxx/. I think the 404 is returned by your Jenkins.

You could try to modify your rewrite rule to nginx.ingress.kubernetes.io/rewrite-target: /ci and see if this works for you.