kubernetes unhealthy ingress backend

Question

I followed the load balancer tutorial: https://cloud.google.com/container-engine/docs/tutorials/http-balancer which is working fine when I use the Nginx image, when I try and use my own application image though the backend switches to unhealthy.

My application redirects on / (returns a 302) but I added a livenessProbe in the pod definition:

    livenessProbe:       httpGet:         path: /ping         port: 4001         httpHeaders:           - name: X-health-check             value: kubernetes-healthcheck           - name: X-Forwarded-Proto             value: https           - name: Host             value: foo.bar.com 

My ingress looks like:

apiVersion: extensions/v1beta1 kind: Ingress metadata:   name: foo spec:   backend:     serviceName: foo     servicePort: 80   rules:   - host: foo.bar.com 

Service configuration is:

kind: Service apiVersion: v1 metadata:   name: foo spec:   type: NodePort   selector:     app: foo   ports:     - port: 80        targetPort: 4001 

Backends health in ingress describe ing looks like:

backends:       {"k8s-be-32180--5117658971cfc555":"UNHEALTHY"} 

and the rules on the ingress look like:

Rules:   Host  Path    Backends   ----  ----    --------   * *   foo:80 (10.0.0.7:4001,10.0.1.6:4001) 

Any pointers greatly received, I've been trying to work this out for hours with no luck.

Update

I have added the readinessProbe to my deployment but something still appears to hit / and the ingress is still unhealthy. My probe looks like:

    readinessProbe:       httpGet:         path: /ping         port: 4001         httpHeaders:           - name: X-health-check             value: kubernetes-healthcheck           - name: X-Forwarded-Proto             value: https           - name: Host             value: foo.com 

I changed my service to:

kind: Service apiVersion: v1 metadata:   name: foo spec:   type: NodePort   selector:     app: foo   ports:     - port: 4001       targetPort: 4001 

Update2

After I removed the custom headers from the readinessProbe it started working! Many thanks.

Answer 1

You need to add a readinessProbe (just copy your livenessProbe).

It's explained in the GCE L7 Ingress Docs.

Health checks

Currently, all service backends must satisfy either of the following requirements to pass the HTTP health checks sent to it from the GCE loadbalancer: 1. Respond with a 200 on '/'. The content does not matter. 2. Expose an arbitrary url as a readiness probe on the pods backing the Service.

Also make sure that the readinessProbe is pointing to the same port that you expose to the Ingress. In your case that's fine since you have only one port, if you add another one you may run into trouble.

Answer 2

I thought it's worth noting that this is a quite important limitation in the documentation:

Changes to a Pod's readinessProbe do not affect the Ingress after it is created.

After adding my readinessProbe I basically deleted my ingress (kubectl delete ingress <name>) and then applied my yaml file again to re-create it and shortly after everything was working again.