OpenFaaS : Receiving Timeout errors during health check of function Pod

Question

Issue:

We have added health check configuration to our function. However pod becomes Unhealthy due to timeout error in liveness and readiness checks and consequently getting restarted. However if I hit same health check url using CURL or browser it returns correct response.

Health check configuration reference.

We are using Kubernetes HPAv2 for auto-scaling (Reference).

test-function.yml

  test-function:
    lang: quarkus-java-with-fonts
    handler: ./test-function
    image: repo.azurecr.io/test-function:0.1
    labels:
      agentpool: openfaas
      com.openfaas.scale.min: "2"
      com.openfaas.scale.max: "10"
      com.openfaas.scale.factor: 0
    annotations:
     com.openfaas.health.http.path: "/health"
     com.openfaas.health.http.initialDelay: "30s"
    environment:
        secret_name: environment-variables
    secrets:
        - environment-variables 
    constraints:
        - agentpool=openfaas
    limits:
      cpu: 1500m
      memory: 1Gi
    requests:
      cpu: 500m
      memory: 500Mi

Error Trace :

Liveness probe failed: Get "http://XX.XXX.XX.XX:8080/health": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Readiness probe failed: Get "http://XX.XXX.XX.XX:8080/health": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

Any idea what could be wrong.

Answer 1

These errors:

Liveness probe failed: Get "http://XX.XXX.XX.XX:8080/health": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Readiness probe failed: Get "http://XX.XXX.XX.XX:8080/health": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

mean that the HTTP request has failed. For the readiness and liveness probe to work properly, this type of request must be successful.

To find out where the problem is, you need to get the pod IP address. Run:

kubectl get pods -o wide

You should see an output similar to this:

NAME                        READY   STATUS    RESTARTS   AGE   IP          NODE                                       NOMINATED NODE   READINESS GATES
<my-pod-name>               1/1     Running   0          25d   10.92.3.4   <my-node-name>                             <none>           1/1

Take your IP and run:

kubectl exec -t <another_pod> -- curl -I <pod's cluster IP>

If you get a 200 response code, it means the endpoint is properly created and configured. Any other answer suggests there is a problem with your image.

See also:

• this similar question (with solution) on github
• very similar question on Stack Overflow
• guide how to set up liveness and readiness probes
• this nice article