Readiness probe failed: Get http://10.32.1.71:80/setting s: net/http: request canceled (Client.Timeout exceeded while awaiting headers)

Question

The configuration I have is Jenkins on Kubernetes and the project is written in PHP.

The issue here is that the pod is attached to an ingress(than on a loadBalancer using GCE) and when the pod is unhealthy it won't add it.

The first time I load the project from 0 it works after I update it, it fails since it's unhealthy.

When I describe the pod I get the following warning:

Readiness probe failed: Get http://10.32.1.71:80/setting s: net/http: request canceled (Client.Timeout exceeded while awaiting headers)

My production configuration:

# Configuration for the SQL connection 
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: wobbl-main-backend-production
spec:
  replicas: 1
  template:
    metadata:
      name: backend
      labels:
        app: wobbl-main
        role: backend
        env: production
    spec:
      containers:
        - name: backend
          image: gcr.io/cloud-solutions-images/wobbl-mobile-backend:1.0.0
          resources:
            limits:
              memory: "500Mi"
              cpu: "100m"
          imagePullPolicy: Always
          readinessProbe:
            httpGet: # make an HTTP request
              port: 80 # port to use
              path: /settings # endpoint to hit
              scheme: HTTP # or HTTPS
            initialDelaySeconds: 3 # how long to wait before checking
            periodSeconds: 5 # how long to wait between checks
            successThreshold: 1 # how many successes to hit before accepting
            failureThreshold: 2 # how many failures to accept before failing
            timeoutSeconds: 10 # how long to wait for a response
          ports:
          - name: backend
            containerPort: 80

Any hints on how to solve this.

Answer 1

The error message implies your HTTP request is not successful. The readiness probe needs to succeed for the pod to be added as an endpoint for the service exposing it.

1) kubectl get po -o wide

This is so you can get the pod's cluster IP

2) kubectl exec -t [another_pod] -- curl -I [pod's cluster IP]

If you get a 200 response, you know the path is configured properly and the readiness probe should be passing. If you get anything other than a 200 response, this is why the readiness probe fails and you need to check your image.

Answer 2

The workaround from github.com/kubernetes/kubernetes/issues/89898

You might want to change httpGet to the exec / command / curl:

      ...
      readinessProbe:
        exec:
          command:
          - "curl"
          - "--fail"
          - "-o"
          - "/dev/null"
          - "http://localhost/settings"
        initialDelaySeconds: 3
        periodSeconds: 5
        successThreshold: 1
        failureThreshold: 2
        timeoutSeconds: 10