Container is not accessible via kubernetes readiness probe

Question

I see this error in kubectl describe podName:

  9m        2s      118 kubelet, gke-wordpress-default-pool-2e82c1f4-0zpw   spec.containers{nginx}      Warning     Unhealthy       Readiness probe failed: Get http://10.24.0.27:80/: net/http: request canceled (Client.Timeout exceeded while awaiting headers)

The container logs (nginx) is of the following:

10.24.1.1 - - [22/Aug/2017:11:09:51 +0000] "GET / HTTP/1.1" 499 0 "-" "Go-http-client/1.1"

However if I exec into the container via kubectl exec -it podName -c nginx sh, and do a wget http://localhost I am able to succesfully get a HTTP 200 response. As well as if I SSH into the host (GCP compute instance), I'm able to successfully get a HTTP 200 response.

I believe this issue occurred shortly after I replace a LoadBalancer service with a NodePort service. I wonder if it's some port conflict?

The service in question: wordpress-service.yaml

apiVersion: v1
kind: Service
metadata:
  labels:
    app: wordpress
  name: wordpress
spec:
  type: NodePort
  ports:
    - port: 80
      targetPort: 80
      protocol: TCP
  selector:
    app: wordpress

The container is a Nginx container serving content on port 80.

What might be the cause of the readiness probe failing?

---

If I remove the readiness probe in my config:

      readinessProbe:
        httpGet:
          path: /
          port: 80
        initialDelaySeconds: 5 
        periodSeconds: 5

Everything works fine, the pods are able to be accessed via a LoadBalancer service.

Answer 1

ARGHUGIHIRHHHHHH.

I've been staring at this error for a day at least and for some reason I didn't comprehend it.

Essentially the error net/http: request canceled (Client.Timeout exceeded while awaiting headers) means the container took longer then the timeout period (default of 1s).