All containers showing ready, but pod is not ready

Question

when I describe my pod, I can see the following conditions:

$ kubectl describe pod blah-84c6554d77-6wn42
...
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   True 
  PodScheduled      True
...
$ kubectl get pods -o wide
NAME                    READY   STATUS    RESTARTS   AGE   IP              NODE                                        NOMINATED NODE   READINESS GATES
blah-84c6554d77-6wn42   1/1     Running   46         23h   10.247.76.179   xxx-x-x-xx-123.nodes.xxx.d.ocp.xxx.xxx.br   <none>           <none>
...

I wonder how can this be possible: All the containers in the pod are showing ready=true but pod is ready=false

Has anyone experienced this before? Do you know what else could be causing the pod to not be ready?

I'm running kubernetes version 1.15.4. I can see on the code that

// The status of "Ready" condition is "True", if all containers in a pod are ready
// AND all matching conditions specified in the ReadinessGates have status equal to "True".

but, I haven't defined any custom Readiness Gates. I wonder how can I check what's the reason for the check failure. I couldn't find this on the docs for pod-readiness-gate

here is the full pod yaml

$ kubectl get pod blah-84c6554d77-6wn42 -o yaml
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: "2019-10-17T04:05:30Z"
  generateName: blah-84c6554d77-
  labels:
    app: blah
    commit: cb511424a5ec43f8dghdfdwervxz8a19edbb
    pod-template-hash: 84c6554d77
  name: blah-84c6554d77-6wn42
  namespace: default
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: ReplicaSet
    name: blah-84c6554d77
    uid: 514da64b-c242-11e9-9c5b-0050123234523
  resourceVersion: "19780031"
  selfLink: /api/v1/namespaces/blah/pods/blah-84c6554d77-6wn42
  uid: 62be74a1-541a-4fdf-987d-39c97644b0c8
spec:
  containers:
  - env:
    - name: URL
      valueFrom:
        configMapKeyRef:
          key: url
          name: external-mk9249b92m
    image: myregistry/blah:3.0.0
    imagePullPolicy: IfNotPresent
    livenessProbe:
      failureThreshold: 5
      httpGet:
        path: /healthcheck
        port: 8080
        scheme: HTTP
      initialDelaySeconds: 30
      periodSeconds: 30
      successThreshold: 1
      timeoutSeconds: 3
    name: blah
    ports:
    - containerPort: 8080
      name: http
      protocol: TCP
    readinessProbe:
      failureThreshold: 10
      httpGet:
        path: /healthcheck
        port: 8080
        scheme: HTTP
      initialDelaySeconds: 10
      periodSeconds: 30
      successThreshold: 1
      timeoutSeconds: 3
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: default-token-4tp6z
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  nodeName: xxxxxxxxxxx
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: default
  serviceAccountName: default
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - name: default-token-4tp6z
    secret:
      defaultMode: 420
      secretName: default-token-4tp6z
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2019-10-17T04:14:22Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2019-10-17T09:47:15Z"
    status: "False"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2019-10-17T07:54:55Z"
    status: "True"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2019-10-17T04:14:18Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: docker://33820f432a5a372d028c18f1b0376e2526ef65871f4f5c021e2cbea5dcdbe3ea
    image: myregistry/blah:3.0.0
    imageID: docker-pullable://myregistry/blah:@sha256:5c0634f03627bsdasdasdasdasdasdc91ce2147993a0181f53a
    lastState:
      terminated:
        containerID: docker://5c8d284f79aaeaasdasdaqweqweqrwt9811e34da48f355081
        exitCode: 1
        finishedAt: "2019-10-17T07:49:36Z"
        reason: Error
        startedAt: "2019-10-17T07:49:35Z"
    name: blah
    ready: true
    restartCount: 46
    state:
      running:
        startedAt: "2019-10-17T07:54:39Z"
  hostIP: 10.247.64.115
  phase: Running
  podIP: 10.247.76.179
  qosClass: BestEffort
  startTime: "2019-10-17T04:14:22Z"

Thanks

Answer 1

You got a ReadinessProbe:

    readinessProbe:
      failureThreshold: 10
      httpGet:
        path: /healthcheck
        port: 8080
        scheme: HTTP

https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#define-readiness-probes

Readiness probes are configured similarly to liveness probes. The only difference is that you use the readinessProbe field instead of the livenessProbe field.

tl;dr; check if /healthcheck on port 8080 is returning a successful HTTP status code and, if not used or not necessary, drop it.