Ensure Kubernetes Deployment has completed and all pods are updated and available

Question

The status of a deployment indicates that you can look at a deployments observedGeneration vs generation and when observedGeneration >= generation then the deployment succeeded. That's fine, but I'm interested in knowing when the new container is actually running in all of my pods, so that if I hit a service I know for sure I'm hitting a server that represents the latest deployed container.

Another tip from a K8S Slack member:

kubectl get deployments | grep <deployment-name> | sed 's/ /,/g' | cut -d ' ' -f 4

I deployed a bad image, resulting in ErrImagePull, yet the deployment still reported the correct number of 8 up-date-date replicas (available replicas was 7).

Answer 1

Update #2: Kubernetes 1.5 will ship with a much better version of kubectl rollout status and improve even further in 1.6, possibly replacing my custom solution/script laid out below.

Update #1: I have turned my answer into a script hosted on Github which has received a small number of improving PRs by now.

Original answer:

First of all, I believe the kubectl command you got is not correct: It replaces all white spaces by commas but then tries to get the 4th field after separating by white spaces.

In order to validate that a deployment (or upgrade thereof) made it to all pods, I think you should check whether the number of available replicas matches the number of desired replicas. That is, whether the AVAILABLE and DESIRED columns in the kubectl output are equal. While you could get the number of available replicas (the 5th column) through

kubectl get deployment nginx | tail -n +2 | awk '{print $5}'

and the number of desired replicas (2nd column) through

kubectl get deployment nginx | tail -n +2 | awk '{print $2}'

a cleaner way is to use kubectl's jsonpath output, especially if you want to take the generation requirement that the official documentation mentions into account as well.

Here's a quick bash script I wrote that expects to be given the deployment name on the command line, waits for the observed generation to become the specified one, and then waits for the available replicas to reach the number of the specified ones:

#!/bin/bash
set -o errexit
set -o pipefail
set -o nounset

deployment=

get_generation() {
  get_deployment_jsonpath '{.metadata.generation}'
}

get_observed_generation() {
  get_deployment_jsonpath '{.status.observedGeneration}'
}

get_replicas() {
  get_deployment_jsonpath '{.spec.replicas}'
}

get_available_replicas() {
  get_deployment_jsonpath '{.status.availableReplicas}'
}

get_deployment_jsonpath() {
  local readonly _jsonpath="$1"

  kubectl get deployment "${deployment}" -o "jsonpath=${_jsonpath}"
}

if [[ $# != 1 ]]; then
  echo "usage: $(basename $0) <deployment>" >&2
  exit 1
fi

readonly deployment="$1"

readonly generation=$(get_generation)
echo "waiting for specified generation ${generation} to be observed"
while [[ $(get_observed_generation) -lt ${generation} ]]; do
  sleep .5
done
echo "specified generation observed."

readonly replicas="$(get_replicas)"
echo "specified replicas: ${replicas}"

available=-1
while [[ ${available} -ne ${replicas} ]]; do
  sleep .5
  available=$(get_available_replicas)
  echo "available replicas: ${available}"
done

echo "deployment complete."