wait for kubernetes pods

First, there's probably a better way to do this with kubectl rollout, but I didn't know that existed till last week.

Anyway ... kubernetes' API is asynchronous, so if an operator that issues a series of kubernetes commands to update various deployments wants to wait till all the pods in those deployments are up and running, then he may take advantage of a script like: kube-wait4-pods

(
    # If new pods are still rolling/starting up, then wait for that to finish
    COUNT=0
    OK_COUNT=0
    # Don't exit till we get 2 consecutive readings with all pods running.
    while [[ "$OK_COUNT" -lt 2 ]]; do
      g3kubectl get pods
      if [[ 0 == "$(g3kubectl get pods -o json |  jq -r '[.items[] | { name: .metadata.generateName, phase: .status.phase, waitingContainers: [ try .status.containerStatuses[] | { waiting:.state|has("waiting"), ready:.ready}|(.waiting==true or .ready==false)|select(.) ]|length }] | map(select(.phase=="Pending" or .phase=="Running" and .waitingContainers > 0)) | length')" ]]; then
        let OK_COUNT+=1
      else
        OK_COUNT=0
      fi
      
      if [[ "$OK_COUNT" -lt 2 ]]; then
        echo ------------
        echo "INFO: Waiting for pods to exit Pending state"
        let COUNT+=1
        if [[ COUNT -gt 30 ]]; then
          echo -e "$(red_color "ERROR:") pods still not ready after 300 seconds"
          exit 1
        fi
        sleep 10
      fi
    done
)

For example - this Jenkins pipeline deploys a new stack of services to a QA environment, then waits till the new versions of pods are deployed before running through an integration test suite.

...
    stage('K8sDeploy') {
      steps {
        withEnv(['GEN3_NOPROXY=true', "vpc_name=$env.KUBECTL_NAMESPACE", "GEN3_HOME=$env.WORKSPACE/cloud-automation"]) {
          echo "GEN3_HOME is $env.GEN3_HOME"
          echo "GIT_BRANCH is $env.GIT_BRANCH"
          echo "GIT_COMMIT is $env.GIT_COMMIT"
          echo "KUBECTL_NAMESPACE is $env.KUBECTL_NAMESPACE"
          echo "WORKSPACE is $env.WORKSPACE"
          sh "bash cloud-automation/gen3/bin/kube-roll-all.sh"
          sh "bash cloud-automation/gen3/bin/kube-wait4-pods.sh || true"
        }
      }
    }
...

BTW - the code referenced above is a product of the great team of developers at the Center for Data Intensive Science.