How to Configure Pod initialization in a specific order in Kubernetes?

Question

I want to know how I can start my deployments in a specific order. I am aware of initContainers but that is not working for me. I have a huge platform with around 20 deployments and 5 statefulsets that each of them has their own service, environment variables, volumes, horizontal autoscaler, etc. So it is not possible (or I don't know how) to define them in another yaml deployment as initContainers.

Is there another option to launch deployments in a specific order?

Answer 1

To create dependency among the deployment, there needs to be a sequence of certain condition true.

For example, Wait for the pod "busybox1" to contain the status condition of type "Ready".

kubectl wait --for=condition=Ready pod/busybox1

after that, you can roll out the next deployment.

For further detail see kubectl-wait

Here is an another example from @Michael Hausenblas job-dependencies, having dependencies among the job objects.

If you’d like to kick off another job after worker has completed? Here you go:

$ kubectl -n waitplayground \
             wait --for=condition=complete --timeout=32s \     
             job/worker
job.batch/worker condition met

Answer 2

It's possible to order the launch of initContainers in a Pod, or Pods that belong in the same StatefulSet. However, those solutions do not apply to your case.

This is because ordering initialization is not the standard approach for solving your issue. In a microservices architecture, and more specifically Kubernetes, you would write your containers such that they try to call the services they depend on (whether they are up or not) and if they aren't available, you let your containers crash. This works because Kubernetes provides a self-healing mechanism that automatically restarts containers if they fail. This way, your containers will try to connect to the services they depend on, and if the latter aren't available, the containers will crash and try again later using exponential back-off.

By removing unnecessary dependencies between services, you simplify the deployment of your application and reduce coupling between different services.

Answer 3

Just launch them all in parallel and let them crash. Kubernetes will restart the failing ones after some delay.

Say you have service A, which depends on B, which depends on C. A starts first and as part of its startup sequence tries to make a call to B. That fails (because B isn't up) and the pod changes to Error status. It may retry once or twice and then go to CrashLoopBackOff status. Kubernetes pauses for a couple of seconds before retrying again. The same thing will happen for B.

Eventually service C (at the bottom of the stack) will come up, and some time after that the automated restart will start B (immediately above it). This time B will start successfully. Some time after that an automated restart will start A, which this time will come up successfully.

The one thing you need to be aware of is that if a Pod does wind up in CrashLoopBackOff state, it could be because of a code bug, or a misconfiguration, or just because a service it depends on isn't up yet. You'll need to look at kubectl logs (and make sure your service code writes out usable diagnostics) to understand which case you're in.

Answer 4

I hope your containers have liveness probe defined. Use them in the dependant deployment to create a initContainer that will check for the app is ready. After the initContainer verifies that the other container is ready, the dependant container starts.

What exactly is the issue you faced with initContainer? A sample link where initContainer is used to start the dependant container is here.

One other approach would be to write a shell wrapper and then create the initial deployment. Then use an until loop to wait till the initial deployment status is ready. Then trigger the deployment that depends on the initial deployment.