Kubernetes dynamic Job scaling

Question

I’m finally dipping my toes in the kubernetes pool and wanted to get some advice on the best way to approach a problem I have:

Tech we are using:

• GCP
• GKE
• GCP Pub/Sub

We need to do bursts of batch processing spread out across a fleet and have decided on the following approach:

1. New raw data flows in
2. A node analyses this and breaks the data up into manageable portions which are pushed onto a queue
3. We have a cluster with Autoscaling On and Min Size ‘0’
4. A Kubernetes job spins up a pod for each new message on this cluster
5. When pods can’t pull anymore messages they terminate successfully

The question is:

• What is the standard approach for triggering jobs such as this?
  • Do you create a new job each time or are jobs meant to be long lived and re-run?
• I have only seen examples of using a yaml file however we would probably want the node which did the portioning of work to create the job as it knows how many parallel pods should be run. Would it be recommended to use the python sdk to create the job spec programatically? Or if jobs are long lived would you simply hit the k8 api and modify the parallel pods required then re-run job?

Answer 1

Jobs in Kubernetes are meant to be short-lived and are not designed to be reused. Jobs are designed for run-once, run-to-completion workloads. Typically they are be assigned a specific task, i.e. to process a single queue item.

However, if you want to process multiple items in a work queue with a single instance then it is generally advisable to instead use a Deployment to scale a pool of workers that continue to process items in the queue, scaling the number of pool workers dependent on the number of items in the queue. If there are no work items remaining then you can scale the deployment to 0 replicas, scaling back up when there is work to be done.

To create and control your workloads in Kubernetes the best-practice would be to use the Kubernetes SDK. While you can generate YAML files and shell out to another tool like kubectl using the SDK simplifies configuration and error handling, as well as allowing for simplified introspection of resources in the cluster as well.