Start kubernetes pod memory depending on size of data job

Question

is there a way to scale dynamically the memory size of Pod based on size of data job (my use case)?

Currently we have Job and Pods that are defined with memory amounts, but we wouldn't know how big the data will be for a given time-slice (sometimes 1000 rows, sometimes 100,000 rows).
So it will break if the data is bigger than the memory we have allocated beforehand.

I have thought of using slices by data volume, i.e. cut by every 10,000 rows, we will know memory requirement of processing a fixed amount of rows. But we are trying to aggregate by time hence the need for time-slice.

Or any other solutions, like Spark on kubernetes?

Another way of looking at it:
How can we do an implementation of Cloud Dataflow in Kubernetes on AWS

Answer 1

It's a best practice always define resources in your container definition, in particular:

• limits:the upper level of CPU and memory
• requests: the minimal level of CPU and memory

This allows the scheduler to take a better decision and it eases the assignment of Quality of Service (QoS) for each pod (https://kubernetes.io/docs/tasks/configure-pod-container/quality-service-pod/) which falls into three possible classes:

• Guaranteed (highest priority): when requests = limits
• Burstable: when requests < limits
• BestEffort (lowest priority): when requests and limits are not set.

The QoS enables a criterion for killing pods when the system is overcommited.

Answer 2

If you don’t know the memory requirement for your pod a priori for a given time-slice, then it is difficult for Kubernete Cluster Autoscaler to automatically scale node pool for you as per this documentation [1]. Therefore for both of your suggestions like running either Cloud Dataflow or Spark on Kubernete with Kubernete Cluster Autoscaler, may not work for your case.

However, you can use custom scaling as a workaround. For example, you can export memory related metrics of the pod to Stackdriver, then deploy HorizontalPodAutoscaler (HPA) resource to scale your application as [2].

[1] https://cloud.google.com/kubernetes-engine/docs/concepts/cluster-autoscaler#how_cluster_autoscaler_works

[2] https://cloud.google.com/kubernetes-engine/docs/tutorials/custom-metrics-autoscaling