Menu
I have a Kubernetes cluster running various apps with different machine types (ie. cpu-heavy, gpu, ram-heavy) and installed cluster-autoscaler (CA) to manage the Auto Scaling Groups (ASG) using auto-discovery.
Kubernetes version: EKS 1.11
Cluster Autoscaler: v1.13.2
I have configured my ASGs such that they contain the appropriate CA tags. These ASGs are usually scaled down to 0 nodes, and will be scaled up according to workload. Going into the CA logs, I have also verified that the CA acknowledges the existence of the ASGs. However, whenever I try to create pods with nodeSelectors, the CAs doesn't scale up the corresponding ASG and responds with "predicate failed: nodeSelector(s) did not match". I have added the appropriate node labels to the ASG's tags as well.
Could it be a limitation due to scaling up from 0 nodes?
Can't seem to find similar problems online. Any feedback would help! Thanks!
539
Cluster Autoscaler will still scale down underutilized nodes regardless of the relative sizes of underlying node groups. We plan to take balancing into account in scale-down in the future. Cluster Autoscaler will only add as many nodes as required to run all existing pods.
Open the Cluster Autoscaler releases page in a web browser and find the latest Cluster Autoscaler version that matches your cluster's Kubernetes major and minor version. For example, if your cluster's Kubernetes version is 1.23 find the latest Cluster Autoscaler release that begins with 1.23 .
The Cluster Autoscaler automatically adds or removes nodes in a cluster based on resource requests from pods. The Cluster Autoscaler doesn't directly measure CPU and memory usage values to make a scaling decision.
When enabled, the cluster autoscaler algorithm checks for pending pods. The cluster autoscaler requests a newly provisioned node if: 1) there are pending pods due to not having enough available cluster resources to meet their requests and 2) the cluster or node pool has not reached the user-defined maximum node count.
Update: CA Documentation update
I had the incorrect ASG tag. Ensure that you have k8s.io/cluster-autoscaler/node-template/label/some-label to ensure that CA can see your node labels when the ASG is scaled down to 0. Currently, CA doesn't create node labels based on ASG tags, but this feature is in the pipeline.
155
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With