Kubernetes Pod anti-affinity - evenly spread pods based on a label?

Question

We are finding that our Kubernetes cluster tends to have hot-spots where certain nodes get far more instances of our apps than other nodes.

In this case, we are deploying lots of instances of Apache Airflow, and some nodes have 3x more web or scheduler components than others.

Is it possible to use anti-affinity rules to force a more even spread of pods across the cluster?

E.g. "prefer the node with the least pods of label component=airflow-web?"

If anti-affinity does not work, are there other mechanisms we should be looking into as well?

Answer 1

Try adding this to the Deployment/StatefulSet .spec.template:

      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: "component"
                  operator: In
                  values:
                  - airflow-web
              topologyKey: "kubernetes.io/hostname"