Deployment fails on anti-affinity rule with topologyKey: kubernetes.io/hostname - Required value: can not be empty

Question

I have an anti-affinity rule that asks kubernetes to schedule pods from the same deployment onto different nodes, we've used it successfully for a long time.

affinity:
  podAntiAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
    - topologyKey: kubernetes.io/hostname
      labelSelector:
        matchExpressions:
        - key: application
          operator: In
          values:
          - {{ $appName }}
        - key: proc  
          operator: In
          values:
          - {{ $procName }}

I'm trying to update my pod affinity rules to be a strong preference instead of a hard requirement, so that we don't need to expand our cluster if a deployment needs more replicas than there are nodes available.

affinity:
  podAntiAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
    - topologyKey: kubernetes.io/hostname
      weight: 100
      labelSelector:
        matchExpressions:
        - key: application
          operator: In
          values:
          - {{ $appName }}
        - key: proc 
          operator: In
          values:
          - {{ $procName }}

However, when I try applying the new rules, I get an unexpected error with the topologyKey:

Error: Deployment.apps "core--web" is invalid:
[spec.template.spec.affinity.podAntiAffinity.preferredDuringSchedulingIgnoredDuringExecution[0].podAffinityTerm.topologyKey: Required value: can not be empty,
spec.template.spec.affinity.podAntiAffinity.preferredDuringSchedulingIgnoredDuringExecution[0].podAffinityTerm.topologyKey: Invalid value: "": name part must be non-empty,
spec.template.spec.affinity.podAntiAffinity.preferredDuringSchedulingIgnoredDuringExecution[0].podAffinityTerm.topologyKey: Invalid value: "": name part must consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyName',  or 'my.name',  or '123-abc', regex used for validation is '([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9]')]

The scheduler seems to be getting an empty string value for the topology key, even though all my nodes have a label for the specified key which match the regex:

$ kubectl describe nodes | grep kubernetes.io/hostname
kubernetes.io/hostname=ip-10-x-x-x.ec2.internal
kubernetes.io/hostname=ip-10-x-x-x.ec2.internal
kubernetes.io/hostname=ip-10-x-x-x.ec2.internal
kubernetes.io/hostname=ip-10-x-x-x.ec2.internal

I didn't expect to see a problem like this from a simple change from required to preferred. What did I screw up to cause the topologyKey error?

Answer 1

There's a slight difference between the syntax of required and preferred, note the reference to podAffinityTerm in the error message path:

spec.template.spec.affinity.podAntiAffinity.preferredDuringSchedulingIgnoredDuringExecution[0].podAffinityTerm.topologyKey

The correct syntax for preferred scheduling is:

affinity:
  podAntiAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
    - weight: 100
      podAffinityTerm:
        topologyKey: kubernetes.io/hostname
        labelSelector:
          matchExpressions:
          - key: application
            operator: In
            values:
            - {{ $appName }}
          - key: proc
            operator: In
            values:
            - {{ $procName }}

Note that weight is a top level key, with a sibling of podAffinityTerm which contains the topologyKey and labelSelector.