Kubernetes: Exclude Node from default scheduling

Question

Is it possible to create a node-pool that the scheduler will ignore by default but that can be targeted by node-selector?

Answer 1

If your node-pool has a static size or at least it's not auto-scaling then this is easy to accomplish.

First, taint the nodes in that pool:

kubectl taint node \
  `kubectl get node -l cloud.google.com/gke-nodepool=my-pool -o name` \
  dedicated=my-pool:NoSchedule

Kubernetes version >= 1.6

Then add affinity and tolerations values under spec: in your Pod(templates) that need to be able to run on these nodes:

spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: dedicated
            operator: In
            values: ["my-pool"]
  tolerations: 
  - key: "key"
    operator: "Equal"
    value: "value"
    effect: "NoSchedule"

Pre 1.6

Then add these annotations to your Pod(templates) that need to be able to run on these nodes:

annotations:
  scheduler.alpha.kubernetes.io/tolerations: >
    [{"key":"dedicated", "value":"my-pool"}]
  scheduler.alpha.kubernetes.io/affinity: >
    {
      "nodeAffinity": {
        "requiredDuringSchedulingIgnoredDuringExecution": {
          "nodeSelectorTerms": [
            {
              "matchExpressions": [
                {
                  "key": "dedicated",
                  "operator": "In",
                  "values": ["my-pool"]
                }
              ]
            }
          ]
        }
      }
    }

See the design doc for more information.

Autoscaling group of nodes

You need to add the --register-with-taints parameter to kubelet:

Register the node with the given list of taints (comma separated <key>=<value>:<effect>). No-op if register-node is false.

In another answer I gave some examples on how to persist that setting. GKE now also has specific support for tainting node pools