How is the preemption notice handled?

Question

I'm currently running on AWS and use kube-aws/kube-spot-termination-notice-handler to intercept an AWS spot termination notice and gracefully evict the pods.

I'm reading this GKE documentation page and I see:

Preemptible instances terminate after 30 seconds upon receiving a preemption notice.

Going into the Compute Engine documentation, I see that a ACPI G2 Soft Off is sent 30 seconds before the termination happens but this issue suggests that the kubelet itself doesn't handle it.

So, how does GKE handle preemption? Will the node do a drain/cordon operation or does it just do a hard shutdown?

Answer 1

Yes you are right, so far there is no built in way to handle ACPI G2 Soft Off.

Notice that if normal preemptible instance supports shutdown scripts (where you could introduce some kind of logic to perform drain/cordon), this is not the case if they are Kubernetes nodes:

Currently, preemptible VMs do not support shutdown scripts.

You can perform some test but quoting again from documentation:

You can simulate an instance preemption by stopping the instance.

And so far if you stop the instance, even if it is a Kubernetes node no action is taken to cordon/drain and gratefully remove the node from the cluster.

However this feature is still in beta therefore it is at its early stage of life and in this moment it is a matter of discussion if and how introduce this feature.

Disclaimer: I work For Google Cloud Platform Support