Correct way to define k8s-user-startup-script

Question

This is like a follow-up question of: Recommended way to persistently change kube-env variables

I was playing around with the possibility to define a k8s-user-startup-script for GKE instances (I want to install additional software to each node).

Adding k8s-user-startup-script to an Instance Group Template "Custom Metadata" works, but that is overwritten by gcloud container clusters upgrade which creates a new Instance Template without "inheriting" the additional k8s-user-startup-script Metadata from the current template.

I've also tried to add a k8s-user-startup-script to the project metadata (I thought that would be inherited by all instances of my project like described here) but that is not taken into account.

What is the correct way to define a k8s-user-startup-script that persists cluster upgrades? Or, more general, what is the desired way to customize the GKE nodes?

Answer 1

Google Container Engine doesn't support custom startup scripts for nodes.

As I mentioned in Recommended way to persistently change kube-env variables you can use a DaemonSet to customize your nodes. A DaemonSet running in privileged mode can do pretty much anything that you could do with a startup script, with the caveat that it is done slightly later in the node bring-up lifecycle. Since a DaemonSet will run on all nodes in your cluster, it will be automatically applied to any new nodes that join (via cluster resize) and because it is a Kubernetes API object, it will be persisted across OS upgrades.