Changing the CPU Manager Policy in Kubernetes

Question

I'm trying to change the CPU Manager Policy for a Kubernetes cluster that I manage, as described here however, I've run into numerous issues while doing so.

The cluster is running in DigitalOcean and here is what I've tried so far.

• 1. Since the article mentions that --cpu-manager-policy is a kubelet option I assume that I cannot change it via the API Server and have to change it manually on each node. (Is this assumption BTW?)
• 2. I ssh into one of the nodes (droplets in DigitalOcean lingo) and run kubelet --cpu-manager-policy=static command as described in the kubelet CLI reference here. It gives me the message Flag --cpu-manager-policy has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
• 3. So I check the file pointed at by the --config flag by running ps aux | grep kubelet and find that its /etc/kubernetes/kubelet.conf.
• 4. I edit the file and add a line cpuManagerPolicy: static to it, and also kubeReserved and systemReserved because they become required fields if specifying cpuManagerPolicy.
• 5. Then I kill the process that was running the process and restart it. A couple other things showed up (delete this file and drain the node etc) that I was able to get through and was able to restart the kubelet ultimately

I'm a little lost about the following things

• How do I need to do this for all nodes? My cluster has 12 of them and doing all of these steps for each seems very inefficient.
• Is there any way I can set these params from the globally i.e. cluster-wide rather than doing node by node?
• How can I even confirm that what I did actually changed the CPU Manager policy?

Answer 1

One issue with Dynamic Configuration is that in case the node fails to restart, the API does not give a reasonable response back that tells you what you did wrong, you'll have to ssh into the node and tail the kubelet logs. Plus, you have to ssh into every node and set the --dynamic-config-dir flag anyways.

The folllowing worked best for me

1. SSH into the node. Edit

vim /etc/systemd/system/kubelet.service

2. Add the following lines

  --cpu-manager-policy=static \
  --kube-reserved=cpu=1,memory=2Gi,ephemeral-storage=1Gi \
  --system-reserved=cpu=1,memory=2Gi,ephemeral-storage=1Gi \

We need to set the --kube-reserved and --system-reserved flags because they're prerequisties to setting the --cpu-manager-policy flag

3. Then drain the node and delete the following folder

rm -rf /var/lib/kubelet/cpu_manager_state

4. Restart the kubelet

sudo systemctl daemon-reload
sudo systemctl stop kubelet
sudo systemctl start kubelet

6. uncordon the node and check the policy. This assumes that you're running kubectl proxy on port 8001.

curl -sSL "http://localhost:8001/api/v1/nodes/${NODE_NAME}/proxy/configz" | grep cpuManager

Answer 2

If you use a newer k8s version and kubelet is configured by a kubelet configuration file, eg:config.yml. you can just follow the same steps of @satnam mentioned above. but instead of adding --kube-reserved --system-reserved --cpu-manager-policy, you need to add kubeReserved systemReserved cpuManagerPolicy in your config.yml. for example:

systemReserved: 
  cpu: "1" 
  memory: "100m" 
kubeReserved:
  cpu: "1" 
  memory: "100m" 
cpuManagerPolicy: "static"

Meanwhile, be sure your CPUManager is enabled.