kubelet Error while processing event /sys/fs/cgroup/memory/libcontainer_10010_systemd_test_default.slice

Question

I have setup Kubernetes 1.15.3 cluster on Centos 7 OS using systemd cgroupfs. on all my nodes syslog started logging this message frequently.

How to fix this error message?

kubelet: W0907 watcher.go:87 Error while processing event ("/sys/fs/cgroup/memory/libcontainer_10010_systemd_test_default.slice": 0x40000100 == IN_CREATE|IN_ISDIR): readdirent: no such file or directory

Thanks

Answer 1

It's a known issue with a bad interaction with runc; someone observed it is actually caused by a repeated etcd health check but that wasn't my experience on Ubuntu, which exhibits that same behavior on every Node

They allege that updating the runc binary on your hosts will make the problem go away, but I haven't tried that myself

Answer 2

I had exactly the same problem with the same kubernetes version and with the same context -that is changing cgroups to systemd. Github ticket for this error is created here.

After changing container runtime, as it is described in this tutorial to systemd error start popping out in kublete service log.

What worked for me was to update docker and containerd to following versions.

docker: v19.03.5
containerd: v1.2.10

I assume that any version higher than above will fix the problem as well.