Make systemctl work from inside a container in a debian stretch image

Question

Purpose - What do I want to achieve?

I want to access systemctl from inside a container running a kubernetes node (ami: running debian stretch)

Working setup:

• Node AMI: kope.io/k8s-1.10-debian-jessie-amd64-hvm-ebs-2018-08-17

• Node Directories Mounted in the container to make systemctl work:

  • /var/run/dbus
  • /run/systemd
  • /bin/systemctl
  • /etc/systemd/system

Not Working setup:

• Node AMI: kope.io/k8s-1.11-debian-stretch-amd64-hvm-ebs-2018-08-17

• Node Directories Mounted in the container to make systemctl work:

  • /var/run/dbus
  • /run/systemd
  • /bin/systemctl
  • /etc/systemd/system

Debugging in an attempt to solve the problem

To debug this issue with the debian-stretch image not supporting systemctl with the same mounts as debian-jessie

1) I began by spinning up a nginx deployment by mounting the above mentioned volumes in it

kubectl apply -f https://k8s.io/examples/application/deployment.yaml

kubectl exec -it nginx-deployment /bin/bash

root@nginx-deployment-788f65877d-pzzrn:/# systemctl
systemctl: error while loading shared libraries: libsystemd-shared- 
232.so: cannot open shared object file: No such file or directory

2) As the above issue showed the file libsystemd-shared-232.so not found. I found the actual path by looking into the node.

admin@ip-10-0-20-11:~$ sudo find / -iname 'libsystemd-shared-232.so'
/lib/systemd/libsystemd-shared-232.so

3) Mounted the /lib/systemd in the nginx pod and ran the systemctl again

 kubectl exec -it nginx-deployment /bin/bash

 root@nginx-deployment-587d866f54-ghfll:/# systemctl
 systemctl: error while loading shared libraries: libcap.so.2:cannot 
 open shared object file: No such file or directory

4) Now the systemctl was failing with a new so missing error

root@nginx-deployment-587d866f54-ghfll:/# systemctl
systemctl: error while loading shared libraries: libcap.so.2: cannot 
open shared object file: No such file or directory

5) To solve the above error i again searched the node for libcap.so.2 Found it in the below path.

admin@ip-10-0-20-11:~$ sudo find / -iname 'libcap.so.2'
/lib/x86_64-linux-gnu/libcap.so.2 

6) Seeing the above directory not mounted in my pod. I mounted the below path in the nginx pod.

/lib/x86_64-linux-gnu mounted in the nginx pod(deployment)

7) The nginx pod is not able to come up after adding the above mount. Getting the below error:

$ k logs nginx-deployment-f9c5ff956-b9wn5
standard_init_linux.go:178: exec user process caused "no such file 
or directory"

Please suggest how to debug further. And what all mounts are required to make systemctl work from inside a container in a debian stretch environment.

Any pointers to take the debugging further could be helpful.

Answer 1

Rather than mounting some of the library files from the host you can just install systemd in the container.

$ apt-get -y install systemd

Now, that won't necessarily make systemctl run. You will need systemd to be running in your container which is spawned by /sbin/init on your system. /sbin/init needs to run as root so essentially you would have to run this with the privileged flag in the pod or container security context on Kubernetes. Now, this is insecure and there is a long history about running systemd in a container where the Docker folks were mostly against it (security) and the Red Hat folks said that it was needed.

Nevertheless, the Red Hat folks figured out a way to make it work without the unprivileged flag. You need:

• /run mounted as a tmpfs in the container.
• /sys/fs/cgroup mounted as read-only is ok.
• /sys/fs/cgroup/systemd/ mounted as read/write.
• Use for STOPSIGNAL SIGRTMIN+3

In Kubernetes you need an emptyDir to mount a tmpfs. The others can be mounted as host volumes.