How to add containers to a Kubernetes pod on runtime

Question

I have a number of Jobs running on k8s.

These jobs run a custom agent that copies some files and sets up the environment for a user (trusted) provided container to run. This agent runs on the side of the user container, captures the logs, waits for the container to exit and process the generated results.

To achieve this, we mount Docker's socket /var/run/docker.sock and run as a privileged container, and from within the agent, we use docker-py to interact with the user container (setup, run, capture logs, terminate).

This works almost fine, but I'd consider it a hack. Since the user container was created by calling docker directly on a node, k8s is not aware of it's existence. This has been causing troubles since our monitoring tools interact with K8s, and don't get visibility to these stand-alone user containers. It also makes pod scheduling harder to manage, since the limits (cpu/memory) for the user container are not accounted as the requests for the pod.

I'm aware of init containers but these don't quite fit this use case, since we want to keep the agent running and monitoring the user container until it completes.

Is it possible for a container running on a pod, to request Kubernetes to add additional containers to the same pod the agent is running? And if so, can the agent also request Kubernetes to remove the user container at will (e.g. certain custom condition was met)?

Answer 1

From this GitHub issue, it seems that the answer is that adding or removing containers to a pod is not possible, since the container list in the pod spec is immutable.

Answer 2

In kubernetes 1.16, there is an alpha feature that would allow for creation of ephemeral containers which could be "added" to running pods. Note, that this requires a feature gate to be enabled on relevant components e.g. kubelet. This may be hard to enable on control plane for cloud provider managed services such as EKS.

API Reference 1.16

Simple tutorial