Docker - init, zombies - why does it matter?

Question

I did read this article: https://blog.phusion.nl/2015/01/20/docker-and-the-pid-1-zombie-reaping-problem/

To set some context: Article is about problem with zombies in containers, it try to convince us that it is a real problem.

Generally, I have mixed feelings. Why does it matter ? After all, even in case zombies in conainer host OS is able to release/kill this zombie. We know that process in container is from point of view host OS normal process (and in general process in container is normal process with some namespaces and cgroups).

Moreover, we can also find information that in order to avoid zombie problem we should use bash -c .... Why ? Maybe, better option is to use --init ?

Can someone try to explain these thing, please ?

Answer 1

For a brief but useful explanation of what an init process gives you, look at tini which is what Docker uses when you specify --init

Using Tini has several benefits:

• It protects you from software that accidentally creates zombie processes, which can (over time!) starve your entire system for PIDs (and make it unusable).
• It ensures that the default signal handlers work for the software you run in your Docker image. For example, with Tini, SIGTERM properly terminates your process even if you didn't explicitly install a signal handler for it.

Both these issues affect containers. A process in a container is still a process on the host, so it takes up a PID on the host. Whatever you run in a container is PID 1 which means it has to install a signal handler to get that signal.

Bash happens to have a process reaper included, so running a command under bash -c can protect against zombies. Bash won't handle signals by default as PID 1 unless you trap them.

Zombies

The first thing to understand is an init process doesn't magically remove zombies. A (normal) init is designed to reap zombies when the parent process that failed to wait on them exits and the zombies hang around. The init process then becomes the zombies parent and they can be cleaned up.

Next, a container is a cgroup of processes running in their own PID namespace. This cgroup is cleaned up when the container is stopped. Any zombies that are in a container are removed on stop. They don't reach the hosts init.

Third is the different ways containers are used. Most run one main process and nothing else. If there is another process spawned it is usually a child of that main process. So until the parent exits, the zombie will exist. Then see point 2 (the zombies will be cleared on container exit).

Running a Node.js, Go or Java app server in a container tends not to rely heavily on forking or spawning of processes.

Running something like a Jenkins worker that spawns large numbers of ad hoc jobs involving shells can result in a lot worse, but is ephemeral so exits regularly and cleans up

Running a Jenkins master that also spawns jobs. The container may hang around for a long time and leave a number of zombie processes which is the type of workload that could present a problem without a zombie reaper.

Signals

The other role an init process can provide is to install signal handlers so signals sent from the host can be passed onto the container process. PID 1 is a bit special as it requires the process to listen for a signal for it to be received.

If you can install a SIGINT and SIGTERM signal handler in your PID 1 process then an init process doesn't add much here.

When to use an init

When you want to run more than 1 service in a container

Multiple processes should be run under an init process. When Docker starts, the init manages how should they be launched. What is required for the container to actually be "running" for the service it represents. When the container stops, how that should be passed onto each process. You may want a more traditional init system though, s6 via s6-overlay provides a number of useful container features for multi process management.

When you run a single process that spawns a lot of child processes

Especially when processes are children of children or beyond. The CI worker (like Jenkins) example is the first that comes to mind where Java spawns command or shells that spawn other commands.

When you can't add signal handlers to the process running as PID 1.

sleep is a simple example of this. A docker run busybox sleep 60 can't be interrupted with ctrl-c or stopped, it will be killed after the default 10 second docker stop timeout. docker run --init busybox sleep 60 works as expected.

Whenever

tini is pretty minimal overhead and widely used, so why not use --init most of the time?

For more details see this github comment which answers the "why?" question from the creator of tini.

Answer 2

I referenced that article in "Use of Supervisor in docker"

Since Sept. 2016 and docker 1.12, docker run --init is helping fighting zombie processes by adding an init process.

That solves typically the following issue

We can't use docker start as we need to pass things like port mappings, and env vars. So we use docker run.
But when upstart sends SIGINT to the docker run client process, the container doesn't die, just the client does. Then when upstart goes to start it back up, it's already running, and the port mapping fails.

Or this issue:

Docker seems to hang when spawning child processes inside executed scripts.

Basically, you want a docker container to kill all sub-processes, in order to clean resources (port, files handlers, ...) used by said sub-processes.