Docker SIGTERM not being delivered to node.js/coffee app when started with flags

Question

I've set up listeners in my application to catch SIGTERM, SIGINT and SIGUSR2:

# kill
process.on 'SIGTERM', ->
    killExecutors 'SIGTERM'

# ctrl + c
process.on 'SIGINT', ->
    killExecutors 'SIGINT'

# nodemon signal
process.on 'SIGUSR2', ->
    killExecutors 'SIGUSR2'

It works as expected. When I run it inside a docker instance:

FROM node:4.4.7

MAINTAINER Newborns <[email protected]>

COPY . /src

EXPOSE 7733

WORKDIR /src
RUN npm install

CMD ["./node_modules/.bin/coffee", "feeder.coffee"]

Everything works fine, too. BUT, when I add a node flag to the execution

FROM node:4.4.7

MAINTAINER Newborns <[email protected]>

COPY . /src

EXPOSE 7733

WORKDIR /src
RUN npm install

CMD ["./node_modules/.bin/coffee", "--nodejs", "--max_old_space_size=384", "feeder.coffee"]

it stops catching the signals. I've tried to change de CMD exec form to

CMD ./node_modules/.bin/coffee --nodejs --max_old_space_size=384 feeder.coffee

but still doesn't work. What changes between the execution with and without flags?

EDIT:

Actually, what happens is that docker starts one process when no flags are passed

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  4.2  1.0 960940 85424 ?        Ssl  20:21   0:01 node ./node_modules/.bin/coffee feeder.coffee
root        16  0.1  0.0  20220  2884 ?        Ss   20:22   0:00 bash
root        20  0.0  0.0  17500  2064 ?        R+   20:22   0:00 ps -aux

and two processes when flags are passed

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.1  0.3 707704 25272 ?        Ssl  20:17   0:00 node ./node_modules/.bin/coffee --nodejs --max_old_space_size=384 feeder.coffee
root        10  1.7  1.1 965900 90068 ?        Sl   20:17   0:01 /usr/local/bin/node --max_old_space_size=384 /src/node_modules/.bin/coffee feeder.coffee

question is: why?

Answer 1

TL;DR Use a Javascript loader file instead of the coffee executable when you need to use extended node options to avoid the technicalities of signals with forked processes under Docker.

require('coffee-script').register();
require('./whatever.coffee').run();

Then

node --max_old_space_size=384 app.js

Now, onto the technicalities...

Docker and signals

The initial process in a container is PID 1 in the containers namespace. PID 1 (or the init process) is treated as a special case by the kernel with regards to signal handling.

1. If the init process does not install a signal handler, that signal won't be sent to it.
2. Signals do not propagate automatically from an init process, the process must manage this.

So a docker process is expected to handle signals itself.

Coffeescripts --nodejs option

As you have noted, coffee will fork a child node process when it has the --nodejs option to be able to pass the extra options on.

This initially presents some odd behaviour outside of docker with signal handling (on osx at least). A SIGINT or SIGTERM will be forwarded onto the child but also kill the parent coffee process immediately, no matter how you handle the signal in your code (which is running in the child).

A quick example

process.on 'SIGTERM', -> console.log 'SIGTERM'
process.on 'SIGINT', -> console.log 'SIGINT'

cb = -> console.log "test"
setTimeout cb, 5000

When you run this and ctrl-c, the signal is forwarded on to the child process and handled. The parent process closes immediately though and returns to the shell.

$ coffee --nodejs --max_old_space_size=384 block_signal_coffee.coffee 
^C
SIGINT
$ <5ish second pause> test

Then the child process with your code continues running in the background for 5 seconds and eventually outputs test.

Docker and coffee --nodejs

The main problem is the parent coffee process does not handle any signals in code, so the signals don't arrive and are not forwarded onto the child. This probably requires a change to coffeescript's launcher code to fix.

The signal quirk coffee --nodejs presents outside of Docker could also be bad if it happened under Docker. If the main container process (the parent of the fork) exits before your signal handlers have a chance to complete in the child, the container will close around them. This scenario is unlikely to happen if the above problem is fixed by just forwarding signals onto the child.

An alternative to using the suggested javascript loader or fixing coffee scripts loader, would be to use an actual init process, like runit or supervisor but that adds another layer of complexity in between docker and your service.