Nginx / PHP FPM graceful stop (SIGQUIT): not so graceful

Question

Running nginx 1.9.* / PHP 7.0.* (but exact same behavior in 5.6.* also)

Attempting to gracefully stop a PHP-FPM / nginx combo for node shutdown during maintenance. To do this, I'm sending the SIGQUIT to php-fpm, which should provide a graceful shutdown.

To test this, I made a dumb script

<?php sleep(5); echo 'done';

Tested locally with the following curl

curl -I x.x.x.x:8080

Which normally produces the output:

HTTP/1.1 200 OK
Server: nginx
Date: Tue, 12 Apr 2016 04:48:00 GMT
Content-Type: text/html; charset=UTF-8
Connection: close

Desired: in the middle of any in-flight request, when a graceful shutdown is requested, the current requests should finish, but any additional requests should fail.

Unfortunately, when I try to trigger this behavior, by sending a SIGQUIT (http://manpages.ubuntu.com/manpages/precise/man8/php5-fpm.8.html) to the PHP-FPM master process:

kill -s SIGQUIT $FPMPID

The connection immediately drops, resulting in an ngnix 502

HTTP/1.1 502 Bad Gateway
Server: nginx
Date: Tue, 12 Apr 2016 04:48:07 GMT
Content-Type: text/html
Content-Length: 166
Connection: close

Any advice? I would love to make this piece of the system as seamless as possible. Thanks!

Answer 1

After struggling with this same situation for a while, I believe I've found the magical config setting to make child processes finish handling requests before dying.

http://php.net/manual/en/install.fpm.configuration.php#process-control-timeout

process_control_timeout

Time limit for child processes to wait for a reaction on signals from master

Basically, by setting this to something like 10s, the child process will wait that long, while handling existing requests before quitting.

Unfortunately, it seems that the php-fpm master process exits immediately, so, inspired by the code here, I wrote a wrapper script:

#!/bin/bash

PHP_FPM_PID='/php-fpm.pid'

wait_for_pid () {
    try=0

    while test $try -lt 35 ; do
        if [ ! -f "$1" ] ; then
            try=''
            break
        fi

        echo -n .
        try=`expr $try + 1`
        sleep 1
    done
}

function clean_up {

    echo "Killing $(cat $PHP_FPM_PID)"

    kill -QUIT `cat $PHP_FPM_PID`
    wait_for_pid $PHP_FPM_PID

    echo "Done!"

    exit 0
}

trap clean_up EXIT

nohup php-fpm --daemonize --pid $PHP_FPM_PID 2>&1 &

while true; do sleep 1; done
# ^ do nothing forever

which waits 35 seconds or until that pid file has been removed (presumably by one of the child processes? I'm still unclear on how it's removed).

Regardless, this wrapper script works well as the CMD for our php-fpm docker container that we're running with Kubernetes.