How do I stop a Supervisord process without killing the program it's controlling?

Question

I am using Supervisord to continually run some indexing programs. Each time an indexer is run, it grabs a certain set of documents, indexes them, then ends. That Supervisord process will then spawn another of the same indexer program, and that indexer will grab a new set of documents to index.

Sometimes I need to stop the Supervisord process that is running these indexer programs. When I do, however, it always kills the indexer program in the middle of work.

What I'd like to do is stop the Supervisord process so that the indexer program that is currently running will execute to completion, but the Supervisord process will not spawn another indexer.

Here is my supervisord.conf settings for this process:

; TRIGGERING INDEXERS
;
[program:indexer]
command=php /data/app/index_company.php
process_name=%(program_name)s_%(process_num)d
redirect_stderr=true
stdout_capture_maxbytes=10MB
stdout_logfile_backups=0
numprocs=5
startsecs=0
autostart=false
autorestart=true
[group:indexers]
programs=indexer

Answer 1

supervisord will emit a SIGTERM signal when a stop is requested. Your child can very probably catch and process this signal (the stopsignal configuration can change the signal sent).

Since you're using PHP, pcntl_signal seems like the way to go. I can't guess how your master actually works I'll assume it's just an infinite loop, single-threaded. Here's a very primitive example:

<?php
    $stop_requested = false;

    pcntl_signal( SIGTERM, function() {
        global $stop_requested;
        $stop_requested = true;
    } );

    while ( true ) {
        do_work();
        if ( $stop_requested ) break;
    }
?>

Now, when you try to stop the process it will not be terminated. However, it will still stop after 10 seconds. Why?

Because stopwaitsecs is set to 10 seconds by default. This is the time supervisord will wait until sending a SIGKILL which will kill your process. Setting it to something you know will work well, like say 600 seconds will allow your indexer to finish work before shutting down.

[program:indexer]
command=php index.php
stopwaitsecs=600

The above should be enough to allow your indexer to gracefully stop. There's at least one issue with long waiting times, though: supervisorctl will keep on waiting and not process further commands until the process stops. You can run another instance. Not sure what happens with the webclient (might keep on loading and expecting a response).

While the above approach will probably work fine for you, you should probably revise your setup and not have supervisord hang around for so long waiting for shutdown to happen. Your indexers should probably stop as soon as they can, by not fetching any more work, flushing the results, or whatever they are doing. Smaller chunks of useful work will make sure they exit within a reasonable amount of time.

Hope this helps. Let me know how it goes.