Parallelizing PHP processes with a Bash Script?

Question

I want to launch ~10 php processes from a bash script. When one of them finishes, I'd like the bash script to launch another php process, and continue on indefinitely, always having ~10 php processes running.

What is the simplest way to do this?

The php file launched will be the same every time, but the php process will know to pull new values from the database so it's processing new data each time. The file I need to launch and all it's classes are already written in php.

Answer 1

You could use GNU Parallel, piping the list of images to manage into parallel as described here.

Answer 2

You could use something like this. Use one file to launch 10 (only run this once) and the bottom of each file could relaunch itself when it finished.

/** 
 * Asynchronously execute/include a PHP file. Does not record the output of the file anywhere.  
 *
 * @param string $filename      file to execute, relative to calling script (or root?)
 * @param string $options       (optional) arguments to pass to file via the command line
 */ 
function asyncInclude($filename, $options = '') {
    exec("/path/to/php -f {$filename} {$options} >> /dev/null &");
}

Answer 3

Seems like a good fit for superivisord. The following configuration will make sure that 10 processes are always running, and deals with log rotation, which is also handy. All output, including stderr, will be written to /var/log/worker.log. With "autorestart=true", supervisord will replace a child process as soon as it exits.

[program:worker]
command=php /path/to/worker.php
process_name=%(program_name)s_%(process_num)d
stdout_logfile=/var/log/%(program_name)s.log
redirect_stderr=true
stdout_capture_maxbytes=512MB
stdout_logfile_backups=3
numprocs=10
numprocs_start=0
autostart=true
autorestart=true

Once you have the supervisor config in place (usually /etc/supervisord/conf.d), you can use supervisorctl as a convenient way to start and stop the process group.

$ supervisorctl start worker
...
$ supervisorctl stop worker
...
$ supervisorctl status
worker:worker_0              RUNNING    pid 8985, uptime 0:09:24
worker:worker_1              RUNNING    pid 10157, uptime 0:08:52
...
worker:worker_9              RUNNING    pid 12459, uptime 0:08:31