Spawning multiple processes with PHP to process data.

Question

I have a queue (Amazon SQS) of data that needs to be processed, and I would like to do it with multiple processes (in PHP).

I want the child workers to do something like this (pseduoish code):

while(true) {

    $array = $queue->fetchNItems(10); // get 10 items

    if(!count($array)) 
        killProcess();

    foreach($array as $item) {
         ... // process the item
         $queue->remove($item);
    }

    sleep(2);
}

I always need 1 child process to be running, but in times of need I want to (fork?) a child process so that it can help process the queue faster.

Can someone help me with a rough PHP skeleton of what I need, or point me in the right direction?

I think I need to take a look at http://php.net/manual/en/function.pcntl-fork.php, but I'm not sure how I can use this to manage multiple processes.

Answer 1

When you fork a process. you make a duplicate of that process. In other words the copy (fork) contains everything the original process had (including file handles)

So how do you know if you are the parent or the forked process?

The example from the linked page shows this pretty clearly

<?php

$pid = pcntl_fork();
if ($pid == -1) {
     die('could not fork');
} else if ($pid) {
     // we are the parent
     pcntl_wait($status); //Protect against Zombie children
} else {
     // we are the child
}

?>

To extend this to what you want

<?php

$pid = pcntl_fork();
if ($pid == -1) {
     die('could not fork');
} else if ($pid) {
     // we are the parent
     pcntl_wait($status); //Protect against Zombie children
} else {
     // we are the child
     while(true) {

         $array = $queue->fetchNItems(10); // get 10 items

         if(!count($array)) {
            exit();
         }

         foreach($array as $item) {
              ... // process the item
              $queue->remove($item);
         }

         sleep(2);
     }
}

?>

This will create on forked process ( a waste in this instance ) use a loop to create multiple processes. when the child has finished exit will kill the child process. and pcntl_wait() will return allowing the parent to continue. I am not sure about php but if the parent process dies or exits it will kill the child process even if the child is not finished. hence the pcntl_wait. a more elaborate system is required if you spawn multiple children.

perhaps rather than forking you should look at the range of exec functions?

A caveat.

forking process can be wrought with problems, database handles being closed when a child exits etc. You can also kill a server with to many processes if something goes wrong. spend a lot of time playing and testing and reading.

DC