Guide to using a worker server with a php application [closed]

Question

I've built a PHP app, and I've read that it's a best-practice to use a 'worker' + queue server when calling api's or performing operations that are time consuming.

A quick search for a tutorial has turned up dry. I've built my app using codeigniter, and I do make various calls to the facebook api + use php-based image manipulation throughout my app. The only thing I wonder is how could a queue server+worker help me if I'm performing api calls or resizing my image and the user would normally not care to get a response back from my server until it's completed.

What situations would be good candidates for a worker + queue server, and are there any guides out there for including these in my application? Recently I've included memcache in my app, a that was trivially easy. I simply wrapped my sql queries with a memcache handler.

Answer 1

In the example that you described (image resizing) you basically keep an Apache connection open for the duration of the time it takes to resize your image. Apache processes are expensive and in order to make your system as scalable as possible you should aim to keep your web requests/responses as short as possible. The other idea is that with a queue you can control concurrency. What if 100+ users upload an image to resize at the same time? can your server handle it? If you had a worker (backend) server to handle these requests, then you'd be able to allow the execution of only X concurrent jobs.

Same applies for web services requests: instead of having a connection that stays open, you basically offload the execution of the web service call to a worker process, this frees up an apache process, and you can implement an AJAX polling mechanism that checks if the request that the backend server issued to the web service completed. On the long run the system will scale better, and users usually don't like to wait for an operation to complete with no feedback on where it's at. Queuing allows you to asynchronously execute a task and provide your visitor with feedback on where the completion status of a task.

I typically work with Zend Server's Job queue (http://devzone.zend.com/article/11907 and http://devzone.zend.com/article/11907) that is available with Zend Server full edition (commercial). However, Gearman is also excellent at doing that and has a PHP extension: http://php.net/manual/en/book.gearman.php and an example: http://www.php.net/manual/en/gearmanclient.do.php.

Hope this helps.

--EDIT--

@Casey, I started out adding a comment, but realized this is quickly going to become too long an answer, so I edited the answer instead. I just read the doc for cloud control which is a service I did not know. However luckily I have used Codeigniter quite extensively, so I'll try to hack an answer for you:

1- Cloudcontrol's concept of a worker is to launch a php script from the command line. Therefore you need a way for Codeigniter to accept firing a script from the command line and making it dispatch to a controller. You will probably want to limit that to one controller. See the code at: http://pastebin.com/GZigWbT3 This file does in essence what CI's index.php file does, except it emulates a request through setting $_REQUEST['SERVER_URI']. Be sure to place that file outside of your document root, and adjust the $system_folder variable accordingly.

2- You need a controller script.php in your controllers folder, from which you will disable web requests. You can do something to the effect of:

<?php
class script extends CI_Controller {
    public function __construct() {
        if(php_sapi_name() !== 'cli') {
            show_404();
        }
        parent::__construct();
    }

    public function resizeImage($arg1, $arg2) {
        //Whatever logic to resize image, or library call to do so.
    }
}

3- The last piece is for you to develop a wrapper library in CI (in your system/application/libraries folder) which would effectively wrap the functionality of CloudController's worker invocation

    public function _construct() {
        $ci = get_instance();

        //add check to make sure that the value is set in the configuration
        //Ideally since this is a library, pass the app_name in a setter to avoid creating a dependancy on the config object.
        //Somewhere in one of your config files add $config['app_name'] = 'YOUR_APP_NAME/YOUR_DEP_NAME';
        //where APP_NAME and DEP_NAME are cloud controller's app_name and dep_name
        $this->_app_name = $ci->config->item('app_name');

        //Also add: $config['utilities_script'] = 'path/to/utilities.php';
        //This is the script created in step 1
        $this->_utilities_script = $ci->config->item('utilities_script');
    }

    public function run() {
        $args = func_get_args();
        if(count($args) < 1 ) {
            //We expect at least one arg which would be the command name
            trigger_error('Run expects at least one argument', E_USER_ERROR);
        }

        $method = array_shift($args);

        //utilities.php is the file created in step 1
        $command = "cctrlapp " . $this->_app_name . " worker.add ".$this->_utilities_script;

        //Add arguments if any
        $command .= ' "'.implode(' ', $args).'"';

        //finally...
        exec($command);
    }
}

4- Now from anywhere in your code where you actually want to queue a job, if from a controller:

$this->load->library('Worker');
//resizeImage will call the method resizeImage in the script controller.
$this->worker->run('resizeImage', $width, $height);

Pelase note that:
1- This could be polished further, it was really to give you an idea of how it could get done
2- Since I have no cloudcontroller account, I have no way of testing the code, so it might need tweaking. The utilities.phph script I use in my projects so this one should be good.
Good luck!