PHP asynchronous processing with beanstalkd. Do you recommend it?

Question

would you recommend using beanstalkd [ http://kr.github.com/beanstalkd/ ] for

asynchronous processing in PHP ?

I need some feedback/comments pros-cons, from anyone that has used this lib.

Thanks,

Answer 1

I've used Beanstalk in production, and also while testing threw millions of simple messages through it - generally en-mass, but the production system had over a 100,000 tasks put through it till I left the company. It may still be running, in which case it would be tens of millions be now - or more, if they had extended it's use further, as I had planned.

I would recommend it, as it's got a number of excellent points.

• named tubes can be used to limit the jobs being delivered. I watch a tube based on the machine hostname, which limited where a worker would be running - useful for uploaded files which are only stored on a particular server).
• The delays can be used to set future events
• Latest versions of the server also support bin-logging, giving persistence, though it has never crashed on me.

My first task I put through it was image processing - and doing that work outside of an Apache/mod_php process allowed me to resize larger images without affecting the server (blowing out the Webserver). With a lightly loaded queue, it had created the thumbnails before the page had refreshed after the upload.

There are many other potential tasks that could also be processed asynchronously.

The only problems I have ever had was making sure that the workers completed without incident - or that any errors were caught so that the job could be 'bury'ed, thus ensuring that the job was not put back into the queue to be run again (and have the worker crash again).

Having the workers also restart to clear memory can also be useful as PHP is less well suited to long-running processes.