Help me understand how message queuing works

Question

Been using PHP for quite some time now and I was wondering what this whole "message queue" is all about. Let's take facebook for example. I can update my status but then I have to show that status updates to all my friends (let's say I have 3000 followers). Even more work if there are comments and they have to be notified to all friends who left a comment via email. With the example I've seen, it appears that all a message queue does is take the "message" (my status update) and puts it into some temporary space (filesystem or DB table). I then have a cron job that pulls it out and updates my table.

With that said, how do I go about manipulating that data? I guess I'm getting confused as to how this would really help me. How do I translate the following function into a message queue and then schedule for function to run at a later time?

1 - Update my status 2 - Now publish it across my page and all my friends. 3 - If comment is left, now email that latest comment to those who "subscribe" to that comment.

My question is, how do I manipulate that data? Do I just insert the "comment" then have a "job" that pulls that comment out and plug it into a function that processes it?

Here's an example I plan on looking into.

http://www.freeopenbook.com/php-hacks/phphks-CHP-5-SECT-18.html

Thanks in advance.

Answer 1

My question is, how do I manipulate that data? Do I just insert the "comment" then have a "job" that pulls that comment out and plug it into a function that processes it?

Exactly.

Publishing status updates across Facebook pages probably doesn't involve message queueing - I don't actually know their specific design, but I'd guess that updated data is just supplied on-demand via a query when users load their pages. (Unless Facebook has a separate process to denormalize status update data.)¹

By contrast, sending status update email notifications is a great candidate for message queuing.

A typical implementation would involve writing a new message (generally minimal, perhaps just your user id) to a specific message queue - perhaps the "EmailStatusUpdateNotifications" queue.

Another process then dequeues messages and knows exactly what to do with them. A process dedicated to sending status update email messages would use the user id (the contents of the message) to load your current status and a list of your friends' email addresses, build the email messages, and dispatch them.

¹It turns out you can find a lot of good information about Facebook's architecture in Why are Facebook, Digg, and Twitter so hard to scale? at High Scalability.