How to use message queueing in a real-life scenario?

Question

Ok, so I'm interested in message queueing. I really did loads of research on this subject already. I read 'programming windows azure' (about Azure Queues), I read loads of tutorials and information about the Azure service bus, I watched channel 9 videos on messaging patterns, etc.

But what I just don't understand: how would one use this in a real-life scenario? All the examples just put a single string or an object with some data in a queue and they read it from the queue on 'the other side'. But how do you know what to do with that data? For instance: I could try to save a Customer, an Order and an Address to the database, so I put these 3 objects in a queue to read the on the other side a put them in my database. How do I know what to do with these kind of objects?

Just some questions I have:

1. What kind of data would one put in a queue. A command maybe, that is executed when it's read, so everything in the queue has the same interface and the object itself figures out what to do?
2. Between what layers would one use queues? I was thinking about putting stuff in the queue in the service layer (after it has been validated) and reading it in the data access layer so I can put it in the database.
3. How many queues should one make? Just one queue between the layers you want to connect, or multiple queues between them (maybe for different purposes, although I can't think of any)?
4. This form of loose coupling allows us to queue requests so they can be processed later (for example: when we'd like to restart the database). That's cool but what if I want to READ data instead of writing it? Then the database should be online. And should I read data through a queue or could my service layer just pull the data from the data access layer and pass it to the presentation layer?

I think that were most of the questions that were buzzing around in my head. I hope anyone can clear this up for me.

Answer 1

In the simplest example, imagine you serialize to the queue the action you'd like performed, then the serialized version of the arguments:

Q: Delete: User 5

which is fine since your Worker Role could read that and act accordingly. Doesn't seem like a big bonus in this example, as queuing probably took as long as the deletion would, but what if you wanted to do BI on all your users? Your web page or application would never refresh while waiting for a synchronous request to do the processing. But if you pushed that request to the queue:

Q: RunSome2HourProcess: User *

Now you can return that you've queued the operation, and go on your merry way, while your server that's pulling from the queue would be able to process at its leisure.

What's more is this enables better scale scenarios. What if that last message was split up? So rather than doing all users, you did something like:

QRunSome2HourProcess: User A*-M*

QRunSome2HourProcess: User N*-Z*

Two azure worker roles would split the load and process your information in parallel.

There are also a number of scenarios around retrying, work flows, cross-role asynchronous communications, etc., but this should provide you some reasons why queuing can be good in the right scenarios.

So, in these examples: 1. we've put actions and identifiers/queries in a queued item. The worker role poling from the queue would know how to deal with it.

1. You could use a queue between your service layer and processing layers. you could use a queue between processing layers. you could use a queue within the same layer. The sky's the limit

2. I've typically made 1 queue per operation. So in the above example, i'd have the Run2HourProcess Queue, and just write the parameters to it. And create a new queue for any other process type. But they're super flexible, so it's up to the developer.

3. Unless it's a long running read (ie. the thumbnail generation example in the samples or the result of slow data processing) you'd be just as fast to go the database.