Adding multiple elements to Azure Queue in a single message

Question

I'm interacting with an AzureStorageAccount with CloudQueueClient similarly to the manner described in this msdn example

CloudStorageAccount storageAccount = CloudStorageAccount.Parse("some connection string");
CloudQueueClient queueClient = storageAccount.CreateCloudQueueClient();

and to add an element to a queue (with some boiler plate removed):

var queue = queueClient.GetQueueReference("queuename");
var message = new CloudQueueMessage("myString");
queue.AddMessageAsync(message);

So that means I can add "myString" to my queue. Great. And if I repeatedly call those lines of code I can add "mystring" lots of time. Also good, but inefficient.

How do I add multiple items to the queue in one message?

I've researched this a bit and found Entity Group Transactions, which may be a suitable fit. However, this looks very different to what I've been doing and don't really give me any code examples. Is there any way to use this and continue to use Microsoft.WindowsAzure.StorageClient libary to construct my messages?

Answer 1

One way you can send multiple messages is to build a wrapper class that contains a list of individual string values (or objects), serialize this into a JSon object for example, and send that as the payload for the message. The issue here is that depending on the size of your objects, you could eventually exceed the size limitation of a message. So that's not a recommended implemtation.

At some point I was dealing with a system that needed to send a lot of messages per second; it was massively distributed. Instead of batching multiple messages we ended up creating a shard of message queues, across multiple storage accounts. The scalability requirements drove us to this implementation.

Keep in mind that Chunky vs Chatty applies to sending more information in order to avoid roundtrips, so as to optimize performance. Message queuing is not as much about performance; it is more about scalability and distribution of information. In other words, eventual consistency and distributed scale out are the patterns to favor in this environment. I am not saying you should ignore chunky vs chatty, but you should apply it where it makes sense.

If you need to send 1 million messages per second for example, then chunkier calls is an option; sharding is another. I typically favor sharding because there are fewer scalability boundaries. But in some cases, if the problem is simple enough, chunking might suffice.