How to speed up Azure Storage Queue

Question

I have tried everything I can think of to increase the speed of inserts. Which is really only a couple of things with no improvement.

I have chunks of identifiers (Int64) that I need to send to a queue so that my multiple worker roles can work on it without having to worry about concurrency.

I have tried a foreach loop (both with .ToString() and BitConverter.GetBytes()):

foreach(long id in ids) {
    queue.AddMessage(new CloudQueueMessage(id.ToString() /*BitConverter.GetBytes(id)*/));
}

And a Parallel .ForAll<T>():

ids.AsParallel().ForAll(id => queue.AddMessage(new CloudMessage(id.ToString())));

Both from local and a WorkerRole inside the same data center, the inserts max out at 5 per second, and average 4.73 per second.

Am I doing something wrong?

Answer 1

Try disabling Nagle on the tcp stack, as this buffers small packets, resulting in upwards of 1/2-second delay shipping your content. Put this in your role start code:

ServicePointManager.UseNagleAlgorithm = false; 

Answer 2

I used a different approach to pump 1500 messages per second from my laptop in a corporate network and hit the 2000 message limit on a VM co-located with the storage account.

I used an async parallel partitioner in combination with some tuning of the default connection limit and partition count.

ServicePointManager.DefaultConnectionLimit = 1000;

public static async Task SendMessagesAsync(CloudQueue queue, IEnumerable<string> messages)
{
    await Task.WhenAll(
            from partition in Partitioner.Create(messages).GetPartitions(500)
            select Task.Run(async delegate
            {
                using (partition)
                    while (partition.MoveNext())
                        await queue.AddMessageAsync(new CloudQueueMessage(partition.Current));
            }));
}

Nagle on or off had no impact on performance.