how to batch process all Azure Storage Queue Messages into a Blob with Azure Cloud Functions?

Question

I want to read all messages of an Azure Queue Storage and write them into a Blob. Ideally I would like to read batches of 10000 or more and write them into a Blob.

I am using Azure Cloud Functions with Queue Storage Binding for input and Blob Storage Binding for output, but I can't seen to find an API or a configuration option that would enable me to read more than 1 message. Does anyone know about such an API?

Answer 1

The official documentation doesn't mention any support for processing Storage Queue messages in batches in a single execution of Azure Function. There is an open issue in WebJobs SDK. So, it's not supported.

If you are flexible which service to use for messaging middleware, you could switch to Event Hubs. Event Hub trigger supports (and encourages) processing messages in batches. It won't probably be 10.000 though: the batch size is limited to 256k of data.

To process Storage Queue messages in batches you'd have to get away from Queue Triggered Functions (e.g. run a function on a timer and connect to the table storage to process all the messages, or have a custom polling Web Job, or use Web Job SDK with a custom trigger).