WebJob processing multiple messages at a time from queue

Question

My WebJob is processing multiple messages from a queue which I don't want it to do. My queue has dependent messages which I want to process one after the other sequentially. I have tried with configuring "BatchSize" to 1 but no luck. It is still processing more than one message at a time.

Answer 1

The BatchSize setting (JobHostConfiguration.Queues.BatchSize) doesn't apply to ServiceBus queue processing, only to Azure Queues. To configure ServiceBus queue processing, use ServiceBusConfiguration. The knob you want to configure is ServiceBusConfiguration.MessageOptions.MaxConcurrentCalls. Set this to 1 to disable concurrent processing on a single instance (the default is 16):

JobHostConfiguration config = new JobHostConfiguration();
ServiceBusConfiguration serviceBusConfig = new ServiceBusConfiguration();
serviceBusConfig.MessageOptions.MaxConcurrentCalls = 1;
config.UseServiceBus(serviceBusConfig);

JobHost host = new JobHost(config);
host.RunAndBlock();

That will ensure that only a single message is processed at a time on a single instance. If your WebApp is scaled out, each scaled out instance would then be running in this mode, meaning you will have concurrent processing across instances. If you don't want that either, you can use SingletonAttribute to ensure only one instance of your function is running across instances. See the Singleton wiki page for more information.

Answer 2

If your messages have some order dependency, you are likely to have a pipeline of operations. I have a particular scenario where I use Azure Queues instead of ServiceBus, but I believe the following idea could apply the same:

I have an automation process which does: 1) create a database, 2) create a web app, 3) configure the web app, and 4) deploy code. For my case, things must be done in that order.

For that case, I have created one queue for each task: create-database-queue,create-webapp-queue,configure-web-app, and deploy-app-queue.

You are much safer to work that way because you isolate tasks. Once one process in one queue finishes, then it can be pushed to the next queue. Also, you could easily insert a new step into your process. For example, I can easily insert a new step 1.5) Restore database backup without a second thought.

Furthermore, you promote decoupling of responsibilities with individual queues.