How to queue background tasks in ASP.NET Web API

Question

I have a webapi that is designed to process reports in a queue fashion. The steps the application takes are as follows:

• Receive content
• Map the content to an object and place it into the queue
• Poll for pending items in the queue
• Process items in queue one at a time

I was thinking to use Entity Framework to create a database of queued items, such as:

public class EFBatchItem
{
    [Key]
    public string BatchId { get; set; }
    public DateTime DateCreated { get; set; }
    public DateTime DateCompleted { get; set; }
    public string BatchItem { get; set; }
    public BatchStatus Status { get; set; }
}

My question - Is there a more efficient way, using NServiceBus, BlockingCollection or ConcurrentQeueue, than to constantly poll the database and pull out pending items one by one? I have not used queues before.

One thought is to create a queue of tasks, and on a separate thread process all pending tasks. Somewhat similar to Most efficient way to process a queue with threads but I want to ensure that I am going the most efficient route.

EDIT: A big question I have here is the best way to display the progress to the user. Once the user submits content, he gets taken to a new page and can view the status by the batch identifier. Is MSMQ necessary, or NServiceBus, in order to notify the user? This seems like a variation of the REquest/Acknowledge/Push paradigm?

Answer 1

IMHO, your ASP.NET Web API application shouldn't run those background tasks by itself. It should only be responsible to receive the request, stick it inside the queue (as you indicated) and return the response indicating the success or failure of the received message. You can use variety of messaging systems such as RabbitMQ for this approach.

As for the notification, you have a few options. You can have an endpoint which the client can check whether the processing is completed or not. Alternatively, you could provide a streaming API endpoint which your client can subscribe to. This way, the client doesn't have to poll the server; the server can notify the clients which are connected. ASP.NET Web API has a great way of doing this. The following blog post explains how:

• Native HTML5 push notifications with ASP.NET Web API and Knockout.js

You can also consider SignalR for this type of server to client notifications.

The reason why background tasks are hard for ASP.NET Web API application is that you're responsible to keep the AppDomain alive. This is a hassle especially when you are hosting under IIS. The following blog posts explains really good what I mean:

• Returning Early from ASP.NET Requests
• The Dangers of Implementing Recurring Background Tasks In ASP.NET