Strategy for managing long running processes in a web app?

Question

I am developing a web asp.net application. A feature of the application is that the user is able to queue up a number of long running operations, then hit run the queue will be processed.

I don't want the user to have to sit and wait for all the operations to  complete or indeed keep the browser / application open. However should they remain or return to the page I would like them to be able to see the status of each job i.e. Waiting, in progress, completed or failed.

I am naturally looking for the most robust, reliable and scalable solution as potentially there maybe a great number of jobs in the queue at any time.

From my research it has been suggested that the asp.net could call into windows service? That perhaps is hosting a WCF web service.

Looking for any advice from anyone who might have had a similar requirement in the past.

Answer 1

Also check out MSMQ,

http://msdn.microsoft.com/en-us/library/ms711472(v=vs.85).aspx

Basically the web application places the "job" or "group of jobs" into a queue.

The service runs constantly, dequeueing the next "job" or "group of jobs" and getting down to business with them.

When completed, the service puts the results into another queue or in a DB (or some other shared resource) that the web application accesses and displays the status of, or the results of.

Enjoy.

Answer 2

A usually good approach is to have a Windows Service that performs scheduled jobs. You can use a library like Quartz.NET which has all kind of scheduling features. If you don't need specific scheduling, you can just have the Windows Service act as the worker process and simply consume the jobs. In addition to that you need to facilitate communication between your web application and the service that is performing the queued up tasks. You can use a number of techniques for this, such as using a table in the database or just using some other kind of inter process communication technique (remoting, named pipes, sockets, message queues etc).

I actually implemented something like this, where there is a Windows Service performing scheduled jobs and there is an ASP.NET web application that is putting up all the requests for the jobs. The ASP.NET application posts some jobs into the database and these are consumed later on by the service process.