Architecture Queuing asp.NET - MSMQ

Question

Problem: Some 300 candidates make a test using Flex. A test consist of some 100 exercises. After each exercise a .NET service is called to store the result. If a candidate finishes a test, all the data of his/her test is denormalized by Asp.NET. This denormalization can take some cpu and can take 5 to 10 seconds. Now, most of the times, some of the candidates have finished their test earlier than the rest, but still some 200 of them wait until their time is up. At that moment, 200 candidates finish their test and 200 sessions are denormalized at the same time. At this point, server load (cpu) is too high and cause calls to the webserver to go wrong. Now, instead of all these sessions being normalized concurrently, I would like to add them to a queue using MSMQ.

Question:

• How do you process the Queue?
• Do you start a separate thread in the Application_Start of global.asax that listens to the queue? If there are messages, they are dealt one at the time.
• Is it necessary to do this in a separate thread? What if in the global.asax you just call a singleton for instance that starts listening to the queue? In what thread will this singleton run? (what's the thread that calls global.asax)
• What are best practices to implement this? Links? Resources? Tutorials? Examples?
• I don't like the idea, but could you put an exe on the root of your website, an exe that starts a process listening to the queue...
• If you get a message out of the queue, do you remove it when you pull it out or do you remove it if denormalization for this session was successful? If you remove it when you pull it out and something goes wrong...
• I could also create my own queue in memory, but restarting the webserver would empty the queue and a lot of sessions would end up not being normalized, so I guess this is really a bad idea.
  • Is MSMQ a good choice or are there better alternatives?

Answer 1

You could consider using a WCF-Service with MSMQ transport. I used this approach in an application that calculates commissions:

User completes asp.net wizard configuring calculation parameters Calculation Job is sent to WCF-Service using MSMQ transport Service transaction is completed as soon as Job entered MSMQ New transaction scope is created for processing Job instances

One drawback is that the transaction will require MSDTC which will add some overhead when targeting MS SQL Server and even more when dealing with Oracle.

IDesign provides a lot of useful samples and best practices on WCF queueing.

Answer 2

Personally, I use a servicebus for scenario's like that. I know this sounds like an overkill, but I think the .net servicebusses are so good that they require the least amount of code written by you, because it's not easy to create a good scheduler for background processes without disturbing the threads of the application pool the webapp is running in. NServicebus and MassTransit are both good an well enough documented servicebuses for your scenario. With a servicebus, you have a framework that writes to msmq and listens to msmq in several apps connected by the messagequeue. The bus makes it easy for you to create a separate app that runs as a background service and is connected with your web-app by the message queue. When you use topself (included in nservicebus and masstransit), an installer/uninstaller for the seperate apps is automatically generated by the service bus.

Question: Why don't you like the idea of having a separate exe?

Answer 3

• How do you process the Queue?
• Do you start a separate thread in the Application_Start of global.asax that listens to the queue? If there are messages, they are dealt one at the time.
• Is it necessary to do this in a separate thread? What if in the global.asax you just call a singleton for instance that starts listening to the queue? In what thread will this singleton run? (what's the thread that calls global.asax)

[skip]

• I don't like the idea, but could you put an exe on the root of your website, an exe that > starts a process listening to the queue...

Normally another program processes the queue - not ASP.NET. Either a windows service or an executable that you run under a scheduler (and there's no reason to put it in the root of your website).

• If you get a message out of the queue, do you remove it when you pull it out or do you remove it if denormalization for this session was successful? If you remove it when you pull it out and something goes wrong...

For critical work, you perform a transactional read. Items aren't removed from the queue until you commit your read operation, but while the transaction is open, no other process can get the item.

• What are best practices to implement this? Links? Resources? Tutorials? Examples?

This tutorial is a good introduction and John Breakwell's blog is excellent and offers a lot of good links (including the ones in his easy-to-find sidebar "MSMQ Documentation").