How does ASP.NET determine whether to queue a request or not?

Question

When ASP.NET receives a request, how does it determine whether to serve it or to queue it? I ask because I'm monitoring performance counters on a server and the CPU is not maxed out and there are a boatload of available worker threads, but I'm still seeing up to 200 requests get queued up.

Answer 1

I've been doing research and I believe I've come upon an acceptable answer. My primary source is this article: http://blogs.msdn.com/b/tmarq/archive/2007/07/21/asp-net-thread-usage-on-iis-7-0-and-6-0.aspx

As I understand there are two main ways the request processing gets throttled. The first is the MaxConcurrentRequestsPerCPU property. Before .NET 4 this was set to 12 by default. In .NET 4 it was changed to 5000. For async requests they wanted to allow a lot, and for synchronous requests they believe that the ASP.NET ThreadPool will throttle synchronous requests well enough. The second of course is the ThreadPool itself. After ASP.NET posts the request there it can decide when it will go.

If you're doing async processing, your limiting factors are likely to be CPU, network and disk and not any ASP.NET request throttling. It could possibly hit the MaxConcurrentRequestsPerCPU limit, but that limit is really high.

If you're doing synchronous processing and blocking on web calls for long periods of time, it's much more likely you run into these limits. MaxConcurrentRequestsPerCPU is something to watch out for before .NET 4 but there's still the ThreadPool.

Performance testing
I put together a simple test to see how this throttling worked. I have a simple page with a 500ms Thread.Sleep() call. One host machine makes 800 simultaneous async requests and a worker machine running ASP.NET processes them all. The results were interesting:

.NET 3.5, no modifications: 46 seconds. Saw 9 worker threads with process explorer.
.NET 3.5, with MaxConcurrentRequestsPerCPU set to 5000: 46 seconds. 9 worker threads.
.NET 4: 42 seconds, or 13 seconds when running hot. Saw around 35 worker threads gradually get created.
.NET 4, async: 3 seconds

A few observations:

• MaxConcurrentRequestsPerCPU wasn't getting hit. It looks like it was the limitation of the ThreadPool itself.
• .NET 3.5 seems rather reluctant to make new threads to process synchronous requests. .NET 4 does a much better job of ramping up to handle load.
• Async is still the best by a country mile.

Answer 2

IIS does not use all of the available threads before it begins queuing requests, as these threads need to remain available for if an executing request needs additional threads. IIS is optimized to give preference to executing requests, and it does not want an executing request to get blocked because the worker process ran out of available threads.

The default maximum thread pool 20, with a minimum free of 8, which means that the system will only go to 12 requests executing before new requests are queued. Maximum Threads is multiplied by the number of cores, but minimum threads is not, so that default will allow 32 requests on a dual-core box before it queues.

And as for there being CPU remaining, ASP.NET does not monitor that. It is purely about the number of threads in use. Those threads may be blocked by disk access, network access transferring DB results, or simply a Thread.Sleep, which would all still contribute to new requests going to the queue even though CPU is not maxed.

More info is available in the MS Patterns & Practices book on Performance. It applies to IIS6/.NET 1.1, but the concepts still remain. http://msdn.microsoft.com/en-us/library/ff647787.aspx#scalenetchapt06_topic8

Reconfiguring for IIS7 / .NET 2+: http://msdn.microsoft.com/en-us/library/e1f13641.aspx