Writing Long-Polling WCF Service

Question

I've been writing a long-polling service in WCF. I'm at the point of load testing and I'm running into a problem that, when I get to 5000 outstanding requests, I start getting:

The HTTP service located at (my service) is too busy.

I've implemented this as a WCF AsyncPattern service and the number of worker/IO threads consumed remains very low (like it should). The number of handles looks good, etc. I'm thinking I'm bumping into one of the anti-DOS limits and just can't find it. Here's some of the stuff I've done:

I've modified the registry so that MaxConcurrentRequestsPerCPU isn't an issue. Indeed the number of ASP.NET queued requests stays 0.

I've set up service throttling in the web config, so that shouldn't not an issue. The WCF traces don't show any exceptions due to that.

I'm using a custom binding which does not include any security, etc. All it has is the encoding and the transport (httpTransport).

I've modified the machine.config to up the requestQueueLimit: Here's the processModel element from that:

<processModel enable="true" 
  timeout="Infinite" 
  idleTimeout="Infinite" 
  shutdownTimeout="0:00:05" 
  requestLimit="Infinite" 
  requestQueueLimit="15000" 
  restartQueueLimit="10" 
  memoryLimit="60" 
  webGarden="false" 
  cpuMask="0xffffffff" 
  userName="machine" 
  password="AutoGenerate" 
  logLevel="Errors" 
  clientConnectedCheck="0:00:05" 
  comAuthenticationLevel="Connect" 
  comImpersonationLevel="Impersonate" 
  responseRestartDeadlockInterval="00:09:00" 
  responseDeadlockInterval="00:03:00" 
  maxWorkerThreads="250" 
  maxIoThreads="250" />

But still I end up with this limit (the machine in question can handle it and the rest of the system is async back to a message queuing system).

Can anybody think of anything else?

I'm IIS 7.5 on WindowsServer 2008R2. Asp.NET 3.5SP1.

---

More Information: When I get the above error, the ASP.Net v2.0.50727 "Requests Rejected" performance counter jumps up. The docs indicate that this happens when the request queue is full. The request queue perf counter is at 0 and never budges.

If I crank up the Maximum Worker Processes, I can exceed 5K concurrent requests.

Answer 1

From: http://blogs.msdn.com/b/tmarq/archive/2007/07/21/asp-net-thread-usage-on-iis-7-0-and-6-0.aspx

For v2.0 and v3.5 set a DWORD registry value @ HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\ASP.NET\2.0.50727.0\MaxConcurrentRequestsPerCPU = 5000.  Restart IIS

For v3.5, you can alternatively set <system.web><applicationPool maxConcurrentRequestsPerCPU="5000"/></system.web> in the aspnet.config file.  If the value is set in both places, the aspnet.config setting overrides the registry setting.

For v4.0, the default maxConcurrentRequestsPerCPU is 5000, so you don't need to do anything.

Increase the HTTP.sys queue limit, which has a default of 1000.  If the operating system is x64 and you have 2 GB of RAM or more, setting it to 5000 should be fine.  If it is too low, you may see HTTP.sys reject requests with a 503 status.  Open IIS Manager and the Advanced Settings for your Application Pool, then change the value of "Queue Length". If your ASP.NET application is using web services (WFC or ASMX) or System.Net to communicate with a backend over HTTP you may need to increase connectionManagement/maxconnection.  For ASP.NET applications, this is limited to 12 * #CPUs by the autoConfig feature.  This means that on a quad-proc, you can have at most 12 * 4 = 48 concurrent connections to an IP end point.  Because this is tied to autoConfig, the easiest way to increase maxconnection in an ASP.NET application is to set System.Net.ServicePointManager.DefaultConnectionLimit programatically, from Application_Start, for example.  Set the value to the number of concurrent System.Net connections you expect your application to use.  I've set this to Int32.MaxValue and not had any side effects, so you might try that--this is actually the default used in the native HTTP stack, WinHTTP.  If you're not able to set System.Net.ServicePointManager.DefaultConnectionLimit programmatically, you'll need to disable autoConfig , but that means you also need to set maxWorkerThreads and maxIoThreads.  You won't need to set minFreeThreads or minLocalRequestFreeThreads if you're not using classic/ISAPI mode.

Answer 2

Is IIS limiting the number of open connections? I think there's a setting on a web site or web application in IIS that can do that.

Answer 3

I find all the different configuration options very confusing, but this blog post is very useful:

http://blogs.msdn.com/b/tmarq/archive/2007/07/21/asp-net-thread-usage-on-iis-7-0-and-6-0.aspx

It mentions a limit of 5000 in various guises quite a lot. The tricky thing seems to be that different settings apply depending on the exact setup (eg, .net 2, 3.5, 4.0, IIS 6, 7 etc.)

Hope this helps.