After doing HttpWebRequests for a while the result starts timing out

Question

I have an application that spiders websites for information. It seems like after 20-45 minutes of creating HttpWebRequests a bunch of them return timeouts. One thing we do is attach a BindIPDelegate anonymous function to give the request a specific IP since we round-robin through about 150 IPs.

I'm setting up the HttpWebRequest object with the following settings..

• Setting User-Agent
• Setting Keep-Alive to false so that the IP isn't re-used
• Setting TimeOut to 60000 (60 seconds)
• Setting ReadWriteTimeout to 60000 (60 seconds)
• Setting Proxy to null
• Setting Accept to /
• Setting CookieContainer to new CookieContainer
• Setting Piplined to true
• Setting Automatic Decompression to Deflate & GZIP

The application is using .NET 4.0 and running on Windows Server 2008 R2.

This definitely seems like something application/TCP/.NET related because if I restart the application it runs fine again. Also it appears more or less like the ones timing out are just queued up waiting on a local port or something.

Any ideas?

Answer 1

You don't say much about the code you actually use to perform the requests but, anyway, here are my guesses:

1. You are using BeginGetResponse()/EndGetResponse() with a callback and the callback takes too long to complete (or blocks!). This could cause a deadlock in the threadpool if you are issuing a lot of requests in a short period of time.

2. Since you are not reusing the connections and, again, if the requests happen very fast and non-stop, you might run out of sockets (last time I tried, ~3k per interface on windows). If setting KeepAlive to true fixes your problem, this is it.

3. You are not calling Dispose()/Close() on the HttpWebRequest or the HttpWebResponse or the Stream you get from the response. This might work for a little bit until you hit the limit of 2 (from the MSDN docs) or 6 (configuration file default) in your application configuration settings for (system.net/connectionManagement/add[address="*",maxconnection="6"]). A simple way to test if this is the problem is to set the limit to 1 and see if the problem happens earlier than before.

Btw, setting KeepAlive to false and Pipelined to true does not make sense.

Answer 2

I would guess it is due to ThreadPool related issues.