HttpWebRequest is extremely slow!

Question

I am using an open source library to connect to my webserver. I was concerned that the webserver was going extremely slow and then I tried doing a simple test in Ruby and I got these results

Ruby program: 2.11seconds for 10 HTTP GETs

Ruby program: 18.13seconds for 100 HTTP GETs

C# library: 20.81seconds for 10 HTTP GETs

C# library: 36847.46seconds for 100 HTTP GETs

I have profiled and found the problem to be this function:

private HttpWebResponse GetRawResponse(HttpWebRequest request) {
  HttpWebResponse raw = null;
  try {
    raw = (HttpWebResponse)request.GetResponse(); //This line!
  }
  catch (WebException ex) {
    if (ex.Response is HttpWebResponse) {
      raw = ex.Response as HttpWebResponse;
    }
  }
  return raw;
}

The marked line is takes over 1 second to complete by itself while the ruby program making 1 request takes .3 seconds. I am also doing all of these tests on 127.0.0.1, so network bandwidth is not an issue.

What could be causing this huge slow down?

UPDATE

Check out the changed benchmark results. I actually tested with 10 GETs and not 100, I updated the results.

Answer 1

What I have found to be the main culprit with slow web requests is the proxy property. If you set this property to null before you call the GetResponse method the query will skip the proxy autodetect step:

request.Proxy = null;
using (var response = (HttpWebResponse)request.GetResponse())
{
}

The proxy autodetect was taking up to 7 seconds to query before returning the response. It is a little annoying that this property is set on by default for the HttpWebRequest object.

Answer 2

It may have to do with the fact that you are opening several connections at once. By default the Maximum amount of open HTTP connections is set to two. Try adding this to your .config file and see if it helps:

<system.net>
  .......
  <connectionManagement>
    <add address="*" maxconnection="20"/>
  </connectionManagement>
</system.net>

Answer 3

I was having a similar issue with a VB.Net MVC project.
Locally on my pc (Windows 7) it was taking under 1 second to hit the page requests, but on the server (Windows Server 2008 R2) it was taking 20+ seconds for each page request.

I tried a combination of setting the proxy to null

  System.Net.WebRequest.DefaultWebProxy = Nothing
  request.Proxy = System.Net.WebRequest.DefaultWebProxy

And changing the config file by adding

 <system.net>
   .......
   <connectionManagement>
     <add address="*" maxconnection="20"/>
   </connectionManagement>
 </system.net>

This still did not reduce the slow page request times on the server. In the end the solution was to uncheck the “Automatically detect settings” option in the IE options on the server itself. (Under Tools -> Internet Options select the Connections tab. Press the LAN Settings button)

Immediately after I unchecked this browser option on the server all the page request times dropped from 20+ seconds to under 1 second.