Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

System.Net.WebClient unreasonably slow

When using the System.Net.WebClient.DownloadData() method I'm getting an unreasonably slow response time.

When fetching an url using the WebClient class in .NET it takes around 10 sec before I get a response, while the same page is fetched by my browser in under 1 sec. And this is with data that's 0.5kB or smaller in size.

The request involves POST/GET parameters and a user agent header if perhaps that could cause problems.

I haven't (yet) tried if other ways to download data in .NET gives me the same problems, but I'm suspecting I might get similar results. (I've always had a feeling web requests in .NET are unusually slow...)

What could be the cause of this?

Edit:
I tried doing the exact thing using System.Net.HttpWebRequest instead, using the following method, and all requests finish in under 1 sec.

public static string DownloadText(string url)         var request = (HttpWebRequest)WebRequest.Create(url);         var response = (HttpWebResponse)request.GetResponse();          using (var reader = new StreamReader(response.GetResponseStream()))         {             return reader.ReadToEnd();         } } 


While this (old) method using System.Net.WebClient takes 15-30s for each request to finish:

public static string DownloadText(string url) {        var client = new WebClient();        byte[] data = client.DownloadData(url);        return client.Encoding.GetString(data); } 
like image 883
moevi Avatar asked Dec 11 '10 05:12

moevi


People also ask

Why is my httpclient so slow?

In .NET Core, the handler owned by HttpClient owns the connection pool, and if you do new HttpClient (), it's creating a new handler that it owns. It's possible the slow down you're seeing is because, due to the way HttpClient is being used, it's creating a new connection on every access in .NET Core and it's not in .NET Framework.

How does the webclient class work?

The WebClient class uses the WebRequest class to provide access to resources. WebClient instances can access data with any WebRequest descendant registered with the WebRequest.RegisterPrefix method.

Is there a performance difference between httpclient and webclient?

Sign in to your account I have conducted a basic performance test of HttpClient in .NET Core 2.0 versus HttpClient in .NET Framework 4.7 and have noticed a gap in performance. The .NET Framework 4.7 HttpClient/WebClient outperforms .NET Core's HttpClient by ~1.5-2x in terms of how long it takes to complete a batch of n requests.

How to fix when webclient not working for the first time?

but for first time it not work and second time it works. Or for the first time you can override the timeout function and set timeout for webclient, on timeout call your function again to download data this will solve your issue where it is not working for the first time.


2 Answers

I had that problem with WebRequest. Try setting Proxy = null;

    WebClient wc = new WebClient();     wc.Proxy = null; 

By default WebClient, WebRequest try to determine what proxy to use from IE settings, sometimes it results in like 5 sec delay before the actual request is sent.

This applies to all classes that use WebRequest, including WCF services with HTTP binding. In general you can use this static code at application startup:

WebRequest.DefaultWebProxy = null; 
like image 78
Alex Burtsev Avatar answered Oct 09 '22 22:10

Alex Burtsev


Download Wireshark here http://www.wireshark.org/

Capture the network packets and filter the "http" packets. It should give you the answer right away.

like image 21
Harvey Kwok Avatar answered Oct 10 '22 00:10

Harvey Kwok