Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

HttpWebRequests using WebProxy work and then fail after time

Odd situation I have here and unfortunately I don't understand a lot about the Windows network side of things outside of netstat:

So I have a proxy that I have configured in my browser (Firefox 42) and am running a simple application that loops through URLs to call them via that proxy. This proxy has credentials in order to use it and I know the proxy works. This is a Windows 7 box.

So at some point during this process, the following happens:

  1. Browser calls just time out. It doesn't ask for credentials at all. (when the issue goes away, it starts to ask for credentials again).
  2. Calls in the application timeout no matter what the timeout is (7 seconds, 20 seconds, etc)

I've confirmed the following:

  1. In my .net application, I 100% know I am closing every network object and am even aborted the request object after I read the response.
  2. After a certain amount of time, without any calls, the problem goes away.
  3. When I use this proxy on another server, it 100% works. So I know it's related to the server I am using and that proxy IP address.
  4. I've looked at the resource manager and there aren't a lot of active TCP connections open. Although I don't know if that means anything.
  5. If I use another proxy, THAT proxy works. It's like it's IP specific, which is baffling me because it's just a web proxy object in the code.

What would cause this? It usually happens after 4-7 calls with the proxy and releases the issue after 30-40 minutes.

Edit 7:

Also happens with AWS instances. Tried that approach. zzz...

Edit 6:

Doesn't go away with a server restart either. You can restart and 15 minutes later SAME proxy times out. Eventually works again.

Edit 5:

Wrote a similar test with Java and Python. Same result.

Edit 4: This is how it works:

Call to Proxy 1... Good!
Call to Proxy 2... Good!
Call to Proxy 3... Good!
Call to Proxy 4... Good!
Call to Proxy 1... Good!
Call to Proxy 2... Good!
Call to Proxy 3... Good!
Call to Proxy 4... Good!
Call to Proxy 1... Timeout...
Call to Proxy 2... Good!
Call to Proxy 3... Good!
Call to Proxy 4... Good!
Call to Proxy 1... Timeout...
Call to Proxy 2... Good!
Call to Proxy 3... Good!
Call to Proxy 4... Good!
Call to Proxy 1... Timeout...
Call to Proxy 2... Good!
Call to Proxy 3... Good!
Call to Proxy 4... Good!
Call to Proxy 1... Timeout...
Call to Proxy 2... Good!
Call to Proxy 3... Good!
Call to Proxy 4... Good!
Call to Proxy 1... Timeout...
Call to Proxy 2... Good!
Call to Proxy 3... Good!
Call to Proxy 4... Good!
Call to Proxy 1... Timeout...
Call to Proxy 2... Good!
Call to Proxy 3... Good!
Call to Proxy 4... Good!
Call to Proxy 1... Timeout...
Call to Proxy 2... Good!
Call to Proxy 3... Good!
Call to Proxy 4... Good!
Call to Proxy 1... Timeout...
Call to Proxy 2... Good!
Call to Proxy 3... Good!
Call to Proxy 4... Good!
Call to Proxy 1... Timeout...
Call to Proxy 2... Good!
Call to Proxy 3... Good!
Call to Proxy 4... Good!
Call to Proxy 1... Timeout...
Call to Proxy 2... Good!
Call to Proxy 3... Good!
Call to Proxy 4... Good!
Call to Proxy 1... Good!
Call to Proxy 2... Good!
Call to Proxy 3... Good!
Call to Proxy 4... Good!

Edit 3: These questions appears to be very similar: Http Post WebRequest getting timed out

HttpWebRequest and GetResponse hangs after 1 request; other solutions doesn't work for me

WebRequest.GetResponse locks up?

HttpWebRequest times out on second call

Edit 2: Looking at Wireshark, I'm seeing a TCP transmission in the info for the proxy affected. But doesn't that happen with other proxies at the same time? So is that coming from the proxy server itself? It doesn't make sense to me since I am not even getting a response back and the request isn't even being processed.

Edit: Adding code for calls in code. This method is called in a while loop over and over again:

            String html = null;

        HttpWebRequest request = null;
        WebProxy webProxy = null;

        try
        {
            request = (HttpWebRequest)WebRequest.Create(url);

            webProxy = new WebProxy(proxyIP, proxyPort);
            webProxy.Credentials = new NetworkCredential(proxyUser, proxyPass);

            request.Proxy = webProxy;
            request.KeepAlive = false;
            request.Timeout = 5000;
            request.ReadWriteTimeout = 5000;
            request.Method = "GET";
            request.UserAgent = generateAgentString();

            using (WebResponse resp = (WebResponse)request.GetResponse())
            {
                using (Stream strm = resp.GetResponseStream())
                {
                    StreamReader reader = new StreamReader(strm, Encoding.UTF8);

                    try
                    {
                        html = reader.ReadToEnd();
                    }
                    catch
                    {
                        Console.WriteLine("Failed");
                        html = null;
                    }
                    finally
                    {
                        strm.Flush();
                        reader.BaseStream.Dispose();
                        reader.Dispose();
                        strm.Dispose();
                        resp.Dispose();
                    }
                }
            }

            if (request != null)
            {
                request.Abort();
            }
        }
        catch(Exception e) { Console.WriteLine(e); }
like image 675
user2124871 Avatar asked Jan 19 '16 02:01

user2124871


1 Answers

After much digging - I went back to the proxy provider again with a bunch of trace information and digging results and... miraculously... they've decided that they do in fact have limitations that they have not disclosed to the purchaser.

Shocker. I'm more annoyed I went this far after initially discussing with them and getting a clean bill a health on their end.

If anyone needs a recommendation for a provider not to use :)

like image 149
user2124871 Avatar answered Sep 27 '22 22:09

user2124871