"An existing connection was forcibly closed by the remote host" sporadic error from HttpWebRequest

Question

I'm having an odd, sporadic problem with a simple test app (Visual Studio console application) that acts as a watchdog for a hosted aspx web site (not app) by sending it simple http requests and timing the response times. I've used this for many weeks in the past without this particular problem. At seemingly random times during the day, my http requests start failing with the above error. They appear to be timeouts since the requests that fail are all taking 60 seconds. After a period of consistent errors as per above (random time periods from a few minutes to 90 minutes in one case) the errros stop and http responses start coming back with no errors at the normal speed (usually about .25s). The watchdog client requests triggers a very simple database lookup, just 1-2 lines of code on the server side. This is hosted at a shared windows hosting web host.

I can also trigger this behavior at will by updating any .cs file on my host site, which, among other things, causes the app pool to recycle. Immediately my watchdog app starts timing out again with the above error.

It smells like some kind of recycled connection problem, since if I simply restart the watchdog app, it works fine and responses start coming back at the normal delay.

I've tried setting request.KeepAlive = false and request.ServicePoint.ConnectionLimit = 1, those did not help.

Another clue, I cannot connect with IIS manager to either of two different websites hosted on this server, which has always worked fine. I'm getting "The underlying connection was closed" trying to connect via IIS Manager. Every time. I have not updated either of the sites in a while, so it wasn't any change of mine.

This is an asp.net 4 website on the backend running on IIS7 with a dedicated app pool in integrated pipeline mode.

Also if I change the sleeptime variable in the watchdog app to something like 30 seconds, the problem doesn't show up. There's some magic number in the range of I believe 10-20 seconds where if the requests are pausing more than that, they never fail.

I think the fact that IIS Manager can't connect is good evidence that something is wrong on the host side independant of my test app but I wanted to cover my bases before opening a support incident... especially since a simple restart of my console app fixes the problem... at least for a while.

class Program
{
            //make a simple web request and just return the status code
    static string SendHttpMsg(string url, string postData)
    {
        HttpWebRequest request = (HttpWebRequest)WebRequest.Create(new Uri(url));
        request.Method = "POST";
        request.ContentType = "application/x-www-form-urlencoded";
        ASCIIEncoding encoding = new ASCIIEncoding();
        byte[] byte1 = encoding.GetBytes(postData);
        request.ContentLength = byte1.Length;
        //request.KeepAlive = false; //no effect
        //request.ServicePoint.ConnectionLimit = 1; //no effect
        Stream requestStream = request.GetRequestStream();
        requestStream.Write(byte1, 0, byte1.Length);
        requestStream.Close();
        HttpWebResponse response = (HttpWebResponse)request.GetResponse();
        response.Close();
        return ((int)response.StatusCode).ToString();
    }

    static void Main(string[] args)
    {
        int sleeptime = 5000;
        string result = "";
        while (true)
        {
            DateTime start = DateTime.Now;
            try
            {
                //this is a very simple call that results in a very simple, fast query in the database
                result = SendHttpMsg("http://example.com/somepage.aspx", "command=test");
            }
            catch (Exception ex)
            {
                                //fancy exception handling here, removed
            }
            DateTime end = DateTime.Now;
            TimeSpan calltime = end - start;
            Console.WriteLine(end.ToString() + ", " + calltime.TotalSeconds.ToString("F") + " seconds " + result);
            Thread.Sleep(sleeptime);
        }
    }
}

Answer 1

You could have dangling connections, and in HTTP 1.1 you are limited to 2 connections.

Try changing the HTTP Protocol Version used in the request:

request.ProtocolVersion = HttpVersion.Version10;

If that doesn't work, it could be taking a long time to resolve the proxy settings, which can be fixed by disabling the proxy settings in your application by adding the following to the .config file:

<system.net>
  <defaultProxy enabled="false">
    <proxy/>
    <bypasslist/>
    <module/>
  </defaultProxy>
</system.net>

If either of the above fixes the problem, I'd recommend adding a try...catch...finally block around your request code, to ensure that each request is properly closed and disposed of (try setting request = null in your finally statement inside the method).