HttpClient throwing "An error occurred while sending the request."

Question

I have three layer application architecture.

My Client --> My service A (REST hosted in IIS) --> Other Team's service X (REST).

Service A is ASP.Net 4.6.1 framework, not ASP.Net Core.

Client is communicating to A with HttpClient and A is communicating to X with HttpClient.

Client is firing almost 2500 calls to my service to A and to X.

Out of 2500 calls service A randomly (may be 10 calls) fails with below exception. Its not reproducible.

System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> 
System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a 
receive. ---> System.IO.IOException: Unable to read data from the transport connection: An     
established connection was aborted by the software in your host machine. ---> 
System.Net.Sockets.SocketException: An established connection was aborted by the software in your 
host machine
at System.Net.Sockets.Socket.BeginReceive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags 
socketFlags, AsyncCallback callback, Object state)
at System.Net.Sockets.NetworkStream.BeginRead(Byte[] buffer, Int32 offset, Int32 size, AsyncCallback 
callback, Object state)
 --- End of inner exception stack trace ---
at System.Net.Security._SslStream.EndRead(IAsyncResult asyncResult)
at System.Net.TlsStream.EndRead(IAsyncResult asyncResult)
at System.Net.Connection.ReadCallback(IAsyncResult asyncResult)
--- End of inner exception stack trace ---
at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
at System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar)
--- End of inner exception stack trace --

Here is my service A call. A in IIS calls below code block and its called by each request. X is taking user credentials and returning data based on user, so we are not sharing HttpClient between calls.

var user = (System.Security.Principal.WindowsIdentity)HttpContext.Current.User.Identity;
            System.Security.Principal.WindowsIdentity.RunImpersonated(user.AccessToken, () =>
      {
        static HttpClient Client = new HttpClient();
        static string CallX(string[] args)
        {
            HttpClientHandler handler = new HttpClientHandler
            {
                UseDefaultCredentials = true
            };    

            Client = new HttpClient(handler)
            {
                BaseAddress = new Uri("http://XserviceUrl/api/")
            };
            Client.Timeout = TimeSpan.FromSeconds(600);    
            var result = Client.PostAsync("Fake X controller"
                , new StringContent(JsonConvert.SerializeObject(args)
                , Encoding.UTF8, "application/json")).Result;    

            result.EnsureSuccessStatusCode();

            var json = result.Content.ReadAsStringAsync().Result;
            return DosomethingWithResult(json);    
        }
    });

Things I tried:

Some SO post suggested might be timeout issue. So I added 600 seconds in Client and in Service A. I also changed IIS request timeout from default 2 minutes to 10 (600 seconds).

Answer 1

After some digging I resolved the issue. When A was sending the request to X, A was setting the Connection: keep-alive and the X was responding with Connection: Close property in header.

So after some calls A was exhaust with opened tcp connection and randomly it was throwing the error.

(Fiddler helped me to figure this out)

So all I had to do was setting ConnectionClose property of HttpClient

_client.DefaultRequestHeaders.ConnectionClose = true;