Random "An existing connection was forcibly closed by the remote host." after a TCP reset

Question

I have two parts, a client and a server. And I try to send data (size > 5840 Bytes) from the client to the server and then the server sends the data back. I loop this a number of times waiting a second between each time. Sometime the server application crash, the crash seems very random the error:

Unhandled Exception: System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. --->

System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host

at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)

at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 s ize)

--- End of inner exception stack trace ---

at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 s ize)

at TCP_Server.Program.Main(String[] args)

Client code (This is inside the loop):

            try
            {
                Int32 port = 13777;
                using (TcpClient client = new TcpClient(ip, port))
                using (NetworkStream stream = client.GetStream())
                {
                    client.Client.SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.KeepAlive, true);
                    var data = GenerateData(size);

                    sw.Start();
                    // Send the message to the connected TcpServer. 
                    stream.Write(data, 0, data.Length);

                    // Buffer to store the response bytes.
                    data = new Byte[size];

                    // Read the first batch of the TcpServer response bytes.
                    Int32 bytes = stream.Read(data, 0, data.Length);

                    sw.Stop();
                    Console.WriteLine(i + ": Done transporting " + size + " bytes to and from " + ip + " time: " +
                                      sw.ElapsedMilliseconds + " ms");
                    // Close everything.
                    stream.Close();
                    client.Close();
                }

            }
            catch (ArgumentNullException e)
            {
                Console.WriteLine("ArgumentNullException: {0}", e);
            }
            catch (SocketException e)
            {
                Console.WriteLine("SocketException: {0}", e);
            }

            sw.Reset();

Server code:

            Byte[] bytes = new Byte[size];

            // Enter the listening loop. 
            for (int i = 0; i < numberOfPackages; i++)
            {
                using (TcpClient client = server.AcceptTcpClient())
                using (NetworkStream stream = client.GetStream())
                {
                    client.Client.SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.KeepAlive, true);
                    // Loop to receive all the data sent by the client. 
                    while ((stream.Read(bytes, 0, bytes.Length)) != 0)
                    {
                        // Send back a response.
                        stream.Write(bytes, 0, size);

                    }
                    client.GetStream().Close();
                    client.Close();
                }


                Console.WriteLine("Receive data size " + size);

            }

I have used wireshark to monitor the tcp packages sent and found that a TCP RST is sent from the client to the server before the program crashes. So I assume the problem is that the the RST is not handles correctly. There is no firewall between the client and the host so that is not the problem.

The wireshark files for both the Client and Server is here: https://www.dropbox.com/sh/ctl2chq3y2c20n7/AACgIJ8IRiclqnyOyw8sqd9La?dl=0

So either I need to get rid of the TCP RST or I need my server to handle it in some way and not crash.

I have tried to use longer waiting time but it does not help. If the data is below 5840 Bytes I do not get any crashes, what I know of.

Any suggestions or ideas?

EDIT: Thanks to the answers I got it to work with the following changes:

Server:

// Loop to receive all the data sent by the client.
int k = 0;
while (k < size)
{
   int bytesRead = stream.Read(bytes, 0, bytes.Length);
   k += bytesRead;
}
// Send back a response.
stream.Write(bytes, 0, size);

And the same when receiving on the client side. Since I first want to send all data and then let the server respond this works for my application.

Answer 1

We had this problem when using HttpClient random errors like

System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.
 ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.
 ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host

or

 ---> System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a send.

Problem in Explicit Congestion Notification, or ECN, in the TCP stack. This is also known as ECN Capability. Explicit Congestion Notification is an extension to the Internet Protocol and to the Transmission Control Protocol and is defined in RFC 3168. ECN allows end-to-end notification of network congestion without dropping packets. If ECN Capability is enabled, you can disable it:

netsh interface tcp set global ecncapability=disabled

A reboot is not required.

Answer 2

Two problems that I see:

1. You are assuming that a read for size bytes will actually read size bytes. It does not. TCP is a streaming protocol. A read reads at least one byte. That's the only guarantee given.
2. Your code will randomly deadlock. The client writes data to the server. The server echos it back. But the client will not read until it has written everything. If it writes more than the network buffers can take this will deadlock. The client must read and write concurrently. Probably, you need another thread/task for reading the data back. A good pattern to solve that issue would be to start one writer task, one reader task and to Task.WhenAll/WaitAll to join them back.

I'm not sure under what exact circumstances a TCP stack would send a RST. Could be due to the deadlock causing a timeout.

You are not swallowing any exceptions, right?

Normally, closing a connection performs an orderly shutdown in the background. But I'm unsure about what happens when the other side is still writing at this point. Maybe the answer is that an RST is generated by the receiver of the writes. Surely, the TCP spec would answer this question. If this answer is to be trusted then indeed a Shutdown(Read)/Close followed by an incoming write would RST the connection.

Fix both issues and report back with your findings.