Why is Socket.Receive timing out on a half-closed connection when timeout is set to infinite?

Question

Below is a C# program demonstrating the problem.

The server starts listening on a socket. The client connects to the server, sends a message, uses Shutdown(SocketShutdown.Send) to close its send half of the connection to let the server know where the end of the message is, and waits for a response from the server. The server reads the message, does some lengthy computation (simulated here with a sleep call), sends a message to the client, and closes the connection.

On Windows, the client's Receive call always fails after exactly 2 minutes with "A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond" even though the timeout is set to infinite.

If I run the program in Linux with Mono, the timeout does not occur even if I set the "lengthy operation" to 10 minutes, but it happens in Windows whether I run it with Mono or .NET. If I set the timeout to 1 second, it times out after 1 second. In other words, it times out in the timeout I set or 2 minutes, whichever is less.

A similar sample program in which the server sends a message to the client, with no message from client to server and no half-close, works as expected with no timeout.

I can get around this by modifying my protocol to use some other method of indicating to the server when a message is complete (perhaps prefixing the message with the length of the message). But I want to know what's going on here. Why does Socket.Receive time out on a half-closed connection when the timeout is set to infinite?

From what I understand, a connection with only its send half closed should be able to continue receiving data indefinitely. It seems unlikely that there would be a bug in such a fundamental part of Windows. Am I doing something wrong?

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading;
using System.Net.Sockets;
using System.Net;
using System.Threading.Tasks;
using System.Diagnostics;

namespace ConsoleApplication1
{
    class Program
    {
        static void Main(string[] args)
        {
            // Start server thread
            Thread serverThread = new Thread(ServerStart);
            serverThread.IsBackground = true;
            serverThread.Start();

            // Give the server some time to start listening
            Thread.Sleep(2000);

            ClientStart();
        }

        static int PortNumber = 8181;

        static void ServerStart()
        {
            TcpListener listener = new TcpListener(new IPEndPoint(IPAddress.Any, PortNumber));
            listener.Start();
            while (true)
            {
                TcpClient client = listener.AcceptTcpClient();
                Task connectionHandlerTask = new Task(ConnectionEntryPoint, client);
                connectionHandlerTask.Start();
            }
            listener.Stop();
        }

        static void ConnectionEntryPoint(object clientObj)
        {
            using (TcpClient client = (TcpClient)clientObj)
            using (NetworkStream stream = client.GetStream())
            {
                // Read from client until client closes its send half.
                byte[] requestBytes = new byte[65536];
                int bufferPos = 0;
                int lastReadSize = -1;
                while (lastReadSize != 0)
                {
                    lastReadSize = stream.Read(requestBytes, bufferPos, 65536 - bufferPos);
                    bufferPos += lastReadSize; 
                }
                client.Client.Shutdown(SocketShutdown.Receive);
                string message = Encoding.UTF8.GetString(requestBytes, 0, bufferPos);

                // Sleep for 2 minutes, 30 seconds to simulate a long-running calculation, then echo the client's message back
                byte[] responseBytes = Encoding.UTF8.GetBytes(message);
                Console.WriteLine("Waiting 2 minutes 30 seconds.");
                Thread.Sleep(150000);

                try
                {
                    stream.Write(responseBytes, 0, responseBytes.Length);
                }
                catch (SocketException ex)
                {
                    Console.WriteLine("Socket exception in server: {0}", ex.Message);
                }
            }
        }

        static void ClientStart()
        {
            using (Socket socket = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp))
            {
                // Set receive timeout to infinite.
                socket.ReceiveTimeout = -1;

                // Connect to server
                socket.Connect(IPAddress.Loopback, PortNumber);

                // Send a message to the server, then close the send half of the client's connection
                // to let the server know it has the entire message.
                string requestMessage = "Hello";
                byte[] requestBytes = Encoding.UTF8.GetBytes(requestMessage);
                socket.Send(requestBytes);
                socket.Shutdown(SocketShutdown.Send);

                // Read the server's response. The response is done when the server closes the connection.
                byte[] responseBytes = new byte[65536];
                int bufferPos = 0;
                int lastReadSize = -1;

                Stopwatch timer = Stopwatch.StartNew();
                try
                {
                    while (lastReadSize != 0)
                    {
                        lastReadSize = socket.Receive(responseBytes, bufferPos, 65536 - bufferPos, SocketFlags.None);
                        bufferPos += lastReadSize;
                    }

                    string responseMessage = Encoding.UTF8.GetString(responseBytes, 0, bufferPos);
                    Console.WriteLine(responseMessage);
                }
                catch (SocketException ex)
                {
                    // Timeout always occurs after 2 minutes. Why?
                    timer.Stop();
                    Console.WriteLine("Socket exception in client after {0}: {1}", timer.Elapsed, ex.Message);
                }
            }
        }
    }
}

The following program prefixes messages with a 4-byte message length rather than using socket.Shutdown(SocketShutdown.Send) to signal end of message. The timeout does not occur in this program.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Net.Sockets;
using System.Net;
using System.Threading.Tasks;
using System.Diagnostics;
using System.Threading;

namespace WithoutShutdown
{
    class Program
    {
        static void Main(string[] args)
        {
            // Start server thread
            Thread serverThread = new Thread(ServerStart);
            serverThread.IsBackground = true;
            serverThread.Start();

            // Give the server some time to start listening
            Thread.Sleep(2000);

            ClientStart();
        }

        static int PortNumber = 8181;

        static void ServerStart()
        {
            TcpListener listener = new TcpListener(new IPEndPoint(IPAddress.Any, PortNumber));
            listener.Start();
            while (true)
            {
                TcpClient client = listener.AcceptTcpClient();
                Task connectionHandlerTask = new Task(ConnectionEntryPoint, client);
                connectionHandlerTask.Start();
            }
            listener.Stop();
        }

        static void SendMessage(Socket socket, byte[] message)
        {
            // Send a 4-byte message length followed by the message itself
            int messageLength = message.Length;
            byte[] messageLengthBytes = BitConverter.GetBytes(messageLength);
            socket.Send(messageLengthBytes);
            socket.Send(message);
        }

        static byte[] ReceiveMessage(Socket socket)
        {
            // Read 4-byte message length from the client
            byte[] messageLengthBytes = new byte[4];
            int bufferPos = 0;
            int lastReadSize = -1;
            while (bufferPos < 4)
            {
                lastReadSize = socket.Receive(messageLengthBytes, bufferPos, 4 - bufferPos, SocketFlags.None);
                bufferPos += lastReadSize;
            }
            int messageLength = BitConverter.ToInt32(messageLengthBytes, 0);

            // Read the message
            byte[] messageBytes = new byte[messageLength];
            bufferPos = 0;
            lastReadSize = -1;
            while (bufferPos < messageLength)
            {
                lastReadSize = socket.Receive(messageBytes, bufferPos, messageLength - bufferPos, SocketFlags.None);
                bufferPos += lastReadSize;
            }

            return messageBytes;
        }

        static void ConnectionEntryPoint(object clientObj)
        {
            using (TcpClient client = (TcpClient)clientObj)
            {
                byte[] requestBytes = ReceiveMessage(client.Client);
                string message = Encoding.UTF8.GetString(requestBytes);

                // Sleep for 2 minutes, 30 seconds to simulate a long-running calculation, then echo the client's message back
                byte[] responseBytes = Encoding.UTF8.GetBytes(message);
                Console.WriteLine("Waiting 2 minutes 30 seconds.");
                Thread.Sleep(150000);

                try
                {
                    SendMessage(client.Client, responseBytes);
                }
                catch (SocketException ex)
                {
                    Console.WriteLine("Socket exception in server: {0}", ex.Message);
                }
            }
        }

        static void ClientStart()
        {
            using (Socket socket = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp))
            {
                // Set receive timeout to infinite.
                socket.ReceiveTimeout = -1;

                // Connect to server
                socket.Connect(IPAddress.Loopback, PortNumber);

                // Send a message to the server
                string requestMessage = "Hello";
                byte[] requestBytes = Encoding.UTF8.GetBytes(requestMessage);
                SendMessage(socket, requestBytes);

                // Read the server's response.
                Stopwatch timer = Stopwatch.StartNew();
                try
                {
                    byte[] responseBytes = ReceiveMessage(socket);
                    string responseMessage = Encoding.UTF8.GetString(responseBytes);
                    Console.WriteLine(responseMessage);
                }
                catch (SocketException ex)
                {
                    // Timeout does not occur in this program because it does not call socket.Shutdown(SocketShutdown.Send)
                    timer.Stop();
                    Console.WriteLine("Socket exception in client after {0}: {1}", timer.Elapsed, ex.Message);
                }
            }
        }
    }
}

Answer 1

This behavior is by design. When the client has closed its half on the connection and the server acknowledges the close, the client is in the FIN_WAIT_2 state, waiting for the server to close the connection. http://support.microsoft.com/kb/923200 states that there is a FIN_WAIT_2 timeout of 2 minutes. If no data is received in a 2 minute period when a connection is in the FIN_WAIT_2 state, the client forcibly closes the connection (with a RST).

By default in Windows Server 2003, TCP connections must close after the TCP connection state has been set to FIN_WAIT_2 for two minutes.

This old Apache article suggests the reason for the timeout: malicious or misbehaving applications could keep the other end of the connection in FIN_WAIT_2 indefinitely by never closing their end of the connection and thus tie up operating system resources.

Linux apparently has a timeout as well You can check the value with

$ cat /proc/sys/net/ipv4/tcp_fin_timeout

I'm not sure why the timeout wasn't occurring for me on Linux. Perhaps because it was a loopback connection and therefore DoS attacks are not a concern or loopback connections use different code that does not use the tcp_fin_timeout setting?

Bottom line: The operating system has a good reason for making the connection time out. Avoid using Shutdown as an application-layer signalling mechanism and use an actual application-layer method instead.