C# TCP socket performance degrading as application runs

Question

I wrote a C# server application. The server utilizes Asynchronous TCP sockets.

The packets are 80-180 bytes of data.

For performance testing I have a single client connect and send packets continuously. With debugging on the first 100 packets (0-100) receive in roughly 5 seconds. By the time the server received packets #300-400 it takes roughly 30 seconds to receive the packets. The performance continues to degrade as more receives occur.

I looked around and have not been able to find a solution. I have tried setting the Socket.NoDelay flag in case the Nagle algorithm was inhibiting the server.

I have disabled all functions within the server; so that it is only receiving to ensure I wasn't losing performance in other code.

I have also checked my CPU utilization and it is ~13%. I have over 2 GB of free memory. When running the application the ram is NOT constantly growing and utilization is minimal.

I am at a loss as to what to debug and look into next...

EDIT: Added Code Sample

public void StartListening()
    {

        try
        {
            IPAddress ipAddress = IPAddress.Parse("192.168.2.60");
            IPEndPoint localEndPoint = new IPEndPoint(ipAddress, m_Port);
            m_MainSocket = new Socket(localEndPoint.Address.AddressFamily, SocketType.Stream, ProtocolType.Tcp);
            m_MainSocket.NoDelay = true;
            m_MainSocket.Bind(localEndPoint);
            m_MainSocket.Listen(10);
            m_MainSocket.BeginAccept(new AsyncCallback(clientConnected), null);

            System.Diagnostics.Debug.WriteLine("Listening on:Local IP Address: " + localEndPoint.Address.ToString() + " Port :" + localEndPoint.Port.ToString() + "\n");
        }
        catch (SocketException se)
        {
            System.Diagnostics.Debug.WriteLine("Listening Exception \n");
            System.Diagnostics.Debug.WriteLine(se.Message);
        }
    }

    void clientConnected(IAsyncResult ar)
    {
        try
        {
            SocketState state = new SocketState(m_MainSocket.EndAccept(ar));
            Client client = new Client(state);

            if (client.SocketState.clientSocket.Connected)
            {
                System.Diagnostics.Debug.WriteLine("Client #?????? Connected \n");
                AddLogText("Client #?????? Connected \r\n\r\n");
                waitForData(client);
                SetSendButton(true);
            }

            m_MainSocket.BeginAccept(new AsyncCallback(clientConnected), null);
        }
        catch (ObjectDisposedException)
        {
           System.Diagnostics.Debug.WriteLine("Client Connected: Socket has been closed\n");
        }
        catch (SocketException se)
        {
            System.Diagnostics.Debug.WriteLine("Client Connected Exception \n");
            System.Diagnostics.Debug.WriteLine(se.Message);
        }
    }

    void waitForData(Client client)
    {
        try
        {
            SocketState state = new SocketState(client.SocketState.clientSocket);
            client.SocketState.clientSocket = null; 
            client.SocketState = state;
            client.SocketState.clientSocket.BeginReceive(client.SocketState.DataBuffer, 0, client.SocketState.DataBuffer.Length, SocketFlags.None, new AsyncCallback(readDataCallback), client);
        }
        catch (SocketException se)
        {
            System.Diagnostics.Debug.WriteLine("Wait For Data Exception \n");
            System.Diagnostics.Debug.WriteLine(se.Message);
        }

    }

    public void readDataCallback(IAsyncResult ar)
    {
        Client client = (Client)ar.AsyncState;
        try
        {                
            // Read data from the client socket.
            int iRx = client.SocketState.clientSocket.EndReceive(ar);
            client.SocketState.SB.Append(Encoding.ASCII.GetString(client.SocketState.DataBuffer, 0, iRx));
            string sPacketString = client.SocketState.SB.ToString();

            Server formServer = this;
            Packet_Helper packet_helper = new Packet_Helper(sPacketString, formServer);

            Packet packet = new Packet(sPacketString);
            client.SerialNumber = packet.SerialNumber;
            client.FirmwareVersion = packet.FirmwareVersion;
            client.ProductID = packet.ProductID;
            client.HardwareVersion = packet.HardwareVersion;
            if (!m_Clients.ContainsKey(packet.SerialNumber))
            {
                m_Clients.Add(packet.SerialNumber, client);
                UpdateClientList();
                string[] packets = client.refreshAll();
                for (int i = 0; i < packets.Length; i++)
                {
                    byte[] byteData = Encoding.ASCII.GetBytes(packets[i]);
                    client.SocketState.clientSocket.BeginSend(byteData, 0, byteData.Length, SocketFlags.None, new AsyncCallback(SendCallback), client);
                    AddPacketsSentText(packets[i] + "--" + (iSent++).ToString() + "\r\n\r\n");
                }
            }

            System.Diagnostics.Debug.WriteLine("Read " + sPacketString.Length.ToString() + " bytes from " + client.SerialNumber + "\n" + sPacketString + "\n");
            AddLogText("Read " + sPacketString.Length.ToString() + " bytes from " + client.SerialNumber + " \r\n");
            AddLogText(sPacketString.ToString() + "\r\n\r\n");

            waitForData(client);
        }
        catch (ObjectDisposedException)
        {
            System.Diagnostics.Debugger.Log(0, "1", "\nOnDataReceived: Socket has been closed\n");
        }
        catch (SocketException se)
        {
            if (se.ErrorCode == 10054) // Error code for Connection reset by peer
            {
                string sclientSerial = "??????";
                if (client.SerialNumber != null || client.SerialNumber != "")
                    sclientSerial = client.SerialNumber;
                AddLogText("Client " + sclientSerial + " Disconnected" + "\r\n\r\n");
                System.Diagnostics.Debug.WriteLine("Client " + sclientSerial + " Disconnected" + "\n");

                m_Clients.Remove(sclientSerial);
                UpdateClientList();
            }
            else
            {
                System.Diagnostics.Debug.WriteLine("Read Data Exception \n");
                System.Diagnostics.Debug.WriteLine(se.Message);
            }
        }
    }


class SocketState
{
    private Socket m_ClientSocket;                   //Socket connection to the client
    private byte[] m_DataBuffer = new byte[256];        //Buffer to store the data sent by the client
    private StringBuilder m_SB = new StringBuilder();  //for building recieved data into a string 

    /// <summary>
    /// Gets or Sets the client Socket
    /// </summary>
    public Socket clientSocket
    {
        get { return m_ClientSocket; }
        set { m_ClientSocket = value; }
    }

    /// <summary>
    /// Gets the DataBuffer
    /// </summary>
    public byte[] DataBuffer
    {
        get { return m_DataBuffer; }
        set { DataBuffer = value; }
    }

    /// <summary>
    /// Gets or Sets the SB
    /// </summary>
    public StringBuilder SB
    {
        get { return m_SB; }
        set { m_SB = value; }
    }

    public SocketState(Socket socket)
    {
        m_ClientSocket = socket;
        m_ClientSocket.ReceiveBufferSize = 256;
        m_ClientSocket.NoDelay = true;
        //m_DataBuffer = Enumerable.Repeat((byte)0, 256).ToArray();
    }
}      

Edit: AddLogText() function added. This function is used to add text to a Text Box that is in the UI.

//Delegate - enables asychronous calls for setting the text property of the tb_ListeningLog

delegate void AddLogTextCallback(string text);

private void AddLogText(string text)
    {
        // InvokeRequired required compares the thread ID of the
        // calling thread to the thread ID of the creating thread.
        // If these threads are different, it returns true.
        if (this.tb_ListeningLog.InvokeRequired)
        {
            AddLogTextCallback d = new AddLogTextCallback(AddLogText);
            this.Invoke(d, new object[] { text });
        }
        else
        {
            this.tb_ListeningLog.Text += text;
            tb_ListeningLog.SelectionStart = tb_ListeningLog.Text.Length;
            tb_ListeningLog.ScrollToCaret();
        }
    }

Answer 1

I'm taking a bit of a shot in the dark with this answer, but the code you've posted certainly helps.

The reason you're probably seeing slow performance as time goes on is because of the code in your readDataCallback method. The way you have it set up, the processing of the data is done before you go for another receive. This means that as the length of the processing increases, the duration between receiving your data increases.

I don't know what code is in a lot of your methods, but you should generally look at any loops that may be taking a while to finish. If you're having trouble finding the bottleneck by looking through your code, try finding which methods take the longest to finish and continue to narrow your code down.

For instance (I'm guessing that the bottleneck is in this area of code):

if (!m_Clients.ContainsKey(packet.SerialNumber))
{
    m_Clients.Add(packet.SerialNumber, client);

    AddLogText("Running UpdateClientList\r\n");

    UpdateClientList();

    AddLogText("Doing client.refreshAll\r\n");

    string[] packets = client.refreshAll();

    AddLogText("Doing for loop\r\n");

    for (int i = 0; i < packets.Length; i++)
    {
        byte[] byteData = Encoding.ASCII.GetBytes(packets[i]);
        client.SocketState.clientSocket.BeginSend(byteData, 0, byteData.Length, SocketFlags.None, new AsyncCallback(SendCallback), client);
        AddPacketsSentText(packets[i] + "--" + (iSent++).ToString() + "\r\n\r\n");
    }
}

Just observe the amount of time between each method with your eyes, or make it easier and use a Stopwatch or DateTime to show exact time.

Also, if you find that the behavior of the code cannot be made more efficient, you could toy around with the idea of processing the data in a separate thread. I'm assuming that this behavior isn't desired, though, because of the question at hand.

---

For your AddLogText method, try using tb_ListeningLog.Text.AppendText instead of +=.