Send more data than what can be stored in send buffer

Question

I am using C# sockets to implement my own basic RTSP server (for learning purposes). I am currently in the process of writing the logic for the server to perform the handshake with the client when negotiating media ports on a DESCRIBE request.

The client application was not written by me and I have no access to update the software, it is a simple RTSP client which negotiates media ports on a Cisco router and then pipes audio to clients connected to it.

The payload I want to send back is 1024 bytes and a 64 byte header, and from what I can see from inspecting my client app, only 400 bytes are transmitted back to the client.

The response payload is defined in C# as:

// Build the RTSP Response string
var body =      "v=0\n" +
                "o=- 575000 575000 IN IP4 " + m_serverIP + "\n" +
                "s=" + stream + "\n" +
                "i=<No author> <No copyright>\n" + 
                "c= IN IP4 0.0.0.0\n"+
                "t=0 0\n" +
                "a=SdpplinVersion:1610641560\n" +
                "a=StreamCount:integer;1\n" +
                "a=control:*\n" +
                "a=Flags:integer;1\n" +
                "a=HasParam:integer;0\n" +
                "a=LatencyMode:integer;0\n" + 
                "a=LiveStream:integer;1\n" + 
                "a=mr:intgner;0\n" +
                "a=nr:integer;0\n" +
                "a=sr:integer;0\n" + 
                "a=URL:string;\"Streams/" + stream + "\"\n" +
                "a=range:npt=0-\n" + 
                "m=audio 0 RTP/AVP 8" + // 49170 is the RTP transport port and 8 is A-Law audio
                "b=AS:90\n" +
                "b=TIAS:64000\n" +
                "b=RR:1280\n" +
                "b=RS:640\n" + 
                "a=maxprate:50.000000\n" +
                "a=control:streamid=1\n" +
                "a=range:npt=0-\n" +
                "a=length:npt=0\n" +
                "a=rtpmap:8 pcma/8000/1\n" +
                "a=fmtp:8" +
                "a=mimetype:string;\"audio/pcma\"\n" +
                "a=ASMRuleBook:string;\"marker=0, Avera**MSG 00053 TRUNCATED**\n" +
                "**MSG 0053 CONTINUATION #01**geBandwidth=64000, Priority=9, timestampdelivery=true;\"\n" +
                "a=3GPP-Adaptation-Support:1\n" +
                "a=Helix-Adaptation-Support:1\n" +
                "a=AvgBitRate:integer;64000\n" +
                "a=AvgPacketSize:integer;160\n" + 
                "a=BitsPerSample:integer;16\n" +
                "a=LiveStream:integer;1\n" + 
                "a=MaxBitRate:integer;64000\n" +
                "a=MaxPacketSize:integer;160\n" +
                "a=Preroll:integer;2000\n" +
                "a=StartTime:integer;0\n" +
                "a=OpaqueData:buffer;\"AAB2dwAGAAEAAB9AAAAfQAABABAAAA==\""; 

var header =    "RTSP/1.0 200 OK\n" +
                "Content-Length: " + body.Length + "\n" +
                "x-real-usestrackid:1\n" +
                "Content-Type: application/sdp\n" +
                "Vary: User-Agent, ClientID\n" +
                "Content-Base: " + requestURI + "\n" +
                "vsrc:" + m_serverIP + "/viewsource/template.html\n" +
                "Set-Cookie: " + Auth.getCookie() + "\n" +
                "Date: " + System.DateTime.Now + "\n" +
                "CSeq: 0\n";

var response = header + body;

var byteArray = System.Text.Encoding.UTF8.GetBytes(response);
respondToClient(byteArray, socket);

The responding method respondToClient is implemented as:

private static void respondToClient(byte[] byteChunk, Socket socket)
{
    socket.BeginSend(byteChunk, 0, byteChunk.Length, SocketFlags.None, new AsyncCallback(SendCallback), socket);
    socket.BeginReceive(m_dataBuffer, 0, m_dataBuffer.Length, SocketFlags.None, new AsyncCallback(ReceiveCallback), socket);
}

With a SendCallback to determine when the packet has been sent to the client:

private static void SendCallback(IAsyncResult res)
{
    try
    {
        var socket = (Socket)res.AsyncState;
        socket.EndSend(res);
    }
    catch (Exception e)
    {
        Console.Write(e.Message);
    }
}

Why is my client application only receiving 400 bytes, and if so how can I make sure my whole packet is sent (I'm using TCP)? Is this a hard limit set by C#? I consulted MSDN and could not find any information to support this theory.

If there are more details required regarding the implementation of my server, please ask.

EDIT Here is some food for thought, and I don't know why it would be happening so if somebody could enlighten me that would be great... @ Gregory A Beamer suggested in the comments that EndSend could be firing prematurely, is this a possibility? From my understanding I thought the async callback would only fire once, and only once all bytes had been sent from the server to the listening client socket.

In order for me to tell that the packet wasn't being fully recieved by the client, the following information states that only 400 bytes are received by the client application:

Supported by a wireshark trace from S->C

UPDATE After speaking earlier with @usr I did some further tests and can now confirm that the client application will only ever receive a 400 byte response back from the server. I h

Jul  1 13:28:29: //-1//RTSP:/rtsp_read_svr_resp: Socket = 0
Jul  1 13:28:29: //-1//RTSP:/rtsp_read_svr_resp: NBYTES = 1384
Jul  1 13:28:29: //-1//RTSP:/rtsp_process_single_svr_resp:
Jul  1 13:28:29: rtsp_process_single_svr_resp: 400 bytes of data:
RTSP/1.0 200 OK
Content-Length: 1012
x-real-usestrackid:1
Content-Type: application/sdp
Vary: User-Agent, ClientID
Content-Base: rtsp://10.96.134.50/streams/stream01.dcw
vsrc:10.96.134.50/viewsource/template.html
Set-Cookie: cbid=aabtvqjcldwjwjbatynprfpfltxaspyopoccbtewiddxuzhsesflnkzvkwibtikwfhuhhzzz;path=/;expires=09/10/2015 14:28:38
Date: 01/07/2015 14:28:38
CSeq: 0  
v=0
o=- 575000 575000 IN IP4

^ This is the log trace dumped by the client application when I dial in to the server, here we can see that it accept 1384 bytes back from the client, but its processing buffer is only capable of storing 400 bytes. I looked at a log for an existing application which handles streaming to the same device, and in the wireshark trace, instead of sending the header in one block, it sends several chunks back with the info [TCP segment of a reassembled PDU] I am not sure what this means (new to network stuff) but I assume its breaking down the server payload and sending it over in several chunks and then reassembling on the client to handle the request.

How can I replicate this behaviour on my server application? I tried the following in my respond method, with hopes that it would pipe back the packet and reassemble on the client, however I just continued to get the same error discussed above:

private static void respondToClient(byte[] byteChunk, Socket socket)
{
    // Divide the byte chunk into 300 byte chunks
    if (byteChunk.Length >= 400)
    {
        Console.WriteLine("Total bytes: " + byteChunk.Length);

        var totalBytes = byteChunk.Length;
        var chunkSize = 400;
        var tmp = new byte[chunkSize];

        var packets  = totalBytes / chunkSize;
        var overflow = totalBytes % chunkSize;

        Console.WriteLine("Sending " + totalBytes + " in " + packets + " packets, with an overflow packets of " + overflow + " bytes.");

        for (var i = 0; i < packets * chunkSize; i+=chunkSize)
        {
            Console.WriteLine("Sending chunk " + i + " to " + (i + chunkSize));
            tmp = byteChunk.Skip(i).Take(chunkSize).ToArray();
            socket.BeginSend(tmp, 0, tmp.Length, SocketFlags.None, new AsyncCallback(SendCallback), socket);
        }
        if (overflow > 0)
        {
            Console.WriteLine("Sending overflow chunk: " + overflow + " at index " + (totalBytes - overflow) + " to " + (totalBytes ));
            tmp = byteChunk.Skip(byteChunk.Length - overflow).Take(overflow).ToArray();
            socket.BeginSend(tmp, 0, tmp.Length, SocketFlags.None, new AsyncCallback(SendCallback), socket);
        }
    }
    else
    {
        socket.BeginSend(byteChunk, 0, byteChunk.Length, SocketFlags.None, new AsyncCallback(SendCallback), socket);
        socket.BeginReceive(m_dataBuffer, 0, m_dataBuffer.Length, SocketFlags.None, new AsyncCallback(ReceiveCallback), socket);
    }
}

Answer 1

TCP is a boundaryless stream of bytes. There are no packets perceivable to applications. Read/Receive calls return any number of bytes in a non-deterministic way (at least one). In the receiver you must deal with the fact that reads can be partial. Use the return value of Receive to determine the number of bytes received.

This is all easier if you read synchronously and use BinaryReader because it does this looping logic for you.

BeginSend could be firing prematurely, is this a possibility

No.

Answer 2

This may seem controversial, but the bug in the code was extremely silly and I have to attribute the success of this question to @Keyz182 in the answers. After discussing privately he noticed that I was using \n to add new line elements.

The problem was I needed to change these returns to \r\n and add one between the header and the body to dictate to the client where the header ended and the body information of the payload started (smacks head against brick wall).

It turns out that there was no 400 byte buffer on the client (so I am still amused as to why it talked about a processing buffer of 400 bytes). Thank-you all for your help!