Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

receiving variable size of data over TCP sockets

I ran into a little issue with transferring data over (TCP) sockets. Small background on what I am doing:

I am sending data from side A to B. Data sent can be of variable lengths, assuming max size to be of 1096 bytes.

A) send(clientFd, buffer, size, NULL)

on B, since I dont know what size to expect, I always try to receive 1096 bytes:

B) int receivedBytes = receive(fd, msgBuff, 1096, NULL)

However, when I did this: I realized A was sending small chunks of data..say around 80-90 bytes. After a few bursts of sending, B was clubbing them together to have receivedBytes to be 1096. This obviously corrupted data and hell broke loose.

To fix this, I broke my data in two parts: header and data.

struct IpcMsg
{
   long msgType;
   int devId;
   uint32_t senderId;
   uint16_t size; 
   uint8_t value[IPC_VALUES_SIZE]; 
};

On A side:

A) send(clientFd, buffer, size, NULL)

on B, I first receive the header and determine the size of payload to receive: and then receive the rest of the payload.

B) int receivedBytes = receive(fd, msgBuff, sizeof(IpcMsg) - sizeof( ((IpcMsg*)0)->value ), 0);
int sizeToPoll = ((IpcMsg*)buffer)->size;
printf("Size to poll: %d\n", sizeToPoll);

if (sizeToPoll != 0)
{
        bytesRead = recv(clientFd, buffer + receivedBytes, sizeToPoll, 0); 
}

So, for every send which has a payload, I end up calling receive twice. This worked for me, but I was wondering if there is a better way of doing this ?

like image 621
brainydexter Avatar asked Sep 12 '14 07:09

brainydexter


People also ask

When using sockets which command will wait for incoming data?

If data is not available for the socket socket, and socket is in blocking mode, the recv() call blocks the caller until data arrives. If data is not available and socket is in nonblocking mode, recv() returns a -1 and sets the error code to EWOULDBLOCK.

What is TCP socket size?

The default buffer size is 8 KB. The maximum size is 8 MB (8096 KB). The optimal buffer size depends on several network environment factors including types of switches and systems, acknowledgment timing, error rates and network topology, memory size, and data transfer size.

What is Max buffer size of TCP socket?

The maximum send buffer size is 1,048,576 bytes. The default value of the SO_SNDBUF option is 32767. For a TCP socket, the maximum length that you can specify is 1 GB.

What happens when you call read () or recv ()) on an open socket UDP or TCP and there is no data datagrams or bytes in the buffer to be read?

If no messages are available at the socket, the receive calls wait for a message to arrive, unless the socket is nonblocking (see fcntl(2)), in which case the value -1 is returned and errno is set to EAGAIN or EWOULDBLOCK.


2 Answers

You're on the right lines with the idea of sending a header that contains basic information about the following data, followed by the data itself. However, this won't always work:

int receivedBytes = receive(fd, msgBuff, sizeof(IpcMsg) - sizeof( ((IpcMsg*)0)->value ), 0);
int sizeToPoll = ((IpcMsg*)buffer)->size;

The reason is that TCP is free to fragment and send your header in as many chunks as it sees fit based on its own assessment of the underlying network conditions applied to what's called the congestion control strategy. On a LAN you'll pretty much always get your header in one packet but try it across the world through the internet and you may get a much smaller number of bytes at a time.

The answer is to not call TCP's 'receive' (usually recv) directly but abstract it away into a small utility function that takes the size you really must receive and a buffer to put it into. Go into a loop receiving and appending packets until all data has arrived or an error occurs.

If you need to go asynchronous and serve multiple clients simultaneously then the same principal applies but you need to go investigate the 'select' call that allows you to be notified when data arrives.

like image 162
Andy Brown Avatar answered Oct 13 '22 00:10

Andy Brown


TCP/IP is a "raw" interface for sending data. It does guarantee that, if the bytes are sent, that they are all there and in the right order, but does not make any guarantees about chunking and knows nothing about the data you are sending.

Therefore, if sending a "packet" over TCP/IP that is to be processed as such, you must know when you have a full packet by one of the following techniques:

  • Fixed-sized packets. In your case 1096 bytes
  • First send / receive a known "header" that will tell you the size of the packet being sent.
  • Use some kind of "end of packet" symbol.

In either of the first two, you know the number of bytes you are expecting to receive so you need to buffer anything you receive until you have the full message, then process that.

If you receive more than you expected, i.e. it spills over into the next packet, you split that, process the completed packet and leave the remainder buffered for processing subsequently.

In the latter case where you have an end of packet symbol, that could be anywhere in your message so anything that follows it, you buffer for the next packet.

like image 30
CashCow Avatar answered Oct 13 '22 01:10

CashCow