C++ TCP socket sending speed

Question

I'm sending messages to remote server using simple locking TCP socket and the problem I have is that for each message it takes very different time to send it.

And here what I get (some example):

Bytes Sent:  217, Time:  34.3336 usec
Bytes Sent:  217, Time:   9.9107 usec
Bytes Sent:  226, Time:  20.1754 usec
Bytes Sent:  226, Time:  38.2271 usec
Bytes Sent:  217, Time:  33.6257 usec
Bytes Sent:  217, Time:  12.7424 usec
Bytes Sent:  217, Time:  21.5912 usec
Bytes Sent:  217, Time:  31.1480 usec
Bytes Sent:  218, Time:  28.3164 usec
Bytes Sent:  218, Time:  13.0963 usec
Bytes Sent:  218, Time:  82.8254 usec
Bytes Sent:  218, Time:  13.0963 usec
Bytes Sent:  227, Time:  30.7941 usec
Bytes Sent:  218, Time:  27.9624 usec
Bytes Sent:  216, Time:   2.1237 usec
Bytes Sent:  218, Time:  12.3884 usec
Bytes Sent:  227, Time:  31.1480 usec
Bytes Sent:  227, Time:  88.4887 usec
Bytes Sent:  218, Time:  93.0901 usec
Bytes Sent:  218, Time:   7.7870 usec
Bytes Sent:  218, Time:  28.3164 usec
Bytes Sent:  227, Time:  89.5505 usec
Bytes Sent:  218, Time:  84.2412 usec
Bytes Sent:  218, Time:  13.8042 usec
Bytes Sent:  227, Time:  99.4612 usec
Bytes Sent:  218, Time:  86.0110 usec
Bytes Sent:  218, Time:  12.3884 usec
Bytes Sent:  218, Time:  87.7807 usec
Bytes Sent:  216, Time:   3.5395 usec
Bytes Sent:  218, Time:   4.6014 usec
Bytes Sent:  218, Time:  36.1034 usec
Bytes Sent:  218, Time:  14.8661 usec
Bytes Sent:  218, Time:  24.0689 usec
Bytes Sent:  218, Time:  18.0517 usec
Bytes Sent:  227, Time:  24.4229 usec

Does anybody know why this can happen? Why for one message it takes 3 usec to be sent and for other 80 usec?
And is there any way to fix this?

Note: the main goal I want to archive is to send each message as fast as possible. I don't need anychronious sockets, At least until they work faster.

Some additional details regarding what I do:

C++, Visual Studio 2013

How I open:

...
hints.ai_family = AF_UNSPEC;
hints.ai_socktype = SOCK_STREAM;
hints.ai_protocol = IPPROTO_TCP;
...
ConnectSocket = socket(ptr->ai_family, ptr->ai_socktype, ptr->ai_protocol);
...

How I send and calculate time:

...
LARGE_INTEGER cT;
QueryPerformanceCounter(&cT);
long long dT = cT.QuadPart;

iBytesSent = send(ConnectSocket, msgFinal, msgFinalLen, 0);

QueryPerformanceCounter(&cT);
dT = cT.QuadPart - dT;
...

Also I'm listening this socket from other thread, I don't know if this can impact sending or not:

iResult = recv(ConnectSocket, recvbuf, DEFAULT_BUFLEN, 0);

Answer 1

Your methodology is invalid. You're only measuring how long it takes to put the data into the send buffer. If there's room for it, there is no network operation at all. If there isn't room, you block until there is room, which depends on the receiver reading what's already there. So what you're seeing is that sometimes there is room and sometimes there isn't, depending on whether the receiver is reading and keeping up.

If you want to measure round-trip time you need to send a timestamp and have the peer echo it back, then when you receive it compare it to the current time.

Answer 2

You are not measuring the time it takes for the message to be "sent", you are measuring the time it takes for the message to make it into the TCP send buffer. That could involve memory allocation, lock contention and many other things. It can also include another process being scheduled, leading to you losing your time slice.

Answer 3

What you are measuring is the amount of time send call takes. It is basically a write (I/O) operation into a buffer at socket layer.

Your process attempts i/o and gets blocked - once the i/o finishes the daemon is woken up. The time diff you see for send calls includes:

i. Actual write time.

ii. interruptible sleep time - since the scheduler will not wake up your process immediately after the I/O is done. There may be another process that may be woken up.

Tweaks:

i. Try to optimize the send/receive window size. This is the amount of data that can be sent without waiting for an ACK.

Visit: Setting TCP receive window in C and working with tcpdump in Linux

ii. The buffers you pass on to the send call must fit the window size. So tcp doesn't wait for an OPTIMUM size to be reached before actually flushing the data on the network.

iii. A flag similar to TCP_NODELAY for your OS implementation will help.

iv. Readjust the nice value for your daemon so it is woken up as soon as the blocking I/O call is done.