Why is non-blocking socket connect so slow?

Question

when I do 100 non-block socket connection in 1 thread,it is very slow(the number of connection increased one by one),but if I do a blocking socket connection in 100 parallel threads(one connect per thread), it is very fast(get done immediately )

sock = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP);
if (fcntl(sock, F_SETFL,O_NONBLOCK)!=0)
{
 perror("fcntl nonblock");
 return -1;
}

if (setsockopt(sock, SOL_SOCKET, SO_REUSEADDR,&reuseAddr, sizeof(reuseAddr))!=0)
{
  perror("reuse addr");
  return -1;
}

sAddr.sin_addr.s_addr = inet_addr(SRV_ADDR);
sAddr.sin_port = htons(1972);

if ((n=connect(sock, (const struct sockaddr *) &sAddr, sizeof(sAddr))) < 0) 
{
  if (errno !=EINPROGRESS) {
      perror("client connect error");
      return -1;
  }     
}
else if (n>=0)
{
  printf("#%d connected\n",sock);
}

return sock;

Answer 1

Awesome question :-). Here's why I think this is happening. The standard says this:

If the connection cannot be established immediately and O_NONBLOCK is set for the file descriptor for the socket, connect() shall fail and set errno to [EINPROGRESS]

The question of course is what "immediately" means. I believe that "immediately" is actually some small time that allows the SYN, SYN-ACK, ACK to happen. If it didn't wait at all, it would have 0-chance of actually succeeding.

So basically:

• The client sends a SYN
• Waits (blocks) for a small period of time ("immediately") for a SYN-ACK.
• Completes the connection

In doing so it returns successfully instead of EADDRINUSE.

Now, when using threads, each thread does this so nobody waits. They all just connect(2) and context switching allows everybody to do it almost simultaneously.