sendto() dgrams do not block for ENOBUFS on OSX

Question

This is more of a observation and also a suggestion for whats the best way to handle this scenario.

I have two threads one just pumps in data and another receives the data and does lot of work before sending it another socket. Both the threads are connected via a Domain socket. The protocol used here is UDP. I did not want to use TCP as it is stream based, which means if there is little space in the queue my data is split and sent. This is bad as Iam sending data that should not be split. Hence I used DGRAM. Interestingly when the send thread overwhelms the recv thread by pumping so much data, at some point the Domain socket buffer gets filled up and sendto() returns ENOBUFS. I was of the opinion that should this happen, sendto() would block until the buffer is available. This would be my desired behaviour. However this does not seem to be the case. I solve this problem in a rather weird way.

• CPU Yield method If I get ENOBUFS, I do a sched_yield(); as there is no pthread_yield() in OSX. After that I try to resend again. If that fails I keep doing the same until it is taken. This is bad as Iam wasting cpu cycles just doing something useless. I would love if sendto() blocked.

• Sleep method I tried to solve the same issue using sleep(1) instead of sched_yield() but this of no use as sleep() would put my process to sleep instead of just that send thread.

Both of them does not seem to work for me and Iam running out of options. Can someone suggest what is the best way to handle this issue? Is there some clever tricks Iam not aware of that can reduce unnecessary cpu cycles? btw, what the man page says about sentto() is wrong, based on this discussion http://lists.freebsd.org/pipermail/freebsd-hackers/2004-January/005385.html

The Upd code in kernel:

The udp_output function in /sys/netinet/udp_usrreq.c, seems clear:

         /*
          * Calculate data length and get a mbuf
          * for UDP and IP headers.
          */
         M_PREPEND(m, sizeof(struct udpiphdr), M_DONTWAIT);
         if (m == 0) {
                 error = ENOBUFS;
                 if (addr)
                         splx(s);
                 goto release;
         }

Answer 1

I'm not sure why sendto() isn't blocking for you... but you might try calling this function before you each call to sendto():

#include <stdio.h>
#include <sys/select.h>

// Won't return until there is space available on the socket for writing
void WaitUntilSocketIsReadyForWrite(int socketFD)
{
   fd_set writeSet;
   FD_ZERO(&writeSet);
   FD_SET(socketFD, &writeSet);
   if (select(socketFD+1, NULL, &writeSet, NULL, NULL) < 0) perror("select");
}

Btw how big are the packets that you are trying to send?