Is there a non-blocking method to check for data faster than select() and poll()?

Question

I have a C program sending data as fast as it can using the sendto() method from the sender to the receiver which receives with recvfrom() method. The data is encapsulated into layer 2 Ethernet frames and the application is writing Ethernet frames directly on to the wire (no TCP or UDP or even IP). This is on x86_64 Linux (dev machines are just stock Ubuntu 14.04). I have no intention of porting to any other OS, the application design scope is for Linux so other OSes don't matter.

Sender:

    while (true)
    {
        sendResult = sendto(sockFD, txBuffer, fSize+headersLength, 0,
            (struct sockaddr*)&socket_address, sizeof socket_address);
    }

Receiver:

    while (true)
    {
        recvfrom(sockFD, rxBuffer, fSizeTotal, 0, NULL, NULL);
    }

I want the sender to be able to check for received packets; should the receiver application quit it sends data back to the sender saying "I quit" so the sender will stop sending data. I used poll() on the sender to check for received messages as below but this significantly reduces the transmission speed from just shy of 1Gbps (968Mbps) to around 10Mbps. I'm testing with a cross-over cable between two PCs with 1Gbps NICs. The sender counts sent frames and the frame size and the receiver counts received frames and the frame size, so to confirm, the application is actually receiving at wire-rate near enough, I'm not just look at NIC utilization or similar.

Poll() Method:

int rv;
struct pollfd ufds[1];
ufds[0].fd = sockFD;
ufds[0].events = POLLIN


while (true)
{
    sendResult = sendto(sockFD, txBuffer, fSize+headersLength, 0, (struct sockaddr*)&socket_address, sizeof(socket_address));

    // wait for events on the sockets, 1ms timeout
    rv = poll(ufds, 1, 1);

    if (rv > 0) {
        if (ufds[0].revents & POLLIN) {
            recvfrom(sockFD, rxBuffer, fSizeTotal, 0, NULL, NULL);
        }
    }

}

1 millisecond is the lowest timeout that can be set for the poll() method. This is why my transmission program could only transmit at 10Mbps. The application can easily saturate a 1Gbps link though with minimal CPU usage, I was getting 968Mbps as I previously stated (I don't mean peak by the way, that is sustained throughput).

I removed the poll() call and switched to select() using the below example, but again, using the smallest delay I can here my transmission application could only get 175Mbps. Not close to the original 968Mbps;

Select() Method:

fd_set readfds;
struct timeval tv;
int rv, n;

FD_ZERO(&readfds);
FD_SET(sockFD, &readfds);
n = sockFD + 1;


while (true)
{
    sendResult = sendto(sockFD, txBuffer, fSize+headersLength, 0, (struct sockaddr*)&socket_address, sizeof(socket_address));

    tv.tv_sec = 0;
    tv.tv_usec = 000001;
    rv = select(n, &readfds, NULL, NULL, &tv);
    if (rv > 0) {
        if (FD_ISSET(sockFD, &readfds)) {
            recvfrom(sockFD, rxBuffer, fSizeTotal, 0, NULL, NULL);
    }

}

It seems that both methods are too slow for today’s systems (my cpu usage was around 2% for all tests above). I wish to move this application on to some 10GigE machines soon and start testing there but I obviously can't using either of these two methods. Is there no faster way I can check?

I though these were supposed to be non-blocking but by requiring a timeout they are blocking in a way; I have seen this thread but it isn't the answer I require. Is there no method I can simply call that checks at that moment in time it was called, for data waiting to be read, then returns immediately if no data is waiting to be read?

As a side node, I haven't read up on the recvfrom method() to see where the delay is yet before posting this but I did try the following because it only took 30 seconds to change the code, it resulted in the worst outcome which was less than 1Mbps;

Sender:

    while (true)
    {
        // Continually send a frame then check for a frame, send a frame then check for a frame...
        sendResult = sendto(sockFD, txBuffer, fSize+headersLength, 0, (struct sockaddr*)&socket_address, sizeof(socket_address));
        recvfrom(sockFD, rxBuffer, fSizeTotal, 0, NULL, NULL);
    }

Answer 1

I wouldn't use non-blocking mode at all. Just dedicate a thread in blocking mode. That way you're only doing one system call: recvfrom(), so you're saving context switches into the kernel.

Answer 2

There is no requirement for poll or select to block for any period of time. If the timeout parameters for poll or select are set to zero both calls return immediately with indications of i/o availability. This eliminates engaging the timer and subsequent rounding.

Note, it is not clear to me why this would be substantially faster than a simple non-blocking polling read if you are only monitoring one file descriptor. I would have expected any benefits from this approach to begin to accrue when multiple FDs were in play so it is interesting that your testing uncovered this.

Answer 3

As you figured, the reason your performance suffered is because you limited yourself to sending no more than 1000 packets per second.

If you are willing to use two threads, then EJP's answer is the best option. If you really only want to use a single thread, the best option is to use select() or poll() to let you know if there is something to do at the point when your transmit queue is saturated. So, you can set your socket into non-blocking mode, or you can use the MSG_DONTWAIT flag when doing your I/O. Stop doing that I/O when you get the EAGAIN/EWOULDBLOCK notification, and then do a blocking wait for the appropriate event in select() or poll() again (do not set a timeout). In pseudocode with simplified error handling:

writeable = true;
readable = false;
make_nonblock(s);
for (;;) {
    if (readable) {
        while (recvfrom(s,...) > 0) {
            done = done_check();
        }
        if (done) break;
        assert(errno == EAGAIN);
        readable = false;
    }
    if (writeable) {
        while (sendto(s,...) > 0) {}
        assert(errno == EAGAIN);
        writeable = false;
    }
    poll_socket(s, &readable, &writeable);
}