Timestamp outgoing packets

Question

I'm trying to get accurate timestamps for outgoing packets (sent using raw sockets). According to Linux/Documentation/networking/timestamping.txt, "For send time stamps the outgoing packet is looped back to the socket's error queue with the send time stamp(s) attached. It can be received with recvmsg(flags=MSG_ERRQUEUE).".

Unfortunately, recvmsg is always returning -1 when called on a raw socket (created with socket(PF_INET, SOCK_RAW, IPPROTO_RAW) and with SO_TIMESTAMP set to 1 with setsockopt). What am I doing wrong? Is there a better way of getting an accurate timestamp for an outgoing packet?

Addendum (information):

I also tried getting the timestamp from a packet sent through an UDP socket (source code below) and recvmsg returns -1: the error is "Resource temporarily unavailable" (EAGAIN).

Addendum (source code):

#include <arpa/inet.h>
#include <linux/net_tstamp.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/types.h>
#include <sys/socket.h>

void die(char* s)
{
    perror(s);
    exit(1);
}

int main(int argc, char* argv[])
{
    char* destination_ip = "10.0.0.1";
    int destination_port = 1234;

    int sock;
    if ((sock = socket(AF_INET, SOCK_DGRAM, IPPROTO_UDP)) < 0) {
        die("socket()");
    }

    int timestamp_flags = SOF_TIMESTAMPING_TX_SOFTWARE;
    if (setsockopt(sock, SOL_SOCKET, SO_TIMESTAMPING, &timestamp_flags, sizeof(timestamp_flags)) < 0) {
        die("setsockopt()");
    }

    struct sockaddr_in si_server;
    memset(&si_server, 0, sizeof(si_server));
    si_server.sin_family = AF_INET;
    si_server.sin_port = htons(destination_port);
    if (inet_aton(destination_ip, &si_server.sin_addr) == 0) {
        die("inet_aton()");
    }

    const int buffer_len = 256;
    char buffer[buffer_len];

    const int n_packets = 10;
    for (int i = 0; i < n_packets; ++i) {
        sprintf(buffer, "Packet %d", i);
        if (sendto(sock, buffer, buffer_len, 0, (const sockaddr*) &si_server, sizeof(si_server)) < 0) {
            die("sendto()");
        }

        // Obtain the sent packet timestamp.
        char data[256];
        struct msghdr msg;
        struct iovec entry;
        struct sockaddr_in from_addr;
        struct {
            struct cmsghdr cm;
            char control[512];
        } control;
        int res;

        memset(&msg, 0, sizeof(msg));
        msg.msg_iov = &entry;
        msg.msg_iovlen = 1;
        entry.iov_base = data;
        entry.iov_len = sizeof(data);
        msg.msg_name = (caddr_t)&from_addr;
        msg.msg_namelen = sizeof(from_addr);
        msg.msg_control = &control;
        msg.msg_controllen = sizeof(control);        
        if (recvmsg(sock, &msg, MSG_ERRQUEUE) < 0) {
            die("recvmsg()");
        }
    }
    return 0;
}

Answer 1

Looking into the Linux kernel source code, I found that the function responsible for putting the message containing the timestamp of the packet on the error queue is skb_tx_timestamp. This function is supposed to be called by the NIC driver and unfortunately, the e1000 driver doesn't call it (there's a similar function for hardware timestamping, but this is obviously dependent on the NIC driver supporting it).

According to this NetDev discussion from last September, "no driver calls skb_tx_timestamp()" and "You'll need to tweak your NIC driver to play with this TX timestamps". After adding a call to skb_tx_timestamp to e1000_xmit_frame on e1000_main.c, I was able to obtain timestamps for outgoing packets (through an UDP socket). I wasn't able to obtain timestamps for outgoing packets on a RAW socket, though (I still get EAGAIN).

Answer 2

It's hard to know what you are doing wrong, since we cannot see your code.

However: The documentation says that SO_TIMESTAMP is for incoming packets, while SO_TIMESTAMPING is for outgoing packets.

The kernel documentation contains a full example which you could use as a base - though it's using UDP, but you should be able to adjust it to using a RAW socket. See the linux kernel Documentation/networking/timestamping/timestamping.c

EDIT: It seems transmit timestamping isn't universally supported, see e.g. here. Even today just a handful of nic drivers implement software support, and a few have hardware support.

Answer 3

sock_tx_timestamp is only called for SOCK_DGRAM sockets in current kernel code.

BTW, the document Documentation/networking/timestamping/timestamping.c isn't very accurate.

SO_TIMESTAMP / SO_TIMESTAMPNS / SO_TIMESTAMPING / SIOCGSTAMP / SIOCGSTAMPNS are similar. Anyone of them will enable application getting the timestamp of a received packet.

With SOF_TIMESTAMPING_TX_SOFTWARE, any one of the above flags will also provide the application a CMSG in MSG_ERRQUEUE, indicating the timestamp of a sent packet.

But SOF_TIMESTAMPING_RX_SOFTWARE is useless at all. It can not even be used to disable the reporting of timestamp of received packets.