IP_PKTINFO socket option not working

Question

I have been banging my head on this one for a few weeks now and am finally submitting to the fact that I just can't figure it out. I have also been working with networking engineers on my team to no avail. My problem is as follows:

I am working on an application that does pretty straight forward UDP group joins on multiple vlans (each vlan is exposed as its own virtual interface, the NIC in this case is a SolarFlare if that is relevant). All of these joins happen on a single socket (where the messages are de-duplicated based on payload sequence numbers). Prior to doing the IP_ADD_MEMBERSHIP I am setting socket options like this:

setsockopt(sock, SOL_SOCKET, SO_TIMESTAMP, &yes, sizeof yes)
setsockopt(sock, IPPROTO_IP, IP_PKTINFO, &yes, sizeof(yes))
setsockopt(sock, IPPROTO_IP, PACKET_AUXDATA, &yes, sizeof(yes))

I need to get at either the interface index via IP_PKTINFO or the vlan id via PACKET_AUXDATA in order to gather statistics downstream. Now, everything initializes without error and I am able to process UDP payloads without issue. Where I run into trouble is when I attempt to access the ancillary / control messages requested above as demonstrated with the simple debug logging:

for (cmsgptr = CMSG_FIRSTHDR(&msg);
    cmsgptr != NULL;
    cmsgptr = CMSG_NXTHDR(&msg, cmsgptr)) {
    printf("Control Message: cmsg_level: %d, cmsg_type %d\n", cmsgptr->cmsg_level, cmsgptr->cmsg_type);
}

For every packet received, this only outputs:

Control Message: cmsg_level: 1, cmsg_type 29

For reference, SOL_SOCKET=1 and SO_TIMESTAMP=29. So, although I am requesting 3 different control message types, only the timestamping is being populated. This behavior is independent of whether I am joining a single UDP group on a single interface or multiple groups on multiple interfaces.

One solution would be to rewrite the application to put each interface on its own socket, and then funnel everything into a queue, but in my experience the context switching kills the performance of the app. According to the manual page ip(7) IP_PKTINFO has been available since Linux kernel 2.2. I am running Ubuntu 14.04.4 which uses kernel 3.13.0-24-generic.

Any help, insight or direction would be greatly appreciated!

Answer 1

Shot in the dark guesses

1) You need to reset yes back to 1 after each successful call to setsockopt. The docs imply this isn't needed, but it's what I would do.

int yes = 1;
setsockopt(sock, SOL_SOCKET, SO_TIMESTAMP, &yes, sizeof(yes));

yes = 1;
setsockopt(sock, IPPROTO_IP, IP_PKTINFO, &yes, sizeof(yes));

yes = 1;
setsockopt(sock, IPPROTO_IP, PACKET_AUXDATA, &yes, sizeof(yes));

2) Are you checking the return value of setsockopt to see if those calls are correctly succeeding. It's not clear if you've validated they are returning "0" for success or "-1" for error. You should print out the return code for each call.

3) You aren't showing your recvmsg code which is what you presumably using to get the extra info. But it's possible the struct msghdr is not initialized correctly. Specifically is your buffer big enough to get all the control data? Here's how I do it in my code:

struct iovec vec;
ssize_t ret;

const size_t CONTROL_DATA_SIZE = 1000;  // THIS NEEDS TO BE BIG ENOUGH.
char controldata[CONTROL_DATA_SIZE]; 
struct msghdr hdr = {};
sockaddr_storage addrRemote = {};

vec.iov_base = buf;
vec.iov_len = len;

hdr.msg_name = &addrRemote;
hdr.msg_namelen = sizeof(addrRemote);
hdr.msg_iov = &vec;
hdr.msg_iovlen = 1;
hdr.msg_control = controldata;
hdr.msg_controllen = CONTROL_DATA_SIZE;

ret = ::recvmsg(sockfd, &hdr, flags);