Packet Sockets not receiving data for custom protocol ID

Question

I am trying to send and receive packets of type SOCK_RAW over PF_SOCKETs using my own custom protocol ID on the same machine. Here is my sender and receiver sample code-

sender.c

#include<sys/socket.h>
#include<linux/if_packet.h>
#include<linux/if_ether.h>
#include<linux/if_arp.h>
#include<stdio.h>
#include<stdlib.h>
#include<string.h>

#define CUSTOM_PROTO 0xB588

int main ()
{
    int sockfd = -1;
    struct sockaddr_ll dest_addr = {0}, src_addr={0};
    char *buffer = NULL;
    struct ethhdr *eh;

    sockfd = socket(PF_PACKET, SOCK_RAW, htons(CUSTOM_PROTO) );

    if ( sockfd == -1 )
    {
        perror("socket");
        return -1;
    }
    buffer = malloc(1518);
    eh = (struct ethhdr *)buffer;

    dest_addr.sll_ifindex  = if_nametoindex("eth0");
    dest_addr.sll_addr[0]  = 0x0;
    dest_addr.sll_addr[1]  = 0xc;
    dest_addr.sll_addr[2]  = 0x29;
    dest_addr.sll_addr[3]  = 0x49;
    dest_addr.sll_addr[4]  = 0x3f;
    dest_addr.sll_addr[5]  = 0x5b;
    dest_addr.sll_addr[6]  = 0x0;
    dest_addr.sll_addr[7]  = 0x0;

    //other host MAC address
    unsigned char dest_mac[6] = {0x0, 0xc, 0x29, 0x49, 0x3f, 0x5b};

    /*set the frame header*/
    memcpy((void*)buffer, (void*)dest_mac, ETH_ALEN);
    memcpy((void*)(buffer+ETH_ALEN), (void*)dest_mac, ETH_ALEN);

    eh->h_proto = htons(PAVAN_PROTO);

    memcpy((void*)(buffer+ETH_ALEN+ETH_ALEN + 2), "Pavan", 6 );

    int send = sendto(sockfd, buffer, 1514, 0, (struct sockaddr*)&dest_addr,
                      sizeof(dest_addr) );
    if ( send == -1 )
    {
        perror("sendto");
        return -1;
    }
    return 0;
}

receiver.c

#include<sys/socket.h>
#include<linux/if_packet.h>
#include<linux/if_ether.h>
#include<linux/if_arp.h>
#include<stdio.h>
#include<stdlib.h>
#include<string.h>

#define CUSTOM_PROTO 0xB588

int main ()
{
    int sockfd = -1;
    struct sockaddr_ll dest_addr = {0}, src_addr={0};
    char *recvbuf = malloc(1514);

    sockfd = socket(PF_PACKET, SOCK_RAW, htons(CUSTOM_PROTO) );

    if ( sockfd == -1 )
    {
        perror("socket");
        return -1;
    }
    int len = recvfrom(sockfd, recvbuf, 1514, 0, NULL, NULL);
    printf("I received: \n");

    return 0;
}

Both sender and receiver are running on Ubuntu Virtualbox. The problem is the receiver hangs in recvfrom. But in receiver.c, if I change htons(CUSTOM_PROTO) to htons(ETH_P_ALL), the receiver works just fine.

Why is the kernel not delivering the packet with my custom protocol ID to my custom protocol ID socket?

I verified in GDB that the ethernet header is formed correctly when I receive packet with htons(ETH_P_ALL)

Update: Instead of interface eth0 and its corresponding MAC, if I choose local loopback lo and a MAC address of 00:00:00:00:00:00, CUSTOM_PROTO works just fine!

Update 2 CUSTOM_PROTO works fine if the sender and receiver are on different machines. This finding and prev update made me suspect that packets being sent out on eth0 are not being received by the same machine. But the fact that ETH_P_ALL works on the same machine, refutes my suspicion.

Answer 1

ETH_P_ALL vs any other protocol

The protocol ETH_P_ALL has the special role of capturing outgoing packets.

Receiver socket with any protocol that is not equal to ETH_P_ALL receives packets of that protocol that come from the device driver.

Socket with protocol ETH_P_ALL receives all packets before sending outgoing packets to the device driver and all incoming packets that are received from the device driver.

Loopback device vs Ethernet device

Packets sent to the loopback device go out from that device and then the same packets are received from the device as incoming. So, when CUSTOM_PROTO is used with loopback the socket captures packets with custom protocol as incoming.

Note that if ETH_P_ALL is used with the loopback device each packet is received twice. Once it is captured as outgoing and the second time as incoming.

In case of eth0 the packet is transmitted from the device. So, such packets go to the device driver and then they can be seen on the other side of the physical Ethernet port. For example, with VirtualBox "Host-only" network adapter those packets can be captured by some sniffer in the host system.

However, packets transmitted to the physical port (or its emulation) are not redirected back to that port. So, they are not received as incoming from the device. That is why such packets can be captured only by ETH_P_ALL in outgoing direction and they cannot be seen by CUSTOM_PROTO in incoming direction.

Technically it should possible to prepare special setup to do external packet loopback (packets from the device port should be sent back to that port). In that case the behavior should be similar to the loopback device.

Kernel implementation

See the kernel file net/core/dev.c. There are two different lists:

struct list_head ptype_base[PTYPE_HASH_SIZE] __read_mostly;
struct list_head ptype_all __read_mostly;   /* Taps */

The list ptype_all is for socket handlers with protocol ETH_P_ALL. The list ptype_base is for handlers with normal protocols.

There is a hook for outgoing packets in xmit_one() called from dev_hard_start_xmit():

    if (!list_empty(&ptype_all))
        dev_queue_xmit_nit(skb, dev);

For outgoing packets the function dev_queue_xmit_nit() is called for ETH_P_ALL processing each item of ptype_all. Finally the sockets of type AF_SOCKET with protocol ETH_P_ALL capture that outgoing packet.

---

So, the observed behavior is not related to any custom protocol. The same behavior can be observed with ETH_P_IP. In that case the receiver is able to capture all incoming IP packets, however it cannot capture IP packets from sender.c that sends from "eth0" to MAC address of "eth0" device.

It can be also seen by tcpdump. The packets sent by the sender are not captured if tcpdump is called with an option to capture only incoming packets (different versions of tcpdump use different command line argument to enable such filtering).

---

The initial task where on the same machines it is needed to distinguish packets by protocol IDs can be solved using ETH_P_ALL. The receiver should capture all packets and check the protocol, for example:

while (1) {
    int len = recvfrom(sockfd, recvbuf, 1514, 0, NULL, NULL);

    if (ntohs(*(uint16_t*)(recvbuf + ETH_ALEN + ETH_ALEN)) == CUSTOM_PROTO) {
        printf("I received: \n");
        break;
    }
}

---

Useful reference "kernel_flow" with a nice diagram http://www.linuxfoundation.org/images/1/1c/Network_data_flow_through_kernel.png

It is based on the 2.6.20 kernel, however in the modern kernels ETH_P_ALL is treated in the same way.