How to create a service that sends/receives UDP broadcasts on multiple interfaces

Question

I need to re-create a service on linux that used to run on an embedded system running the LwIP stack (light-weight IP).

The service is using UDP broadcasts to INADDR_BROADCAST (255.255.255.255) to find and configure devices on the same physical subnet. It sends a "scan" and all devices that run this service reply with their full network setup (all NICs, all their MACs and IPs). Then a user gets a list of those devices and can change the IP setup (using the already existing protocol).
[yes, I know people use DHCP for this, but we're talking about the industrial sector here and the protocol/service already exists, so I have no choice but implement something compatible]

Since the device has several NICs, I need to be able to receive this broadcast, know which NIC received it and send a reply via that NIC. Also the service is configurable so that it doesn't open a socket on specific NICs.

The LwIP stack is not as sophisticated as the Linux stack, so a socket bound to an IP still receives all packets to INADDR_BROADCAST. Thus it was pretty straight forward to implement this.

On Linux, I figured I have several options to do this:

• open individual sockets for each NIC with SO_BROADCAST and SO_BINDTODEVICE, so I can bind() them to INADDR_ANY and receive broadcasts. When I send a reply via that socket, Linux routing is ignored and it is sent via the desired NIC.
  BUT: I'd like the service to not run as root...
• Have a single INADDR_ANY bound socket (possibly with IP_PKTINFO to easily know which NIC the packet arrived on), have one socket per NIC, bound to a valid address, with SO_BROADCAST and send replies via those. If I go this way I would like to make sure the sending sockets can never receive anything (because I never call recv() on them. Resource starvation?).
  Maybe SO_RCVBUFSIZE = 0 would be enough?

What would be the correct way to implement this?

Answer 1

You could install the binary with CAP_NET_RAW (and CAP_NET_BIND_SERVICE if ports ≤ 1024 are used); setcap 'cap_net_raw=ep' yourdaemon as root. For IP, SO_BROADCAST does not require any capabilities (in particular, CAP_NET_BROADCAST is not used for IP).

(For the exact capabilities needed, see e.g. net/core/sock.c:sock_setbindtodevice(), net/core/sock.c:sock_setsockopt(), and include/net/sock.h:sock_set_flag() in the Linux kernel sources for verification.)

However, daemons are typically started as root. Here, the above would not suffice, as changing the user ID for the process (to drop privileges) also clears the effective capabilities. Yet, I too prefer my services to run with limited privileges.

I would choose between two basic approaches:

1. Require that the daemon is executed by root, or with CAP_NET_RAW (and optionally CAP_NET_BIND_SERVICE) capabilities.

   Use prctl(), setgroups() or initgroups(), setresuid(), setresgid(), and from libcap, cap_init(), cap_set_flag(), and cap_set_proc() to drop privileges by switching to a dedicated user and group, but retaining the CAP_NET_RAW (and optionally CAP_NET_BIND_SERVICE) capabilities and them only.

   This allows the daemon to respond to e.g. HUP signal without completely restarting, as it has the necessary privileges to enumerate interfaces and read its own configuration files to open sockets for new interfaces.

2. Use a privileged "loader", that opens all the necessary sockets, drops privileges, and executes the actual daemon.

   The daemon should get the socket and interface details as command-line parameters, or perhaps via standard input. The daemon is completely unprivileged.

   Unfortunately, if new interfaces are opened, or the configuration is changed, the daemon cannot do much except exit. (It cannot even execute the privileged loader, because privileges have already been dropped.)

The first approach is more common, and easier to implement in practice; especially if the daemon is only supposed to be executed by root. (Remember, the daemon can respond to configuration changes, since it has the necessary capabilities but not root privileges in general.) I have only used the second approach for "black box" binaries I do not trust.

---

Here is some example code.

privileges.h: #ifndef PRIVILEGES_H #define PRIVILEGES_H

#define   NEED_CAP_NET_ADMIN          (1U << 0)
#define   NEED_CAP_NET_BIND_SERVICE   (1U << 1)
#define   NEED_CAP_NET_RAW            (1U << 2)

extern int drop_privileges(const char *const user, const unsigned int capabilities);

#endif /* PRIVILEGES_H */

privileges.c:

#define _GNU_SOURCE
#define _BSD_SOURCE
#include <stdlib.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/capability.h>
#include <sys/prctl.h>
#include <errno.h>
#include <pwd.h>
#include <grp.h>
#include "privileges.h"

/* Only three NEED_CAP_ constants defined. */
#define MAX_CAPABILITIES 3

static int permit_effective(cap_t caps, const unsigned int capabilities)
{
    cap_value_t  value[MAX_CAPABILITIES];
    int          values = 0;

    if (capabilities & NEED_CAP_NET_ADMIN)
        value[values++] = CAP_NET_ADMIN;

    if (capabilities & NEED_CAP_NET_BIND_SERVICE)
        value[values++] = CAP_NET_BIND_SERVICE;

    if (capabilities & NEED_CAP_NET_RAW)
        value[values++] = CAP_NET_RAW;

    if (values < 1)
        return 0;

    if (cap_set_flag(caps, CAP_PERMITTED, values, value, CAP_SET) == -1)
        return errno;
    if (cap_set_flag(caps, CAP_EFFECTIVE, values, value, CAP_SET) == -1)
        return errno;

    return 0;
}

static int add_privileges(cap_t caps)
{
    cap_value_t  value[3] = { CAP_SETPCAP, CAP_SETUID, CAP_SETGID };

    if (cap_set_flag(caps, CAP_PERMITTED, sizeof value / sizeof value[0], value, CAP_SET) == -1)
        return errno;

    if (cap_set_flag(caps, CAP_EFFECTIVE, sizeof value / sizeof value[0], value, CAP_SET) == -1)
        return errno;

    return 0;
}

int drop_privileges(const char *const user, const unsigned int capabilities)
{
    uid_t uid;
    gid_t gid;
    cap_t caps;

    /* Make sure user is neither NULL nor empty. */
    if (!user || !user[0])
        return errno = EINVAL;

    /* Find the user. */
    {
        struct passwd *pw;

        pw = getpwnam(user);
        if (!pw
#ifdef UID_MIN
            || pw->pw_uid < (uid_t)UID_MIN
#endif
#ifdef UID_MAX
            || pw->pw_uid > (uid_t)UID_MAX
#endif
#ifdef GID_MIN
            || pw->pw_gid < (gid_t)GID_MIN
#endif
#ifdef GID_MAX
            || pw->pw_gid > (gid_t)GID_MAX
#endif
               )
            return errno = EINVAL;

        uid = pw->pw_uid;
        gid = pw->pw_gid;

        endpwent();
    }

    /* Install privileged capabilities. */
    caps = cap_init();
    if (!caps)
        return errno = ENOMEM;
    if (permit_effective(caps, capabilities)) {
        const int cause = errno;
        cap_free(caps);
        return errno = cause;
    }
    if (add_privileges(caps)) {
        const int cause = errno;
        cap_free(caps);
        return errno = cause;
    }
    if (cap_set_proc(caps) == -1) {
        const int cause = errno;
        cap_free(caps);
        return errno = cause;
    }
    cap_free(caps);

    /* Retain permitted capabilities over the identity change. */
    prctl(PR_SET_KEEPCAPS, 1UL, 0UL,0UL,0UL);

    if (setresgid(gid, gid, gid) == -1)
        return errno = EPERM;

    if (initgroups(user, gid) == -1)
        return errno = EPERM;

    if (setresuid(uid, uid, uid) == -1)
        return errno = EPERM;

    /* Install unprivileged capabilities. */
    caps = cap_init();
    if (!caps)
        return errno = ENOMEM;
    if (permit_effective(caps, capabilities)) {
        const int cause = errno;
        cap_free(caps);
        return errno = cause;
    }
    if (cap_set_proc(caps) == -1) {
        const int cause = errno;
        cap_free(caps);
        return errno = cause;
    }
    cap_free(caps);

    /* Reset standard KEEPCAPS behaviour. */
    prctl(PR_SET_KEEPCAPS, 0UL, 0UL,0UL,0UL);

    /* Done. */
    return 0;
}

udp-broadcast.h:

#ifndef   UDP_BROADCAST_H
#define   UDP_BROADCAST_H
#include <stdlib.h>
#include <sys/socket.h>
#include <netinet/in.h>

struct udp_socket {
    struct sockaddr_in  broadcast;  /* Broadcast address */
    unsigned int        if_index;   /* Interface index */
    int                 descriptor; /* Socket descriptor */
};

extern int open_udp_broadcast(struct udp_socket *const udpsocket,
                              const char        *const interface,
                              int                const port);

extern int udp_broadcast(const struct udp_socket *const udpsocket,
                         const void *const              data,
                         const size_t                   size,
                         const int                      flags);

extern size_t udp_receive(const struct udp_socket *const udpsocket,
                          void *const                    data,
                          const size_t                   size_max,
                          const int                      flags,
                          struct sockaddr_in      *const from_addr,
                          struct sockaddr_in      *const to_addr,
                          struct sockaddr_in      *const hdr_addr,
                          unsigned int            *const if_index);

#endif /* UDP_BROADCAST_H */

udp-broadcast.c:

#include <unistd.h>
#include <string.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <net/if.h>
#include <errno.h>
#include "udp-broadcast.h"


int udp_broadcast(const struct udp_socket *const udpsocket,
                  const void *const              data,
                  const size_t                   size,
                  const int                      flags)
{
    ssize_t  n;

    if (!udpsocket || udpsocket->broadcast.sin_family != AF_INET)
        return errno = EINVAL;

    if (!data || size < 1)
        return 0;

    n = sendto(udpsocket->descriptor, data, size, flags,
               (const struct sockaddr *)&(udpsocket->broadcast),
               sizeof (struct sockaddr_in));

    if (n == (ssize_t)-1)
        return errno;
    if (n == (ssize_t)size)
        return 0;
    return errno = EIO;    
}


size_t udp_receive(const struct udp_socket *const udpsocket,
                   void *const                    data,
                   const size_t                   size_max,
                   const int                      flags,
                   struct sockaddr_in      *const from_addr,
                   struct sockaddr_in      *const to_addr,
                   struct sockaddr_in      *const hdr_addr,
                   unsigned int            *const if_index)
{
    char            ancillary[512];
    struct msghdr   msg;
    struct iovec    iov[1];
    struct cmsghdr *cmsg;
    ssize_t         n;

    if (!data || size_max < 1 || !udpsocket) {
        errno = EINVAL;
        return (size_t)0;
    }

    /* Clear results, just in case. */
    if (from_addr) {
        memset(from_addr, 0, sizeof *from_addr);
        from_addr->sin_family = AF_UNSPEC;
    }
    if (to_addr) {
        memset(to_addr, 0, sizeof *to_addr);
        to_addr->sin_family = AF_UNSPEC;
    }
    if (hdr_addr) {
        memset(hdr_addr, 0, sizeof *hdr_addr);
        hdr_addr->sin_family = AF_UNSPEC;
    }
    if (if_index)
        *if_index = 0U;

    iov[0].iov_base = data;
    iov[0].iov_len  = size_max;

    if (from_addr) {
        msg.msg_name = from_addr;
        msg.msg_namelen = sizeof (struct sockaddr_in);
    } else {
        msg.msg_name = NULL;
        msg.msg_namelen = 0;
    }

    msg.msg_iov = iov;
    msg.msg_iovlen = 1;

    msg.msg_control = ancillary;
    msg.msg_controllen = sizeof ancillary;

    msg.msg_flags = 0;

    n = recvmsg(udpsocket->descriptor, &msg, flags);
    if (n == (ssize_t)-1)
        return (size_t)0; /* errno set by recvmsg(). */
    if (n < (ssize_t)1) {
        errno = EIO;
        return (size_t)0;
    }

    /* Populate data from ancillary message, if requested. */
    if (to_addr || hdr_addr || if_index)
        for (cmsg = CMSG_FIRSTHDR(&msg); cmsg != NULL; cmsg = CMSG_NXTHDR(&msg, cmsg))
            if (cmsg->cmsg_level == IPPROTO_IP && cmsg->cmsg_type == IP_PKTINFO) {
                const struct in_pktinfo *const info = CMSG_DATA(cmsg);
                if (!info)
                    continue;
                if (if_index)
                    *if_index = info->ipi_ifindex;
                if (to_addr) {
                    to_addr->sin_family = AF_INET;
                    to_addr->sin_port = udpsocket->broadcast.sin_port; /* This is a guess. */
                    to_addr->sin_addr = info->ipi_spec_dst;
                }
                if (hdr_addr) {
                    hdr_addr->sin_family = AF_INET;
                    hdr_addr->sin_port = udpsocket->broadcast.sin_port; /* A guess, again. */
                    hdr_addr->sin_addr = info->ipi_addr;
                }
            }

    errno = 0;
    return (size_t)n;
}

int open_udp_broadcast(struct udp_socket *const udpsocket,
                       const char        *const interface,
                       int                const port)
{
    const size_t interface_len = (interface) ? strlen(interface) : 0;
    const int    set_flag = 1;
    int          sockfd;

    if (udpsocket) {
        memset(udpsocket, 0, sizeof *udpsocket);
        udpsocket->broadcast.sin_family = AF_INET;
        udpsocket->broadcast.sin_addr.s_addr = INADDR_BROADCAST;
        if (port >= 1 && port <= 65535)
            udpsocket->broadcast.sin_port = htons(port);
        udpsocket->descriptor = -1;
    }

    if (!udpsocket || interface_len < 1 || port < 1 || port > 65535)
        return errno = EINVAL;

    /* Generic UDP socket. */
    sockfd = socket(AF_INET, SOCK_DGRAM, 0);
    if (sockfd == -1)
        return errno;

    /* Set SO_REUSEADDR if possible. */
    setsockopt(sockfd, SOL_SOCKET, SO_REUSEADDR, &set_flag, sizeof set_flag);

    /* Set IP_FREEBIND if possible. */
    setsockopt(sockfd, IPPROTO_IP, IP_FREEBIND, &set_flag, sizeof set_flag);

    /* We need broadcast capability. */
    if (setsockopt(sockfd, SOL_SOCKET, SO_BROADCAST, &set_flag, sizeof set_flag) == -1) {
        const int real_errno = errno;
        close(sockfd);
        return errno = real_errno;
    }

    /* We want the IP_PKTINFO ancillary messages, to determine target address
     * and interface index. */ 
    if (setsockopt(sockfd, IPPROTO_IP, IP_PKTINFO, &set_flag, sizeof set_flag) == -1) {
        const int real_errno = errno;
        close(sockfd);
        return errno = real_errno;
    }

    /* We bind to the broadcast address. */
    if (bind(sockfd, (const struct sockaddr *)&(udpsocket->broadcast), sizeof udpsocket->broadcast) == -1) {
        const int real_errno = errno;
        close(sockfd);
        return errno = real_errno;
    }

    /* Finally, we bind to the specified interface. */
    if (setsockopt(sockfd, SOL_SOCKET, SO_BINDTODEVICE, interface, interface_len) == -1) {
        const int real_errno = errno;
        close(sockfd);
        return errno = real_errno;
    }

    udpsocket->descriptor = sockfd;

    udpsocket->if_index = if_nametoindex(interface);

    errno = 0;
    return 0;
}

main.c:

#include <stdlib.h>
#include <unistd.h>
#include <string.h>
#include <signal.h>
#include <stdio.h>
#include <netdb.h>
#include <errno.h>
#include "privileges.h"
#include "udp-broadcast.h"

static volatile sig_atomic_t    done_triggered = 0;
static volatile sig_atomic_t    reload_triggered = 0;

static void done_handler(int signum)
{
    __sync_bool_compare_and_swap(&done_triggered, (sig_atomic_t)0, (sig_atomic_t)signum);
}

static void reload_handler(int signum)
{
    __sync_bool_compare_and_swap(&reload_triggered, (sig_atomic_t)0, (sig_atomic_t)signum);
}

static int install_handler(const int signum, void (*handler)(int))
{
    struct sigaction act;
    memset(&act, 0, sizeof act);
    sigemptyset(&act.sa_mask);
    act.sa_handler = handler;
    act.sa_flags = 0;
    if (sigaction(signum, &act, NULL) == -1)
        return errno;
    return 0;
}

/* Return 0 if done_triggered or reload_triggered, nonzero otherwise.
 * Always clears reload_triggered.
*/
static inline int keep_running(void)
{
    if (done_triggered)
        return 0;
    return !__sync_fetch_and_and(&reload_triggered, (sig_atomic_t)0);
}

static const char *ipv4_address(const void *const addr)
{
    static char    buffer[16];
    char          *end = buffer + sizeof buffer;
    unsigned char  byte[4];

    if (!addr)
        return "(none)";

    memcpy(byte, addr, 4);

    *(--end) = '\0';
    do {
        *(--end) = '0' + (byte[3] % 10);
        byte[3] /= 10U;
    } while (byte[3]);
    *(--end) = '.';
    do {
        *(--end) = '0' + (byte[2] % 10);
        byte[2] /= 10U;
    } while (byte[2]);
    *(--end) = '.';
    do {
        *(--end) = '0' + (byte[1] % 10);
        byte[1] /= 10U;
    } while (byte[1]);
    *(--end) = '.';
    do {
        *(--end) = '0' + (byte[0] % 10);
        byte[0] /= 10U;
    } while (byte[0]);

    return (const char *)end;
}

int main(int argc, char *argv[])
{
    int  port;
    char dummy;

    /* Check usage. */
    if (argc != 4 || !strcmp(argv[1], "-h") || !strcmp(argv[1], "--help")) {
        fprintf(stderr, "\n");
        fprintf(stderr, "Usage: %s [ -h | --help ]\n", argv[0]);
        fprintf(stderr, "       %s USERNAME INTERFACE PORT\n", argv[0]);
        fprintf(stderr, "Where:\n");
        fprintf(stderr, "       USERNAME    is the unprivileged user to run as,\n");
        fprintf(stderr, "       INTERFACE   is the interface to bind to, and\n");
        fprintf(stderr, "       PORT        is the UDP/IPv4 port number to use.\n");
        fprintf(stderr, "\n");
        return EXIT_FAILURE;
    }

    /* Parse the port into a number. */
    if (sscanf(argv[3], "%d %c", &port, &dummy) != 1 || port < 1 || port > 65535) {
        struct servent *serv = getservbyname(argv[3], "udp");
        if (serv && serv->s_port > 1 && serv->s_port < 65536) {
            port = serv->s_port;
            endservent();
        } else {
            endservent();
            fprintf(stderr, "%s: Invalid port.\n", argv[3]);
            return EXIT_FAILURE;
        }
    }

    /* Drop privileges. */
    if (drop_privileges(argv[1], NEED_CAP_NET_RAW)) {
        fprintf(stderr, "%s.\n", strerror(errno));
        return EXIT_FAILURE;
    }

    /* Install signal handlers. */
    if (install_handler(SIGINT, done_handler) ||
        install_handler(SIGTERM, done_handler) ||
        install_handler(SIGHUP, reload_handler) ||
        install_handler(SIGUSR1, reload_handler)) {
        fprintf(stderr, "Cannot install signal handlers: %s.\n", strerror(errno));
        return EXIT_FAILURE;
    }

    fprintf(stderr, "Send a SIGINT (Ctrl+C) or SIGTERM to stop the service:\n");
    fprintf(stderr, "\tkill -SIGTERM %ld\n", (long)getpid());
    fprintf(stderr, "Send a SIGHUP or SIGUSR1 to have the service reload and rebroadcast:\n");
    fprintf(stderr, "\tkill -SIGHUP %ld\n", (long)getpid());
    fprintf(stderr, "Privileges dropped successfully.\n\n");
    fflush(stderr);

    while (!done_triggered) {
        struct udp_socket  s;

        if (open_udp_broadcast(&s, argv[2], port)) {
            fprintf(stderr, "%s port %s: %s.\n", argv[2], argv[3], strerror(errno));
            return EXIT_FAILURE;
        }

        if (udp_broadcast(&s, "Hello?", 6, MSG_NOSIGNAL)) {
            fprintf(stderr, "%s port %s: Broadcast failed: %s.\n", argv[2], argv[3], strerror(errno));
            close(s.descriptor);
            return EXIT_FAILURE;
        }

        if (s.if_index)
            fprintf(stderr, "Broadcast sent using interface %s (index %u); waiting for responses.\n", argv[2], s.if_index);
        else
            fprintf(stderr, "Broadcast sent using interface %s; waiting for responses.\n", argv[2]);
        fflush(stderr);

        while (keep_running()) {
            struct sockaddr_in  from_addr, to_addr, hdr_addr;
            unsigned char       data[512];
            unsigned int        if_index;
            size_t              size, i;

            size = udp_receive(&s, data, sizeof data, 0, &from_addr, &to_addr, &hdr_addr, &if_index);
            if (size > 0) {
                printf("Received %zu bytes:", size);
                for (i = 0; i < size; i++)
                    if (i & 15)
                        printf(" %02x", data[i]);
                    else
                        printf("\n\t%02x", data[i]);
                if (if_index)
                    printf("\n\t Index: %u", if_index);
                printf("\n\t  From: %s", ipv4_address(&from_addr.sin_addr));
                printf("\n\t    To: %s", ipv4_address(&to_addr.sin_addr));
                printf("\n\tHeader: %s", ipv4_address(&hdr_addr.sin_addr));
                printf("\n");
                fflush(stdout);
            } else
            if (errno != EINTR) {
                fprintf(stderr, "%s\n", strerror(errno));
                break;
            }
        }

        close(s.descriptor);
    }

    fprintf(stderr, "Exiting.\n");
    return EXIT_SUCCESS;
}

Compile using

gcc -Wall -Wextra -O2 -c privileges.c
gcc -Wall -Wextra -O2 -c udp-broadcast.c
gcc -Wall -Wextra -O2 -c main.c
gcc -Wall -Wextra main.o udp-broadcast.o privileges.o -lcap -o example

and run the example as root, specifying an unprivileged user name to run as, the interface to bind to, and the UDP port number as parameters:

sudo ./example yourdaemonuser eth0 4000

Right now I only have one laptop in use, so receive side is basically untested. I do know that CAP_NET_RAW is sufficient here (Linux kernel 4.2.0-27 on x86-64), and that the UDP broadcast sends show up as outgoing from the ethernet interface address to 255.255.255.255:port, but I don't have another machine to send example responses to the daemon (which would be easy using e.g. NetCat: printf 'Response!' | nc -u4 -q2y interface-address port).

Please note that the code quality above is only initial test grade. Since I don't need this myself for anything, and only wanted to verify I am not talking out of my butt, I have not spent any effort in making the code clean or reliable.

Questions? Comments?