Broadcast IPC on Linux

Question

I have the following requirements for an IPC mechanism on Linux:

1. There is a single producer process, but multiple consumer processes. The consumer processes are not children of the producer process. They are brought up independently.

2. The messages being transported are fixed size POD structs.

3. We need to use a fixed amount of memory for this mechanism. A ring buffer like mechanism seems ideal here.

4. The producer needs to run really fast and can never wait for the consumers. Instead it needs to overwrite entries in the fixed size buffer (for IPC) and the consumers need to detect this problem and just catch up with the producer by skipping intermediate messages in case of a wrap around.

5. Consumers can come up and go down any time and there should be no explicit handshake between the single producer and the ephemeral consumers as they come up and go. So when consumers come up they just start reading from the latest message available and go down whenever they want without the producers ever knowing.

I have the following solution right now:

1. I am creating a ring buffer of fixed size entries that is written to by one process and read by many. The ring buffer is built on top of a memory-mapped file in /tmp.

2. The ring buffer code is such that the producers never block waiting for any of the consumers to consume entries. Instead consumers detect when the ring buffer has wrapped around in the middle of reading/processing an entry and just catch up to the latest. I do it with a couple of producer sequences. I can expand on this if needed but doesn't seem very relevant. So producers gallop at full speed and consumers that fall behind detect that mid read corruption and skip to the latest entry.

As of now this works fine. Now I am trying to figure out how to add some signaling to the mix so consumers do not have to spin reading producer sequences waiting for a new message. I have some additional requirements for the signaling part:

1. The signaling needs to be some kind of broadcast mechanism. This follows from the one producer/multiple_consumers requirement. So multiple processes should be able to be woken up when the producers signals them. Given that the producer doesn't know about any of the consumers, it seems like we need some kind of named resource to do this signaling.

2. The signaling mechanism needs to be composable with other regular signals that can be waited on using select/epoll_wait. The consumers that are reading form this IPC mechanism/ring_buffer are waiting on writes to other unrelated pipes/sockets etc and they use a select on these FDs. It would be ideal to be able to just produce an FD from this mechanism that consumers could just add to their select call. Similarly if the consumer is waiting on multiple such ring_buffers, we need to be able to block on all of them and wake up as soon as any of them signal.

With these requirements in mind I eliminated a few options:

1. Condition variables: We can't block on multiple of these from a consumer. They are not selectable either.

2. Named pipes: We would need a named pipe per consumer and this implies some kind of producer/consumer handshake which we want to avoid.

3. eventfd: eventfds are not named so seems like they are only a solution if the signaling is between parent and children processes. My scenario has processes that are started independently.

4. Unix domain socket: There doesn't seem to be any broadcast facility here, so I am not sure if this will work without an explicit socket per consumer. This goes against the no handshake requirement.

I am at a bit of a loss and don't see any other good option. To me it seems like UDP multicast will probably work. The consumers can all be part of a multicast group (create socket with SO_REUSEADDR) and the sole producer can send messages on that group to signal the consumers. But this seems really heavy weight and elaborate. Are there any other good mechanisms to make this happen? I am willing to work directly with the Futex API if it helps as long as it composes with blocking on other unrelated FDs using select/epoll.

Answer 1

I recommend sending a signal, via DBus. Why roll your own IPC, when you can use a mature framework that already does what you want?

This page should get you started:

https://dbus.freedesktop.org/doc/dbus-tutorial.html

At the end, it includes links to the Qt and GLib APIs. I've used neither, instead writing my own Boost.ASIO-based wrapper around the low-level API, though it's a rather involved undertaking.

https://dbus.freedesktop.org/doc/api/html/

BTW, for sending blocks of binary data, you'll want to append an arg of type DBUS_TYPE_ARRAY of DBUS_TYPE_BYTE, to your message. Note that DBUS_TYPE_STRING can't contain zero bytes or invalid unicode sequences.

Update: Another library that's recently come to my attention is called sd-bus. Here's a good overview & tutorial:

http://0pointer.net/blog/the-new-sd-bus-api-of-systemd.html