How to check if message is dropped due to HWM at send in ZeroMQ PUB-SUB pattern

Question

I have implemented a message bus in Linux for IPC using ZeroMQ (more specifically CZMQ). Here is what I have implemented.

My question is, how do I know that send dropped the message when the publisher buffer is full?

In my simple test setup, I am using a publisher-subscriber with a proxy. I have a fast sender and a very slow receiver causing messages to hit HWM and drop on send. My exception is that send would fail with 'message dropped' error, but it is not the case. the zmq_msg_send() is not giving me any error even though the messages get dropped (I can verify this by seeing gaps in messages in subscriber end).

How can I know when the messages get dropped? If this is the intended behaviour and ZeroMQ does not let us know that, what is a workaround to find if my send dropped the message?

Answer 1

What you appear to be asking for is fault tolerance for which PUB/SUB isn't ideal. Not only may the HWM be reached, but consider what happens if a subscribing client dies and gets restarted - it will miss messages sent by the publisher for the duration. FWIW. In ZMQ v2, the default HWM was infinite for PUB/SUB, but got changed to 1000 in v3 because systems were choking for memory due to messages being queued faster than they could be sent. The 1000 seemed like a reasonable value for bursts of messages when the average message rate was within the network bandwidth. YMMV.

If you just want to know when messages get dropped, it's as simple as adding an incrementing message number to the message and having the subscribers monitor that. You could choose to place this number in it's own frame or not; overall simplicity will be the decider. I don't believe it's possible to determine when messages get dropped specifically because the HWM has been reached.