How to add topic filters when calling recv_pyobj() in ZeroMQ?

Question

ZeroMQ provides pretty good documentation about how to set up a pub-sub pattern with the topic filter, as described in the api docs. ZeroMQ also provides the methods socket.send_json() and socket.send_pyobj() (and the recv counterparts) for convenience.

In the pub-sub example, the topic filter (a string) is appended to the beginning of the message (also a string). Is there any way to set up a topic filter when using the built-in serialization? If I am sending a dict or Class using send_pyobj() I cannot append a string in front of it.

Answer 1

First thing to note is that ZeroMQ does not provide send_json() or send_pyobj() as convenience methods, these are provided by the pyzmq binding. So, ZMQ does not know anything about those data types - basically what's happening is that those methods just perform the serialization and unserialization of the the data under the hood.

Like you, I haven't seen a single example of pub/sub working with these convenience methods that didn't just subscribe to ''. But, it should be possible, if a bit of a hack to do so.

As you can see here in the source, send_pyobj() uses pickle to serialize the data. You can use that fact to see what your data looks like once it's serialized. You can add an extra element to your dict or Class, provided you can be certain it goes first in the serialized string, and then look at the serialization and just use the beginning of the string as your subscribe topic. If you can't be certain your topic element will come first, then you'll have to create an envelope of some kind that you have a bit more control over, and send that with your data inside it and just dereference it when you receive it.

Hacky, ugly, and ultimately probably a bad idea, even according to the writers of the pyzmq binding themselves - relevant quote (emphasis added):

we do provide three builtin serialization methods for convenience, to help Python developers learn libzmq... These methods designed for convenience, not for performance, so developers who do want to emphasize performance should use their own serialized send/recv methods.

It's probably far better to just serialize the data yourself and send a proper multiframe message with your topic in the first frame. You can find an example of such here.

// publisher
self.socket.send_multipart([b'status',pickle.dumps(msg2)])

// subscriber
socket.setsockopt(zmq.SUBSCRIBE, 'status')
[topic,msg] = socket.recv_multipart()
msg2 = pickle.loads(msg)
print msg2['game']