lists or dicts over zeromq in python

Question

What is the correct/best way to send objects like lists or dicts over zeromq in python? What if we use a PUB/SUB pattern, where the first part of the string would be used as a filter?

• I am aware that there are multipart messages, but they where originally meant for a different purpose. Further you can not subscribe all messages, which have a certain string as the first element.

Answer 1

Manual serialization

You turn the data into a string, concatenate or else, do your stuff. It's fast and doesn't take much space but requires work and maintenance, and it's not flexible.

If another language wants to read the data, you need to code it again. No DRY.

Ok for very small data, but really the amount of work is usually not worth it unless you are looking for speed and memory effiency and that you can measure that your implementation is significantly better.

Pickle

Slow, but you can serialize complex objects, and even callable. It's powerfull, and it's so easy it's a no brainer.

On the other side it's possible to end up with something you can't pickle and break your code. Plus you can't share the data with any lib written in an other language.

Eventually, the format is not human readable (hard do debug) and quite verbose.

Very nice to share objects and tasks, not so nice for messages.

json

Reasonably fast, easy to implement with simple to averagely complex data structures. It's flexible, human readible and data can be shared accross languages easily.

For complex data, you'll have to write a bit of code.

Unless you have a very specific need, this is probably the best balance between features and complexity. Espacially since the last implementation in the Python lib is in C and speed is ok.

xml

Verbose, hard to create and a pain to maintain unless you got some heavy lib that that does all the job for you. Slow.

Unless it's a requirement, I would avoid it.

In the end

Now as usual, speed and space efficiency is relative, and you must first answer the questions:

• what efficiency do I need ?
• what am I ready to pay (money, time, energy) for that ?
• what solution fits in my current system ?

It's all what matters.

That wonderful moment of philosophy passed, use JSON.

Answer 2

JSON:

# Client
socket.send(json.dumps(message))

---

# Server
message = json.loads(socket.recv())

More info:

• JSON encoder and decoder
• hwserver.py
• hwclient.py