Python: How to transfer varrying length arrays over a network connection

Question

I need to transfer an array of varying length in which each element is a tuple of two integers. As an example:

path = [(1,1),(1,2)]
path = [(1,1),(1,2),(2,2)]

I am trying to use pack and unpack, however, since the array is of varying length I don't know how to create a format such that both know the format. I was trying to turn it into a single string with delimiters, such as:

msg = 1&1~1&2~
sendMsg = pack("s",msg) or sendMsg = pack("s",str(msg))

on the receiving side:

path = unpack("s",msg)

but that just prints 1 in this case. I was also trying to send 4 integers as well, which send and receive fine, so long as I don't include the extra string representing the path.

sendMsg = pack("hhhh",p.direction[0],p.direction[1],p.id,p.health)

on the receive side:

x,y,id,health = unpack("hhhh",msg)

The first was for illustration as I was trying to send the format "hhhhs", but either way the path doesn't come through properly.

Thank-you for your help. I will also be looking at sending a 2D array of ints, but I can't seem to figure out how to send these more 'complex' structures across the network.

Thank-you for your help.

Answer 1

While you can use pack and unpack, I'd recommend using something like YAML or JSON to transfer your data.

• Pack and unpack can lead to difficult to debug errors and incompatibilities if you change your interface and have different versions trying to communicate with each other.
• Pickle can give security problems and the pickle format might change between Python versions.

JSON is included in the standard Python distribution since 2.6. For YAML there is PyYAML.

Answer 2

You want some sort of serialization protocol. twisted.spread provides one such (see the Banana specification or Perspective Broker documentation). JSON or protocol buffers would be more verbose examples.

See also Comparison of data serialization formats.