Simplest python network messaging

Question

I have a machine control system in Python that currently looks roughly like this

goal = GoalState()
while True:
    current = get_current_state()
    move_toward_goal(current,goal)

Now, I'm trying to add in the ability to control the machine over the network. The code I want to write would be something like this:

goal = GoalState()
while True:
    if message_over_network():
        goal = new_goal_from_message()
    current = get_current_state()
    move_toward_goal(current,goal)

What would be the simplest and most Pythonic way of adding this sort of networking capability into my application? Sockets could work, thought they don't particularly feel Pythonic. I've looked at XMLRPC and Twisted, but both seemed like they would require major revisions to the code. I also looked at ØMQ, but it felt like I was adding an external dependency that didn't offer anything that I didn't already have with sockets.

I'm not opposed to using any of the systems that I've addressed above, as what I believe to be failings are probably misunderstandings on my part. I'm simply curious as to the idiomatic way of handling this simple, common issue.

Answer 1

The are at least two issues you need to decide on:

1. How to exchange messages?
2. In what format?

Regarding 1. TCP sockets are the lowest level and you would need to deal with low level things like recognizing messages boundaries. Also, TCP connection gives you reliable delivery but only as long as the connection is not reset (due to for example a temporary network failure). If you want your application to gracefully recover when a TCP connection resets, you need to implement some form of messages acknowledgements to keep track what needs to be resend over the new connection. OMQ gives you higher level of abstraction than plain TCP connection. You don't need to deal with a stream of bytes but with whole messages. It still does not give you reliable delivery, messages can get lost, but it gives several communication patterns that can be used to ensure reliable delivery. 0MQ is also highly performant, IMO it is a good choice.

Regarding 2, if interoperability with other languages is not needed, Pickle is a very convenient and Pythonic choice. If interoperability is needed, you can consider JSON, or, if performance is an issue, binary format, such as Google protocol buffers. This last choice would require the most work (you'll need to define messages formats in .idl files) this would definitely not feel Pythonic.

Take a look how exchange of messages (any serializable Python object) over a plain socket can look like:

def send(sockfd, message):
    string_message = cPickle.dumps(message)
    write_int(sockfd, len(string_message))
    write(sockfd, string_message)

def write_int(sockfd, integer):
    integer_buf = struct.pack('>i', integer)       
    write(sockfd, integer_buf)

def write(sockfd, data):
    data_len = len(data)
    offset = 0
    while offset != data_len:
        offset += sockfd.send(data[offset:])

Not bad, but as you can see having to deal with serialization of a message length is quite low level.

And to receive such message:

def receive(self):
    message_size = read_int(self.sockfd)
    if message_size == None:
        return None
    data = read(self.sockfd, message_size)
    if data == None:
        return None
    message = cPickle.loads(data)
    return message

def read_int(sockfd):
    int_size = struct.calcsize('>i')
    intbuf = read(sockfd, int_size)
    if intbuf == None:
        return None
    return struct.unpack('>i', intbuf)[0]

def read(sockfd, size):
    data = ""
    while len(data) != size:
        newdata = sockfd.recv(size - len(data))
        if len(newdata) == 0:
           return None
        data = data + newdata
    return data

But this does not gracefully deal with errors (no attempt to determine which messages were delivered successfully).

Answer 2

If you're familiar with sockets, I would consider SocketServer.UDPServer (see http://docs.python.org/library/socketserver.html#socketserver-udpserver-example). UDP is definitely the simplest messaging system, but, obviously, you'll have to deal with fact that some messages can be lost, duplicated or delivered out of order. If your protocol is very simple, it's relatively easy to handle. The advantage is you don't need any additional threads and no external dependencies are needed. It also might be very good choice if you application doesn't have concept of session.

Might be good for a start, but there are much more details to be considered that are not included in your question. I also wouldn't be worried of the fact, that sockets are not very Pythonic. At the very end you're going to use sockets anyway, someone will just wrap them for you and you'll be forced to learn the framework, which in best case may be overwhelming for your requirements.

(Please note my opinion is highly biased, as I love dealing with raw sockets.)