What is the most lightweight way to transmit data over the internet using Python?

Question

I have two computers in geographically dispersed locations, both connected to the internet. On each computer I am running a Python program, and I would like to send and receive data from one to the other. I'd like to use the most simple approach possible, while remaining somewhat secure.

I have considered the following solutions, but I'm not sure which is the simplest:

• HTTP server and client, using protobuf*;
• SOAP web service and client (pywebsvcs maybe?);
• Some sort of IPC over an SSH tunnel -- again, protobuf maybe?

Like I said, I'd like the solution to be somewhat secure, but simplicity is the most important requirement. The data is very simple; object of type A, which contains a list of objects of type B, and some other fields.

*I have used protobuf in the past, so the only difficulty would be setting up the HTTP server, which I guess would be cherrypy.

Answer 1

Protocol buffers are "lightweight" in the sense that they produce very compact wire representation, thus saving bandwidth, memory, storage, etc -- while staying very general-purpose and cross-language. We use them a lot at Google, of course, but it's not clear whether you care about these performance characteristics at all -- you seem to use "lightweight" in a very different sense from this, strictly connected with (mental) load on you, the programmer, and not al all with (computational) load on computers and networks;-).

If you don't care about spending much more bandwidth / memory / etc than you could, and neither do you care about the ability to code the participating subsystems in different languages, then protocol buffers may not be optimal for you.

Neither is pickling, if I read your "somewhat secure" requirement correctly: unpickling a suitably constructed malicious pickled-string can execute arbitrary code on the unpickling machine. In fact, HTTP is not "somewhat secure" in a slightly different sense: there's nothing in that protocol to stop intruders from "sniffing" your traffic (so you should never use HTTP to send confidential payloads, unless maybe you use strong encryption on the payload before sending it and undo that after receiving it). For security (again depending on what meaning you put on the word) you need HTTPS or (simpler to set up, doesn't require you to purchase certificates!-) SSH tunnels.

Once you do have an SSH tunnel established between two machines (for Python, paramiko can help, but even doing it via shell scripts or otherwise by directly controlling the ssh commandline client isn't too bad;-) you can run any protocol on it (HTTP is fine, for example), as the tunnel endpoints are made available as given numbered ports on which you can open socket. I would personally recommend JSON instead of XML for encoding the payloads -- see here for an XMLRPC-like JSON-based RPC server and client, for example -- but I guess that using the XMLRPC server and client that come with Python's standard library is even simpler, thus probably closer to what you're looking for. Why would you want cherrypy in addition? Is performance now suddenly trumping simplicity, for this aspect of the whole architecture only, while in every other case simplicity was picked over performance? That would seem a peculiarly contradictory set of architectural choices!-)