Mixing Python and Go

Question

I have been working on a library in Python and I would like to do some performance improvement.

Is it possible to write some code in Python and some code in Go, and pass data between them? And if it's possible, are there any examples on how to do this?

Like such:

# Python
def python_foo():
    data = {'foo': 'val', 'bar': [1, 2, 3]}
    go_process(json.dumps(data))


def python_got_data_from_go(data):
    # deal with data from Go


# Go
func go_process(json string) {
    // do some processing
    python_got_data_from_go(someData)
}

Answer 1

You need a glue between them, for example C programming language or communication over network. Most painful solution if you mix https://docs.python.org/2/extending/extending.html with http://golang.org/cmd/cgo/ and good programming skills in C.

You might create server in python http://pymotw.com/2/socket/tcp.html and in go https://coderwall.com/p/wohavg and communicate between them.

Edit: see Writing a Python extension in Go (Golang) and Calling Python function from Go and getting the function return value

Answer 2

The easiest way to mix languages is to to communicate over a socket, such as TCP/IP or Unix domain socket or through a high level protocol like HTTP or XML-RPC. This comes with very high overhead though due to the request processing and serialization/deserialization to/from JSON/XML, which can be significant if you have to make lots of calls. Communicating over socket is usually best if the amount of workload per request is high.

If you are not willing to pay for the overhead of socket (say if you make thousands of requests back and forth between python and go per seconds), there are other solutions that may have lower overhead. You may be able to use shared memory in OSes that have them. Shared memory usually incurs much cheaper cost to access data, but it may incur boxing/unboxing cost from the shared memory structure to Python datatypes. Also, note that you may have to manage the locks yourself with this.

Unless you only make very small number of calls and they do not need to share state between calls I would not recommend communicating using the standard stdin/stdout.

The last alternative is to write Python Extension; I would not recommend this for the faint of heart.