What's the closest I can get to calling a Python function using a different Python version?

Question

Say I have two files:

# spam.py
import library_Python3_only as l3

def spam(x,y)
    return l3.bar(x).baz(y)

and

# beans.py
import library_Python2_only as l2

...

Now suppose I wish to call spam from within beans. It's not directly possible since both files depend on incompatible Python versions. Of course I can Popen a different python process, but how could I pass in the arguments and retrieve the results without too much stream-parsing pain?

Answer 1

Here is a complete example implementation using subprocess and pickle that I actually tested. Note that you need to use protocol version 2 explicitly for pickling on the Python 3 side (at least for the combo Python 3.5.2 and Python 2.7.3).

# py3bridge.py

import sys
import pickle
import importlib
import io
import traceback
import subprocess

class Py3Wrapper(object):
    def __init__(self, mod_name, func_name):
        self.mod_name = mod_name
        self.func_name = func_name

    def __call__(self, *args, **kwargs):
        p = subprocess.Popen(['python3', '-m', 'py3bridge',
                              self.mod_name, self.func_name],
                              stdin=subprocess.PIPE,
                              stdout=subprocess.PIPE)
        stdout, _ = p.communicate(pickle.dumps((args, kwargs)))
        data = pickle.loads(stdout)
        if data['success']:
            return data['result']
        else:
            raise Exception(data['stacktrace'])

def main():
    try:
        target_module = sys.argv[1]
        target_function = sys.argv[2]
        args, kwargs = pickle.load(sys.stdin.buffer)
        mod = importlib.import_module(target_module)
        func = getattr(mod, target_function)
        result = func(*args, **kwargs)
        data = dict(success=True, result=result)
    except Exception:
        st = io.StringIO()
        traceback.print_exc(file=st)
        data = dict(success=False, stacktrace=st.getvalue())

    pickle.dump(data, sys.stdout.buffer, 2)

if __name__ == '__main__':
    main()

The Python 3 module (using the pathlib module for the showcase)

# spam.py

import pathlib

def listdir(p):
    return [str(c) for c in pathlib.Path(p).iterdir()]

The Python 2 module using spam.listdir

# beans.py

import py3bridge

delegate = py3bridge.Py3Wrapper('spam', 'listdir')
py3result = delegate('.')
print py3result

Answer 2

Assuming the caller is Python3.5+, you have access to a nicer subprocess module. Perhaps you could user subprocess.run, and communicate via pickled Python objects sent through stdin and stdout, respectively. There would be some setup to do, but no parsing on your side, or mucking with strings etc.

Here's an example of Python2 code via subprocess.Popen

p = subprocess.Popen(python3_args, stdin=subprocess.PIPE, stdout=subprocess.PIPE)
stdout, stderr = p.communicate(pickle.dumps(python3_args))
result = pickle.load(stdout)