Python embedded in CPP: how to get data back to CPP

Question

While working on a C++ project, I was looking for a third party library for something that is not my core business. I found a really good library, doing exactly what's needed, but it is written in Python. I decided to experiment with embedding Python code in C++, using the Boost.Python library.

The C++ code looks something like this:

#include <string>
#include <iostream>
#include <boost/python.hpp>

using namespace boost::python;

int main(int, char **) 
{
    Py_Initialize();

    try 
    {
        object module((handle<>(borrowed(PyImport_AddModule("__main__")))));

        object name_space = module.attr("__dict__");
        object ignored = exec("from myModule import MyFunc\n"
                          "MyFunc(\"some_arg\")\n",
                          name_space);

        std::string res = extract<std::string>(name_space["result"]);
    } 
    catch (error_already_set) 
    {
        PyErr_Print();
    }

    Py_Finalize();
    return 0;
}

A (very) simplified version of the Python code looks like this:

import thirdparty

def MyFunc(some_arg):
    result = thirdparty.go()
    print result

Now the problem is this: 'MyFunc' executes fine, i can see the print of 'result'. What i cannot do is read 'result' back from the C++ code. The extract command never finds 'result' in any namespace. I tried defining 'result' as a global, i even tried returning a tuple, but i cannot get it to work.

Answer 1

First of all, change your function to return the value. printing it will complicate things since you want to get the value back. Suppose your MyModule.py looks like this:

import thirdparty

def MyFunc(some_arg):
    result = thirdparty.go()
    return result

Now, to do what you want, you have to go beyond basic embedding, as the documentation says. Here is the full code to run your function:

#include <Python.h>

int
main(int argc, char *argv[])
{
    PyObject *pName, *pModule, *pFunc;
    PyObject *pArgs, *pArg, *pResult;
    int i;

    Py_Initialize();
    pName = PyString_FromString("MyModule.py");
    /* Error checking of pName left out as exercise */

    pModule = PyImport_Import(pName);
    Py_DECREF(pName);

    if (pModule != NULL) {
        pFunc = PyObject_GetAttrString(pModule, "MyFunc");
        /* pFunc is a new reference */

        if (pFunc) {
            pArgs = PyTuple_New(0);
            pArg = PyString_FromString("some parameter")
            /* pArg reference stolen here: */
            PyTuple_SetItem(pArgs, 0, pArg);
            pResult = PyObject_CallObject(pFunc, pArgs);
            Py_DECREF(pArgs);
            if (pResult != NULL) {
                printf("Result of call: %s\n", PyString_AsString(pResult));
                Py_DECREF(pResult);
            }
            else {
                Py_DECREF(pFunc);
                Py_DECREF(pModule);
                PyErr_Print();
                fprintf(stderr,"Call failed\n");
                return 1;
            }
        }
        else {
            if (PyErr_Occurred())
                PyErr_Print();
            fprintf(stderr, "Cannot find function");
        }
        Py_XDECREF(pFunc);
        Py_DECREF(pModule);
    }
    else {
        PyErr_Print();
        fprintf(stderr, "Failed to load module");
        return 1;
    }
    Py_Finalize();
    return 0;
}

Answer 2

Based on ΤΖΩΤΖΙΟΥ, Josh and Nosklo's answers i finally got it work using boost.python:

Python:

import thirdparty

def MyFunc(some_arg):
    result = thirdparty.go()
    return result

C++:

#include <string>
#include <iostream>
#include <boost/python.hpp>

using namespace boost::python;

int main(int, char **) 
{
    Py_Initialize();

    try 
    {
        object module = import("__main__");
        object name_space = module.attr("__dict__");
        exec_file("MyModule.py", name_space, name_space);

        object MyFunc = name_space["MyFunc"];
        object result = MyFunc("some_args");

        // result is a dictionary
        std::string val = extract<std::string>(result["val"]);
    } 
    catch (error_already_set) 
    {
        PyErr_Print();
    }

    Py_Finalize();
    return 0;
}

Some important points:

1. I changed 'exec' to 'exec_file' out of convenience, it also works with plain 'exec'.
2. The main reason it failed is that i did not pass a "local" name_sapce to 'exec' or 'exec_file' - this is now fixed by passing name_space twice.
3. If the python function returns unicode strings, they are not convertible to 'std::string', so i had to suffix all python strings with '.encode('ASCII', 'ignore')'.