How To catch python stdout in c++ code

Question

I have a program which during it's run sometimes needs to call python in order to preform some tasks. I need a function that calls python and catches pythons stdout and puts it in some file. This is a declaration of the function

  pythonCallBackFunc(const char* pythonInput) 

My problem is to catch all the python output for a given command (pythonInput). I have no experience with python API and I don't know what is the right technique to do this. First thing I've tried is to redirect python's sdtout and stderr using Py_run_SimpleString this is some example of the code i've written.

#include "boost\python.hpp" #include <iostream>  void pythonCallBackFunc(const char* inputStr){         PyRun_SimpleString(inputStr);  }   int main () {     ...    //S0me outside functions does this    Py_Initialize();    PyRun_SimpleString("import sys");    PyRun_SimpleString("old_stdout = sys.stdout");    PyRun_SimpleString("fsock = open('python_out.log','a')");    PyRun_SimpleString("sys.stdout = fsock");    ...     //my func       pythonCallBackFunc("print 'HAHAHAHAHA'");    pythonCallBackFunc("result = 5");    pythonCallBackFunc("print result");     pythonCallBackFunc("result = 'Hello '+'World!'");    pythonCallBackFunc("print result");     pythonCallBackFunc("'KUKU '+'KAKA'");    pythonCallBackFunc("5**3");     pythonCallBackFunc("prinhghult");     pythonCallBackFunc("execfile('stdout_close.py')");    ...      //Again anothers function code    PyRun_SimpleString("sys.stdout = old_stdout");    PyRun_SimpleString("fsock.close()");     Py_Finalize();    return 0; } 

Is there a better way to do this? Besides, for some reason PyRun_SimpleString does nothing when it gets some mathematical expression, for example PyRun_SimpleString("5**3") prints nothing (python conlsul prints the result: 125)

maybe it is important, i am using visual studio 2008. Thanks, Alex

---

Changes I've made according Mark's suggestion:

  #include <python.h>   #include <string>    using namespace std;    void PythonPrinting(string inputStr){       string stdOutErr =     "import sys\n\      class CatchOut:\n\         def __init__(self):\n\            self.value = ''\n\         def write(self, txt):\n\            self.value += txt\n\      catchOut = CatchOut()\n\      sys.stdout = catchOut\n\      sys.stderr = catchOut\n\     "; //this is python code to redirect stdouts/stderr       PyObject *pModule = PyImport_AddModule("__main__"); //create main module      PyRun_SimpleString(stdOutErr.c_str()); //invoke code to redirect       PyRun_SimpleString(inputStr.c_str());      PyObject *catcher = PyObject_GetAttrString(pModule,"catchOut");       PyObject *output = PyObject_GetAttrString(catcher,"value");      printf("Here's the output: %s\n", PyString_AsString(output));       }    int main(int argc, char** argv){          Py_Initialize();       PythonPrinting("print 123");      PythonPrinting("1+5");      PythonPrinting("result = 2");          PythonPrinting("print result");           Py_Finalize();          return 0;   } 

The output i get after running main:

 Here's the output: 123   Here's the output:  Here's the output:   Here's the output: 2 

It is good for me , but only one problem, it should be

 Here's the output: 123   Here's the output: 6   Here's the output:   Here's the output: 2 

I dont know why but after running this command: PythonPrinting("1+5"), PyString_AsString(output) command returns an empty string (char*) instead of 6... :( Is there somthing i can do not to loose this output?

Thaks, Alex

Answer 1

Here is a C++ friendly solution I have developed lately.

I explain a few details of it on my blog: Python sys.stdout redirection in C++ where I also point to repository at my GitHub where most recent version can be found. Here is complete example based on the current code at the time of posting this answer:

#include <functional> #include <iostream> #include <string> #include <Python.h>  namespace emb {  typedef std::function<void(std::string)> stdout_write_type;  struct Stdout {     PyObject_HEAD     stdout_write_type write; };  PyObject* Stdout_write(PyObject* self, PyObject* args) {     std::size_t written(0);     Stdout* selfimpl = reinterpret_cast<Stdout*>(self);     if (selfimpl->write)     {         char* data;         if (!PyArg_ParseTuple(args, "s", &data))             return 0;          std::string str(data);         selfimpl->write(str);         written = str.size();     }     return PyLong_FromSize_t(written); }  PyObject* Stdout_flush(PyObject* self, PyObject* args) {     // no-op     return Py_BuildValue(""); }  PyMethodDef Stdout_methods[] = {     {"write", Stdout_write, METH_VARARGS, "sys.stdout.write"},     {"flush", Stdout_flush, METH_VARARGS, "sys.stdout.flush"},     {0, 0, 0, 0} // sentinel };  PyTypeObject StdoutType = {     PyVarObject_HEAD_INIT(0, 0)     "emb.StdoutType",     /* tp_name */     sizeof(Stdout),       /* tp_basicsize */     0,                    /* tp_itemsize */     0,                    /* tp_dealloc */     0,                    /* tp_print */     0,                    /* tp_getattr */     0,                    /* tp_setattr */     0,                    /* tp_reserved */     0,                    /* tp_repr */     0,                    /* tp_as_number */     0,                    /* tp_as_sequence */     0,                    /* tp_as_mapping */     0,                    /* tp_hash  */     0,                    /* tp_call */     0,                    /* tp_str */     0,                    /* tp_getattro */     0,                    /* tp_setattro */     0,                    /* tp_as_buffer */     Py_TPFLAGS_DEFAULT,   /* tp_flags */     "emb.Stdout objects", /* tp_doc */     0,                    /* tp_traverse */     0,                    /* tp_clear */     0,                    /* tp_richcompare */     0,                    /* tp_weaklistoffset */     0,                    /* tp_iter */     0,                    /* tp_iternext */     Stdout_methods,       /* tp_methods */     0,                    /* tp_members */     0,                    /* tp_getset */     0,                    /* tp_base */     0,                    /* tp_dict */     0,                    /* tp_descr_get */     0,                    /* tp_descr_set */     0,                    /* tp_dictoffset */     0,                    /* tp_init */     0,                    /* tp_alloc */     0,                    /* tp_new */ };  PyModuleDef embmodule = {     PyModuleDef_HEAD_INIT,     "emb", 0, -1, 0, };  // Internal state PyObject* g_stdout; PyObject* g_stdout_saved;  PyMODINIT_FUNC PyInit_emb(void)  {     g_stdout = 0;     g_stdout_saved = 0;      StdoutType.tp_new = PyType_GenericNew;     if (PyType_Ready(&StdoutType) < 0)         return 0;      PyObject* m = PyModule_Create(&embmodule);     if (m)     {         Py_INCREF(&StdoutType);         PyModule_AddObject(m, "Stdout", reinterpret_cast<PyObject*>(&StdoutType));     }     return m; }  void set_stdout(stdout_write_type write) {     if (!g_stdout)     {         g_stdout_saved = PySys_GetObject("stdout"); // borrowed         g_stdout = StdoutType.tp_new(&StdoutType, 0, 0);     }      Stdout* impl = reinterpret_cast<Stdout*>(g_stdout);     impl->write = write;     PySys_SetObject("stdout", g_stdout);     }  void reset_stdout() {     if (g_stdout_saved)         PySys_SetObject("stdout", g_stdout_saved);      Py_XDECREF(g_stdout);     g_stdout = 0; }  } // namespace emb  int main() {     PyImport_AppendInittab("emb", emb::PyInit_emb);     Py_Initialize();     PyImport_ImportModule("emb");      PyRun_SimpleString("print(\'hello to console\')");      // here comes the ***magic***     std::string buffer;     {         // switch sys.stdout to custom handler         emb::stdout_write_type write = [&buffer] (std::string s) { buffer += s; };         emb::set_stdout(write);         PyRun_SimpleString("print(\'hello to buffer\')");         PyRun_SimpleString("print(3.14)");         PyRun_SimpleString("print(\'still talking to buffer\')");         emb::reset_stdout();     }      PyRun_SimpleString("print(\'hello to console again\')");     Py_Finalize();      // output what was written to buffer object     std::clog << buffer << std::endl; } 

This allows to intercept sys.stdout.write output with any kind of callable C++ entity: free function, class member function, named function objects or even anonymous functions as in the example above where I use C++11 lambda.

Note, this is a minimal example to present the essential concept. In production-ready code, it certainly needs more attention around reference counting of PyObject, getting rid of global state, and so on.