Python "print" not working when embedded into MPI program

Question

I have an Python 3 interpreter embedded into an C++ MPI application. This application loads a script and passes it to the interpreter.

When I execute the program on 1 process without the MPI launcher (simply calling ./myprogram), the script is executed properly and its "print" statements output to the terminal. When the script has an error, I print it on the C++ side using PyErr_Print().

However when I lauch the program through mpirun (even on a single process), I don't get any output from the "print" in the python code. I also don't get anything from PyErr_Print() when my script has errors.

I guess there is something in the way Python deals with standard output that do not match the way MPI (actuall Mpich here) deals with redirecting the processes' output to the launcher and finally to the terminal.

Any idea on how to solve this?

Answer 1

[edit, following the advice from this issue]

You need to flush_io() after each call to PyErr_Print, where flush_io could be this function:

void flush_io(void)
{
  PyObject *type, *value, *traceback;
  PyErr_Fetch(&type, &value, &traceback);  // in Python/pythonrun.c, they save the traceback, let's do the the same

  for (auto& s: {"stdout", "stderr"}) {
    PyObject *f = PySys_GetObject(s);
    if (f)  PyObject_CallMethod(f, "flush", NULL);
    else    PyErr_Clear();
  }
  
  PyErr_Restore(type, value, traceback);
}

[below my old analysis, it still has some interesting info]

I ended up with the same issue (PyErr_Print not working from an mpirun). Tracing back (some gdb of python3 involved) and comparing the working thing (./myprogram) and non-working thing (mpirun -np 1 ./myprogram), I ended up in _io_TextIOWrapper_write_impl at ./Modules/_io/textio.c:1277 (python-3.6.0 by the way).

The only difference between the 2 runs is that self->line_buffering is 1 vs. 0 (at this point self represents sys.stderr). Then, in pylifecycle.c:1128, we can see who decided this value:

if (isatty || Py_UnbufferedStdioFlag)
    line_buffering = Py_True;

So it seems that MPI does something to stderr before launching the program, which makes it not a tty. I haven't investigated if there's an option in mpirun to keep the tty flag on stderr ... if someone knows, it'd be interesting (though on second thought mpi probably has good reasons to put his file descriptors in place of stdout&stderr, for its --output-filename for example).

With this info, I can come out with 3 solutions (the first 2 are quick-fixes, the 3rd is better):

1/ in the C code that starts the python interpreter, set the buffering flag before creating sys.stderr. The code becomes :

Py_UnbufferedStdioFlag = 1;   // force line_buffering for _all_ I/O
Py_Initialize(); 

This brings Python's traceback back to screen in all situations; but will probably give catastrophic I/O ... so only an acceptable solution in debug mode.

2/ in the python (embedded) script, at the very beginning add this :

import sys
#sys.stderr.line_buffering = True  # would be nice, but readonly attribute !
sys.stderr = open("error.log", 'w', buffering=1 )

The script then dumps the traceback to this file error.log.

I also tried adding a call to fflush(stderr) or fflush(NULL) right after the PyErr_Print() ... but this didn't work (because sys.stderr has its own internal buffering). That'd be a nice solution though.

3/ After a little more digging, I found the perfect function in

Python/pythonrun.c:57:static void flush_io(void);

It is in fact called after each PyErr_Print in this file. Unfortunately it's static (only exists in that file, no reference to it in Python.h, at least in 3.6.0). I copied the function from this file to myprogram and it turns out to do exactly the job.