mpirun retaining redirected stdout

Question

I am running in a cluster. I tried to run my executable with 4 different forms:

1. In serial, with

   myexec
   

   This starts giving output in stdout right away, as expected.

2. In serial, redirecting stdout and stderr, with

   myexec > out-err.log 2>&1
   

   This starts giving output in out-err.log right away, as expected (verified with cat out-err.log in another terminal).

3. In parallel, with

   mpirun -n 2 myexec
   

   This starts giving output in stdout right away, as expected.

4. In parallel, redirecting stdout and stderr, with

   mpirun -n 2 myexec > out-err.log 2>&1
   

   This retains output until job is finished (due to completion or time allowance).

Is there any way of having stdout/stderr "flushed" at runtime in case 4, so I can check out-err.log?

Answer 1

This is a known feature/issue with redirection in mpi. I found how to solve this:

1. Add export OMPI_MCA_opal_event_include=poll in ~/.bashrc, or

2. Add opal_event_include=poll in ~/.openmpi/mca-params.conf (create the dir and/or file if they do not exist).

The sources used to get info are:

https://www.cfd-online.com/Forums/openfoam-installation/162664-openfoam-2-4-0-openmpi-epoll-warning-parallel-job.html

https://github.com/open-mpi/ompi/issues/341

https://www.open-mpi.org/doc/v2.0/man1/mpirun.1.php#sect20