MPI - Suppress output from some processors

Question

Is there a way to make MPI just print the output from one (or a subset) of the processes? I know how to do that on code level, but I am wondering whether there is some way to specify this after the executable has already been compiled. I am thinking of getting the output as in one of the windows in the following command

mpirun -np [#processes]  xterm -e [path to executable]

just without using xterm.

I know that I can tag the output using the -tag-output option and then can filter it by hand, but this is kind of tedious. As MPI knows exactly where the output is coming from, is there no easier way to accomplish this goal?

Answer 1

You could write a wrapper script. Since Open MPI makes the process rank in MPI_COMM_WORLD available in an environment variable called OMPI_COMM_WORLD_RANK, it is very easy to do something like:

#!/bin/bash

UNMUTE=$1
shift 1

if [ "$OMPI_COMM_WORLD_RANK" == "$UNMUTE" ]; then
  exec $*
else
  exec $* >/dev/null 2>&1
fi

Save this to unmute.sh and execute it like:

$ mpiexec -n #procs unmute.sh #rank executable [params]

The script redirects the standard output and error streams to /dev/null for all ranks except for the one specified. You could make the comparison logic more elaborate, e.g. compare against a range or a list of ranks.

Here is a sample of the above script in action:

$ mpiexec -n 6 printenv | grep COMM_WORLD_RANK
OMPI_COMM_WORLD_RANK=0
OMPI_COMM_WORLD_RANK=1
OMPI_COMM_WORLD_RANK=2
OMPI_COMM_WORLD_RANK=3
OMPI_COMM_WORLD_RANK=4
OMPI_COMM_WORLD_RANK=5
$ mpiexec -n 6 unmute.sh 3 printenv | grep COMM_WORLD_RANK
OMPI_COMM_WORLD_RANK=3