Interact with a MPI binary via a (Non-MPI) python script

Question

I would like to somehow trigger the execution of certain functions of a MPI program (written in C++) via, e.g., a (serial) python script. This python script should launch the mpi program at the beginning with, e.g.,

subprocess.call(['mpirun','-np', '4', 'mpibinary', 'args' ])

I need to call a function of this MPI program multiple times and I want to avoid restarting the program for different inputs, as i have to reinitialize all my data structures which is costly. Therefore, I have thought about externally triggering a function when the MPI program is idle. I think this could be done with file IO, i.e., the root rank of the MPI program watches a certain file in a while(1) loop and as soon as its content changes it parses the new content notifies the other ranks and calls a function. Is there a more elegant solution to my problem?

The best solution would by to have a python class which wraps the important functions of the C++ MPI program so that i can call them from python with

mpiprogram.superfunction(a,b)

Answer 1

Perhaps the most elegant solution would be to make the Python code part of your MPI application. Then it would be able to directly send data (via MPI messages) to the rest of the MPI application as it will be part of it. There are two different approaches here:

1) Insert the Python binary as rank 0 in your MPI job. In order to exclude it from participation in collective operations in mpibinary, you would have to make a subcommunicator that excludes rank 0 and use it for all further collective communication in mpibinary. The first step is the easy part. In Open MPI you would do:

mpirun --hostfile hosts -np 1 pythonbinary args : -np 32 mpibinary args

This is called MPMD (multiple programs multiple data) launch and it will start one copy of pythonbinary that will become rank 0 and also 32 copies of mpibinary that will become rank 1, rank 2, ... up to rank 32 (33 processes in total). Other MPI implementations also provide very similar mechanisms for MPMD launch. Then you'd use MPI_Comm_split() in order to create a new communicator that does not include the Python program. Splitting a communicator is a collective operation. That's why you have to call it both in your Python code and your C++ application. MPI_Comm_split() takes a "color" and a key and splits the communicator in multiple subcommunicators according to the different colors. Processes with the same color are then sorted based on the key value. You will most likely want to call it like this:

in Python:

python_comm = mpi.mpi_comm_split(mpi.MPI_COMM_WORLD, 0, 0)

in C++:

int rank;
MPI_Comm c_comm;

MPI_Comm_rank(MPI_COMM_WORLD, &rank);
MPI_Comm_split(MPI_COMM_WORLD, 1, rank, &c_comm);

By using the rank as key one guarantees that the order of the processes in c_comm will be the same as it was before the split, i.e. rank 1 from MPI_COMM_WORLD will become rank 0 in c_comm, rank 2 will become rank 1, etc.

From now on the C++ application can use c_comm to perform collective operations as usual. In order to communicate between the Python and the C++ code, you still have to use MPI_COMM_WORLD and the Python code would still be rank 0 in it.

2) Use the MPI-2 process management facilities. First you would run an MPI job that consists of the Python binary only:

mpirun --hostfile hosts -np 1 pythonbinary args

Then the Python binary would spawn the other MPI binary directly using MPI_Comm_spawn() with the desired number of new processes. The newly spawned processes will have their own MPI_COMM_WORLD and you would not need to use MPI_Comm_split(). Also the spawn operation will establish an intercommunicator that will allow the Python code to send messages to the other part of the MPI application.

---

In both cases the hosts file would contain definition of all execution hosts that can execute the MPI binaries. You would also need to use one of the available Python MPI bindings.

Note that you only need to add some MPI calls to your Python script like MPI_Init, MPI_Finalize, MPI_Comm_split and the relevant MPI_Send/MPI_Recv. You do not need to make it parallel. MPI is quite versatile in that allows you to not only use it for parallel worksharing but also as a general messaging framework. But please note that the Python bindings should use the same MPI library as the rest of the program.

Another solution would be to use some message queuing library or file pooling (which is really a crude MQ implementation).