How to run python script with mpi4py (using mpiexec) from within pycharm?

Question

I have a python script that depends on mpi4py implementation of MPI. This needs to be run through mpiexec (or mpirun).

My Question is : How to run (and hopefully debug) a python script using mpiexec (or mpirun) from PyCharm directly?

Running mpiexec as an external tool is not a good idea, because I will not be able to debug it.

What can I do instead?

Answer 1

I see two ways of debugging an mpi4py script with pycharm and one that involves Visual Studio.

Both are quite painful because PyCharm is not really meant to be at this stage a debug IDE for MPI application contrary to Visual Studio.

First option (the less painful) : You have PyCharm Professional 2019
We will use the remote debugger functionnality of PyCharm professional.
Start PyCharm create a project and go to Run->Edit Configurations, there you will find a template for Python Remote Debug (which is a feature reserved for PyCharm professional). Create one : leave port to 0 and tick allow parallel run. Click Apply and go back to main screen.
If you click on

You will see this console, the debug server is waiting for a connection; note the port on which it will be communicating (in my case 64777): Click as many time as you want mpi processes and note the port each time. This is the tidious part.

In my example I will run 4 processes, so clicked four times and got [64777, 64890, 64891, 64893]

Now for the beginning of the script you want to debug:

from mpi4py import MPI
import time
import os
size=MPI.COMM_WORLD.Get_size()
rank = MPI.COMM_WORLD.Get_rank()
import pydevd_pycharm
port_mapping=[65117, 65118, 65119, 65115]
pydevd_pycharm.settrace('localhost', port=port_mapping[rank], stdoutToServer=True, stderrToServer=True)

print os.getpid()


if(rank==0):
    print size, rank
    print 'go boy go !!!'
else:
    print size, rank
    print 'run Forrest...'

See how i reported the port.

Now open a terminal and execute :

And you are up and running, an automatic breakpoint is set just after the pydevd_pycharm.settrace on all process.
Debug console on port 65117, process on rank 0
Debug console on port 65118, process on rank 1

And that's it, except that the stepping and execution is not synchronized so it's painful but if you want to debug send recv it might do the trick.

Second option (very painful) : You have PyCharm Community 2019

At the beginning of your script print the PID of the process and put a sleep in order to leave you time to attach each process

from mpi4py import MPI
import time
import os
size=MPI.COMM_WORLD.Get_size()
rank = MPI.COMM_WORLD.Get_rank()

print os.getpid()
sys.stdout.flush()
time.sleep(200)

if (rank == 0):
    print size, rank
    print 'go boy go !!!'
else:
    print size, rank
    print 'run Forrest...'

Run


(Quickly you only got 200 seconds in my example), Go to Run -> Attach to process and select the process you want to debug knowing the PID (hence the print os.getpid())

And you should be allright.

Third option :

Buy a PC with Window Server 2012.

Install Visual Studio 2019 + PTVS (Python for Visual Studio) extension

Install HPC Pack/SDK 2016

Follow this tutorial : YouTube tutorial PTVS with MPI and discover the benefits of using Visual Studio for debugging python (and C++/C encapsulated at the same time).
Sorry for the publicity but the Microsoft team has really made an effort on this one…
Only forcing to have the head node of your HPC Server to be under Window Server 2012 is a bummer but anyway…