I want Python as front end, Fortran as back end. I also want to make fortran part parallel - best strategy?

Question

I have a python script I hope to do roughly this:

1. calls some particle positions into an array

2. runs algorithm over all 512^3 positions to distribute them to an NxNxN matrix

3. feed that matrix back to python

4. use plotting in python to visualise matrix (i.e. mayavi)

First I have to write it in serial but ideally I want to parrallelize step 2 to speed up computation. What tools/strategy might get me started. I know Python and Fortran well but not much about how to connect the two for my particular problem. At the moment I am doing everything in Fortran then loading my python program - I want to do it all at once.I've heard of py2f but I want to get experienced people's opinions before I go down one particular rabbit hole. Thanks

Edit: The thing I want to make parallel is 'embarrassingly parallel' in that is is just a loop of N particles and I want to get through that loop as quickly as possible.

Answer 1

You have two basic options for binding. First to use f2py, the other to use interoperability with C in your Fortran and bind using Cython. Tutorial for f2py is here. It is not at all difficult, there are some directives for f2py to place to your Fortran code, but often they are not needed.

For parallelization, the first approach to use is probably OpenMP, if parallelization on a single machine is enough for you. It uses threads and is easy to use for loops with embarrassing parallelism. Just make sure you do not write to any global variables in the threads and if yes, use synchronization directives for that.

Answer 2

An alternative approach to VladimirF's suggestion, could be to set up the two parts as a client server construct, where your Python part could talk to the Fortran part using sockets. Though this comes with the burden to implement some protocol for the interaction, it has the advantage, that you get a clean separation and can even go on running them on different machines with an interaction over the network.

In fact, with this approach you even could do the embarrassing parallel part, by spawning as many instances of the Fortran application as needed and feed them all with different data.

Answer 3

More an answer than a comment in the same direction as @haraldkl - I am currently using sockets to send data from Fortran to Ptyhon and viceversa. A minimal working example can be found in this repo. Sockets in Fortran are a bit tricky since there is no built-in library supporting them and you need to use a wrapper around the C library. Still, works great.

My idea is to incorporate this strategy in a bigger parallel Fortran code I have even though I have not yet figured out that part. Probably threading the sockets in Python and using multiple ports could work.