When do I need to use MPI_Barrier()?

Question

I wonder when do I need to use barrier? Do I need it before/after a scatter/gather for example? Or should OMPI ensure all processes have reached that point before scatter/gather-ing? Similarly, after a broadcast can I expect all processes to already receive the message?

Answer 1

All collective operations in MPI before MPI-3.0 are blocking, which means that it is safe to use all buffers passed to them after they return. In particular, this means that all data was received when one of these functions returns. (However, it does not imply that all data was sent!) So MPI_Barrier is not necessary (or very helpful) before/after collective operations, if all buffers are valid already.

Please also note, that MPI_Barrier does not magically wait for non-blocking calls. If you use a non-blocking send/recv and both processes wait at an MPI_Barrier after the send/recv pair, it is not guaranteed that the processes sent/received all data after the MPI_Barrier. Use MPI_Wait (and friends) instead. So the following piece of code contains errors:

/* ERRORNOUS CODE */  Code for Process 0: Process 0 sends something using MPI_Isend MPI_Barrier(MPI_COMM_WORLD); Process 0 uses buffer passed to MPI_Isend // (!)  Code for Process 1: Process 1 recvs something using MPI_Irecv MPI_Barrier(MPI_COMM_WORLD); Process 1 uses buffer passed to MPI_Irecv // (!) 

Both lines that are marked with (!) are unsafe!

MPI_Barrier is only useful in a handful of cases. Most of the time you do not care whether your processes sync up. Better read about blocking and non-blocking calls!

Answer 2

One use of MPI_Barrier is for example to control access to an external resource such as the filesystem, which is not accessed using MPI. For example, if you want each process to write stuff to a file in sequence, you could do it like this:

int rank, size; MPI_Comm_rank(MPI_COMM_WORLD, &rank); MPI_Comm_size(MPI_COMM_WORLD, &size); for ( int ii = 0; ii < size; ++ii ) {     if ( rank == ii ) {         // my turn to write to the file         writeStuffToTheFile();     }     MPI_Barrier(MPI_COMM_WORLD); } 

That way, you can be sure that no two processes are concurrently calling writeStuffToTheFile.