Same send and receive buffers in MPI

Question

In my code, each process works on certain portion of an array. I want each process to send the portion it worked on to other processes and receive other portions from other processes. For this I used MPI_Allgatherv but I kept send and receive buffers the same:

MPI_Allgatherv (&vel[0],  localSizesFaceV[rank], MPI_DOUBLE, &vel[0],  localSizesFaceV, displsFaceV, MPI_DOUBLE, MPI_COMM_WORLD);

I used this function before for other purposes with different send and receive buffers and it worked. That is why I am sure there is no problem with other parameters.

In the case of 2 processes, one of the processes does not return. When I copied send buffer to another std::vector

vector <double> vel2;
vel2 = vel;

and used vel2 as send buffer then all processes returned. Why?

Answer 1

Generally speaking, MPI requires that the argument is not aliased. This is explicitly mentioned chapter 2.3 of the current standard.

Unless specified otherwise, an argument of type OUT or type INOUT cannot be aliased with any other argument passed to an MPI procedure.

This explains why your code has problems. However, there is the possibility to solve your issue very easily, without having to explicitly copy your buffer: the MPI_IN_PLACE keyword. It specifies that the communication will be done "in-place" using the output buffer as an input buffer too wherever relevant.

Your code would become:

MPI_Allgatherv( MPI_IN_PLACE, 0, MPI_DATATYPE_NULL, &vel[0],  localSizesFaceV, displsFaceV, MPI_DOUBLE, MPI_COMM_WORLD);

NB: the actual type to use for the send buffer is irrelevant. You can keep MPI_DOUBLE if you want to, but I tend to prefer using MPI_DATATYPE_NULL to make clear that the parameter is ignored.