Merging arrays from all ranks using MPI

Question

I have an array of same length on all ranks (Lets assume 10). Some values in the array contain the rank of the processor. For example ...

Proc 1: [1 0 0 0 0 1 0 0 0 1]

Proc 2: [0 2 2 0 0 0 0 2 2 0]

Proc 3: [0 0 0 3 3 0 3 0 0 0]

Now what is the most efficient way (using MPI-2) that all processors end with the following array

[1 2 2 3 3 1 3 2 2 1]

which can be thought of as the sum of all arrays (distributed on all ranks). Performance is important as I want to do this fast on 1K+ cores.

Answer 1

This is doable by MPI_Allreduce() with MPI_SUM or MPI_MAX operator. See the documentation of MPI_Allreduce. It is supposed to be implemented in one of the best possible ways for the given architecture.

int arr_a[LEN], arr_b[LEN];
...
// Fill in arr_a
MPI_Allreduce(arr_a, arr_b, LEN, MPI_INT, MPI_SUM, MPI_COMM_WORLD);
// Result is now in arr_b

Or if short on memory you could use an in-place operation but that will hurt the performance:

MPI_Allreduce(MPI_IN_PLACE, arr_a, LEN, MPI_INT, MPI_SUM, MPI_COMM_WORLD);