How to use MPI_Reduce to Sum different values from Different groups of processors independently

Question

I am trying to divide my processors into groups then add the summation of each group independently ... but I couldn't find the result correctly until now. a simple example is as follows:

int main(int argc, char** argv) 
{
    int size, rank,i=0,localsum1=0,globalsum1=0,localsum2=0,globalsum2=0;

    MPI_Init(&argc,&argv);
    MPI_Comm_size(MPI_COMM_WORLD,&size);
    MPI_Comm_rank(MPI_COMM_WORLD,&rank);

    if(rank==0)
    {
    }
    else if(rank==1)
    {
        localsum1 += 5;
        MPI_Reduce(&localsum1,&globalsum1,2,MPI_INT,MPI_SUM,0,MPI_COMM_WORLD);
    }
    else if(rank==2)
    {
        localsum2 += 10;
        MPI_Reduce(&localsum2,&globalsum2,2,MPI_INT,MPI_SUM,0,MPI_COMM_WORLD);
    }

    if(rank==0)
    {
        printf("globalsum1 = %d \n",globalsum1);
        printf("globalsum2 = %d \n",globalsum2);
    }
    MPI_Finalize();

    return (EXIT_SUCCESS);
}

I can't figure out what is missing here ... can anyone help?

Answer 1

MPI_Reduce is a collective operation. What that means is that all tasks in the participating communicator must make the MPI_Reduce() call. In the above, rank 0 never calls MPI_Reduce() so this program will hang as some of the other processors wait for participation from rank 0 which will never come.

Also, because it is a collective operation on the entire communicator, you need to do some work to partition the reduction. One way is just to reduce an array of ints, and have each processor contribute only to its element in the array:

#include <stdio.h>
#include <stdlib.h>
#include <mpi.h>

int main(int argc, char** argv)
{
    int size, rank;

    MPI_Init(&argc,&argv);
    MPI_Comm_size(MPI_COMM_WORLD,&size);
    MPI_Comm_rank(MPI_COMM_WORLD,&rank);

    int localsum[2] = {0,0};
    int globalsum[2] = {0,0};

    if(rank % 2 == 1)
    {
        localsum[0] += 5;
    }
    else if( rank > 0 && (rank % 2 == 0))
    {
        localsum[1] += 10;
    }

    MPI_Reduce(localsum,globalsum,2,MPI_INT,MPI_SUM,0,MPI_COMM_WORLD);

    if(rank==0)
    {
        printf("globalsum1 = %d \n",globalsum[0]);
        printf("globalsum2 = %d \n",globalsum[1]);
    }

    MPI_Finalize();

    return (EXIT_SUCCESS);
}

where running now gives

$ mpicc -o reduce reduce.c
$ mpirun -np 3 ./reduce
globalsum1 = 5 
globalsum2 = 10 

Otherwise, you can create communicators that only connect the processors you want to be involved in each sum, and do the reductions within each commuicator. Below is a not-very-pretty way to do this. This is quite powerful in general but more complicated than the first solution:

#include <stdio.h>
#include <stdlib.h>
#include <mpi.h>

int main(int argc, char** argv)
{
    int size, rank;

    MPI_Init(&argc,&argv);
    MPI_Comm_size(MPI_COMM_WORLD,&size);
    MPI_Comm_rank(MPI_COMM_WORLD,&rank);

    int localsum = 0;
    int globalsum = 0;

    MPI_Comm  comm_evens_plus_root, comm_odds_plus_root;
    MPI_Group grp_evens_plus_root, grp_odds_plus_root, grp_world;

    MPI_Comm_group(MPI_COMM_WORLD, &grp_world);
    int *ranks = malloc((size/2 + 1) * sizeof(rank));
    int i,j;
    for (i=1, j=0; i<size; i+=2, j+=1)
        ranks[j] = i;
    MPI_Group_excl(grp_world, j, ranks, &grp_evens_plus_root);
    MPI_Comm_create(MPI_COMM_WORLD, grp_evens_plus_root, &comm_evens_plus_root);

    for (i=2, j=0; i<size; i+=2, j+=1)
        ranks[j] = i;
    MPI_Group_excl(grp_world, j, ranks, &grp_odds_plus_root);
    MPI_Comm_create(MPI_COMM_WORLD, grp_odds_plus_root, &comm_odds_plus_root);

    free(ranks);

    if(rank % 2 == 1)
    {
        localsum += 5;
        MPI_Reduce(&localsum,&globalsum,1,MPI_INT,MPI_SUM,0,comm_odds_plus_root);
    }
    else if( rank > 0 && (rank % 2 == 0))
    {
        localsum += 10;
        MPI_Reduce(&localsum,&globalsum,1,MPI_INT,MPI_SUM,0,comm_evens_plus_root);
    }

    if(rank==0)
    {
        MPI_Reduce(&localsum,&globalsum,1,MPI_INT,MPI_SUM,0,comm_odds_plus_root);
        printf("globalsum1 = %d \n",globalsum);
        MPI_Reduce(&localsum,&globalsum,1,MPI_INT,MPI_SUM,0,comm_evens_plus_root);
        printf("globalsum2 = %d \n",globalsum);
    }

    MPI_Comm_free(&comm_odds_plus_root);
    MPI_Comm_free(&comm_evens_plus_root);
    MPI_Group_free(&grp_odds_plus_root);
    MPI_Group_free(&grp_evens_plus_root);
    MPI_Finalize();

    return (EXIT_SUCCESS);
}

Running gives

$ mpicc -o reduce2 reduce2.c 
$ mpirun -np 3 ./reduce
globalsum1 = 5 
globalsum2 = 10