What is the right way to "notify" processors without blocking?

Question

Suppose I have a very large array of things and I have to do some operation on all these things. In case operation fails for one element, I want to stop the work [this work is distributed across number of processors] across all the array.

I want to achieve this while keeping the number of sent/received messages to a minimum. Also, I don't want to block processors if there is no need to.

How can I do it using MPI?

Answer 1

A possible strategy to derive this global stop condition in a non-blocking fashion is to rely on MPI_Test.

scenario

Consider that each process posts an asynchronous receive of type MPI_INT to its left rank with a given tag to build a ring. Then start your computation. If a rank encounters the stop condition it sends its own rank to its right rank. In the meantime each rank uses MPI_Test to check for the completion of the MPI_Irecv during the computation if it is completed then enter a branch first waiting the message and then transitively propagating the received rank to the right except if the right rank is equal to the payload of the message (not to loop).

This done you should have all processes in the branch, ready to trigger an arbitrary recovery operation.

Complexity

The topology retained is a ring as it minimizes the number of messages at most (n-1) however it augments the propagation time. Other topologies could be retained with more messages but lower spatial complexity for example a tree with a n.ln(n) complexity.

Implementation

Something like this.

int rank, size;
MPI_Init(&argc,&argv);
MPI_Comm_rank( MPI_COMM_WORLD, &rank);
MPI_Comm_size( MPI_COMM_WORLD, &size);

int left_rank = (rank==0)?(size-1):(rank-1);
int right_rank = (rank==(size-1))?0:(rank+1)%size;

int stop_cond_rank;
MPI_Request stop_cond_request;
int stop_cond= 0;

while( 1 )
{
         MPI_Irecv( &stop_cond_rank, 1, MPI_INT, left_rank, 123, MPI_COMM_WORLD, &stop_cond_request);

         /* Compute Here and set stop condition accordingly */

         if( stop_cond )
         {
                 /* Cancel the left recv */
                 MPI_Cancel( &stop_cond_request );
                 if( rank != right_rank )
                            MPI_Send( &rank, 1, MPI_INT, right_rank, 123, MPI_COMM_WORLD ); 

                   break;
         }

         int did_recv = 0;
         MPI_Test( &stop_cond_request, &did_recv, MPI_STATUS_IGNORE );
         if( did_recv )
         {
                  stop_cond = 1;
                  MPI_Wait( &stop_cond_request, MPI_STATUS_IGNORE );
                  if( right_rank != stop_cond_rank )
                            MPI_Send( &stop_cond_rank, 1, MPI_INT, right_rank, 123, MPI_COMM_WORLD );

                   break;
          }
}

if( stop_cond )
{
      /* Handle the stop condition */
}
else
{
      /* Cleanup */
     MPI_Cancel( &stop_cond_request );
}