MPI: What to do when number of expected MPI_Recv is unkown

Question

I've got many slave nodes which might or might not send messages to the master node. So currently there's no way the master node knows how many MPI_Recv to expect. Slave nodes had to sent minimum number of messages to the master node for efficiency reasons.

I managed to find a cool trick, which sends an additional "done" message when its no longer expecting any messages. Unfortunately, it doesn't seem to work in my case, where there're variable number of senders. Any idea on how to go about this? Thanks!

if(rank == 0){ //MASTER NODE

    while (1) {

        MPI_Recv(&buffer, 10, MPI_INT, MPI_ANY_SOURCE, MPI_ANY_TAG, MPI_COMM_WORLD, &status);

        if (status.MPI_TAG == DONE) break;


        /* Do stuff */
    }

}else{ //MANY SLAVE NODES

    if(some conditions){
        MPI_Send(&buffer, 64, MPI_INT, root, 1, MPI_COMM_WORLD);
    }

}


MPI_Barrier(MPI_COMM_WORLD);
MPI_Send(NULL, 1, MPI_INT, root, DONE, MPI_COMM_WORLD);

Not working, the program seem to be still waiting for a MPI_Recv

Answer 1

A simpler and more elegant option would be to use the MPI_IBARRIER. Have each worker call all of the sends that it needs to and then call MPI_IBARRIER when it's done. On the master, you can loop on both an MPI_IRECV on MPI_ANY_SOURCE and an MPI_IBARRIER. When the MPI_IBARRIER is done, you know that everyone has finished and you can cancel the MPI_IRECV and move on. The pseudocode would look something like this:

if (master) {
  /* Start the barrier. Each process will join when it's done. */
  MPI_Ibarrier(MPI_COMM_WORLD, &requests[0]);

  do {
    /* Do the work */
    MPI_Irecv(..., MPI_ANY_SOURCE, &requests[1]);

    /* If the index that finished is 1, we received a message.
     * Otherwise, we finished the barrier and we're done. */
    MPI_Waitany(2, requests, &index, MPI_STATUSES_IGNORE);
  } while (index == 1);

  /* If we're done, we should cancel the receive request and move on. */
  MPI_Cancel(&requests[1]);
} else {
  /* Keep sending work back to the master until we're done. */
  while( ...work is to be done... ) {
    MPI_Send(...);
  }

  /* When we finish, join the Ibarrier. Note that
   * you can't use an MPI_Barrier here because it
   * has to match with the MPI_Ibarrier above. */
  MPI_Ibarrier(MPI_COMM_WORLD, &request);
  MPI_Wait(&request, MPI_STATUS_IGNORE);
}