MPI: Canceling Non-blocking Send

Question

I'm using Open MPI library to implement the following algorithm: we have two processes p1 and p2. They're both executing some iterations and at the end of each iteration, they communicate their results. The problem is that the execution is not necessarily balanced, so p1 may execute 10 iterations in the time p2 executes 1. Even though, I want p2 to read the latest result from the last iteration executed by p1.

Thus, my idea is that p1 sends its results at each iteration. But, before sending the result from an iteration i, it should check if p2 actually read the information from iteration i-1. If not, it should cancel the previous send so that when p2 reads from p1, it will read the most recent result.

Unfortunately, I'm not sure how to do that. I've tried using MPI_Cancel, as in the following code:

int main (int argc, char *argv[]){

    int myrank, numprocs;
    MPI_Status status;
    MPI_Request request;

    MPI_Init(&argc, &argv);
    MPI_Comm_size(MPI_COMM_WORLD, &numprocs);
    MPI_Comm_rank(MPI_COMM_WORLD, &myrank);

    if(myrank == 0){
        int send_buf = 1, flag;
        MPI_Isend(&send_buf, 1, MPI_INT, 1, 123, MPI_COMM_WORLD, 
                  &request);
        MPI_Cancel(&request);
        MPI_Wait(&request, &status);
        MPI_Test_cancelled(&status, &flag);
        if (flag) printf("Send cancelled\n");
        else printf("Send NOT cancelled\n");
        send_buf = 2;
        MPI_Isend(&send_buf, 1, MPI_INT, 1, 123, MPI_COMM_WORLD, 
                  &request);
    }
    else {
        sleep(5);
        int msg;
        MPI_Recv(&msg, 1, MPI_INT, 0, 123,
                 MPI_COMM_WORLD, &status);
        printf("%d\n", msg);
    }
    MPI_Finalize();

    return 0;
}

But when I execute, it says that the send could not be cancelled and p2 prints 1 instead of 2.

I'd like to know if there's any way to achieve what I'm proposing or if there's an alternative to code the behaviour between p1 and p2.

Answer 1

I would reverse the control of communications. Instead of p1 sending unnecessary messages that it has to cancel, p2 should signal that it is ready to receive a message, and p1 would send only then. In the meantime, p1 simply overwrites its send buffer with the latest results.

In (untested) code:

if ( rank == 0 )
{
    int ready;
    MPI_Request p2_request;
    MPI_Status p2_status;
    // initial request
    MPI_Irecv(&ready, 1, MPI_INT, 1, 123, MPI_COMM_WORLD, &p2_request);
    for (int i=0; true; i++)
    {
        sleep(1);
        MPI_Test(&p2_request, &ready, &p2_status);
        if ( ready )
        {
            // blocking send: p2 is ready to receive
            MPI_Send(&i, 1, MPI_INT, 1, 123, MPI_COMM_WORLD);
            // post new request
            MPI_Irecv(&ready, 1, MPI_INT, 1, 123, MPI_COMM_WORLD, &p2_request);
        }
    }
}
else
{
    int msg;
    MPI_Status status;
    while (true)
    {
        sleep(5);
        // actual message content doesn't matter, just let p1 know we're ready
        MPI_Send(&msg, 1, MPI_INT, 0, 123, MPI_COMM_WORLD);
        // receive message
        MPI_Recv(&msg, 1, MPI_INT, 0, 123, MPI_COMM_WORLD, &status);
    }
}

Now like I said, that's untested code, but you can probably see what I'm getting at there. MPI_Cancel should only be used when things go horribly wrong: no message should be cancelled during normal execution.