When would you use different counts or types for sending and receiving processes?

Question

Many routines in MPI that describe both sending and receiving - MPI_Sendrecv, MPI_Scatter, etc - have arguments for counts and types for both the send and the receive. For instance, in Fortran the signature for MPI_Scatter is:

MPI_SCATTER(SENDBUF, SENDCOUNT, SENDTYPE, 
            RECVBUF, RECVCOUNT, RECVTYPE, ROOT, COMM, IERROR)

If the amount of data sent has to be the same as the amount received, why are both needed? Doesn't this just introduce the possibility of inconsistency?

What is the use case where different counts/types are needed?

Answer 1

MPI requires that the sending and receiving processes agree on the type of data and amount (sort of; for point to point communications receiver can request more than is sent). But MPI data types also describe the layout in memory of the data, and that's a very common reason to need to use different types for the sender and the receiver.

You ask in particular about Scatter and Fortran, so let's take a look at that case. Let's consider scattering an size*n matrix, by rows, to different processes

    |---n---|  ---
     0 0 0 0    |
a =  1 1 1 1   size
     2 2 2 2    |
               ---

So that rank 0 gets [0 0 0 0], rank 1 gets [1 1 1 1], etc.

In Fortran, those aren't contiguous in memory; so to describe a row, you would have to use an MPI_Type_vector:

    call MPI_Type_vector(n, 1, size, MPI_REAL, row_type, ierr)

That describes n reals, but with each separated by size reals between them.

On the other hand, if the receiving process is just receiving that data into a 1-d array:

    real, dimension(n) :: b

Then it can not use that type to describe the data; b doesn't have enough room to hold n reals each with a gap of size between them! It wants to receive the data just as n * MPI_REAL. This mismatch would be the same in C if you had to send columns of data.

And so this is a common reason for specifying the type (and thus the count) of data differently; for the scatter-er, the data has to be described with a data type which includes the layout of the larger data structure holding the values to be sent; but the scatter-ee may well be receiving the data into a different data structure, with a different layout.

A working simple example follows.

program scatterdemo
    use mpi
    implicit none
    real, allocatable, dimension(:,:) :: a
    real, allocatable, dimension(:) :: b
    integer :: ierr, rank, comsize
    integer, parameter :: n=4
    integer :: i
    integer :: row_type, row_type_sized, real_size
    integer(kind=MPI_ADDRESS_KIND) :: lb=0, extent

    call MPI_Init(ierr)
    call MPI_Comm_rank(MPI_COMM_WORLD, rank, ierr)
    call MPI_Comm_size(MPI_COMM_WORLD, comsize, ierr)

    if (rank == 0) then
        allocate( a(comsize, n) )
        do i=1,comsize
            a(i,:) = i-1
        enddo
    endif

    allocate( b(n) )

    call MPI_Type_size(MPI_REAL, real_size, ierr)

    call MPI_Type_vector(n, 1, comsize, MPI_REAL, row_type, ierr)
    extent = real_size*1
    call MPI_Type_create_resized(row_type, lb, extent, row_type_sized, ierr)
    call MPI_Type_commit(row_type_sized, ierr)

    call MPI_Scatter(a, 1, row_type_sized, b, n, MPI_REAL, 0, MPI_COMM_WORLD, ierr)

    print *, rank, b

    if (rank == 0) deallocate (a)
    deallocate(b)

    call MPI_Finalize(ierr)

end program scatterdemo

and running it with six processors gives

$ mpirun -np 6 ./scatter
           0  0.0000000E+00  0.0000000E+00  0.0000000E+00  0.0000000E+00
           3   3.000000       3.000000       3.000000       3.000000
           1   1.000000       1.000000       1.000000       1.000000
           5   5.000000       5.000000       5.000000       5.000000
           2   2.000000       2.000000       2.000000       2.000000
           4   4.000000       4.000000       4.000000       4.000000

Answer 2

When you use MPI derived types, you can view the array as n elements of some basic numeric type and also just as one or more elements of some MPI derived type. In that case not only the counts but also the datatypes can differ although they correspond to the same buffer.

On the other hand it is NOT affected by the number of processes in the communicator. The communicator size is always implicit and you do not enter it directly anywhere when calling a collective.