How do MPI processes communicate in the same node [duplicate]

Question

Assuming that we have one node with 12 cores. What are the differences between:

1. Run one MPI process to manage 12 threads for each core.
2. Just run 12 MPI processes for each core.

The former communicate via shared memory, and the latter communicate via IPC. So, which one is faster? are the differences negligible or significant？

Answer 1

Well, it depends ...

Most MPI implementations use shared memory (instead of the interconnect or even IPC) for intra-node communications.

Generally speaking, MPI+X is used for hybrid programming :

• MPI for inter node communications
• X within the same node

OpenMP is commonly used as X. MPI RMA (e.g one sided communications) can also be used, and even more options are available.

From a performance point of view, then once again, it depends. some applications run faster in flat MPI (e.g. one MPI process per core), whereas some other applications run faster in hybrid MPI+OpenMP. (keep in mind OpenMP was designed for shared memory system with a flat access to memory, so there is generally one MPI task per NUMA domain (e.g. socket most of the time) vs one MPI task per node.

Last but not least, MPI+OpenMP memory overhead and wire-up time is generally lower than flat MPI, and this can be an important factor.