What is the difference and relationship between mpirun, mpiexec and mpiexec.hydra?

Question

I got confused about 3 things: mpirun, mpiexec and mpiexec.hydra

On my cluster, all of them exist, and all of them belong to intel.

What is the difference and relationship between them? Especially, what on earth is mpiexec.hydra? Why there is a dot between mpiexec and hydra, what does it mean?

Answer 1

mpirun and mpiexec are basically the same - the name of the process launcher in many MPI implementations. The MPI standard says nothing about how the ranks should be started and controlled, but it recommends (though does not demand) that, if there is a launcher of any kind, it should be named mpiexec. Some MPI implementations started with mpirun, then adopted mpiexec for compatibility. Other implementations did the reverse. In the end, most implementations provide their launcher under both names. In practice, there should be no difference in what mpirun and mpiexec do.

Different MPI implementations have different means of launching and controlling the processes. MPICH started with an infrastructure called MPD (Multi-Purpose Daemon or something). Then it switched to the newer Hydra process manager. Since Hydra does things differently than MPD, the Hydra-based mpiexec takes different command-line arguments than the MPD-based one and to make it possible for users to explicitly select the Hydra-based one, it is made available as mpiexec.hydra. The old one is called mpiexec.mpd. It is possible to have an MPICH-based MPI library that only provides the Hydra launcher and then mpiexec and mpiexec.hydra will be the same executable. Intel MPI is based on MPICH and its newer versions use the Hydra process manager.

Open MPI is built on top of Open Run-Time Environment (ORTE) and its own process launcher is called orterun. For compatibility, orterun is also symlinked as mpirun and mpiexec.

To summarise:

• mpiexec.something is a specific version of the MPI process launcher for a given implementation
• mpiexec and mpirun are the generic names, usually copies of or symbolic links to the actual launcher
• both mpiexec and mpirun should do the same
• some implementations name their launcher mpiexec, some name it mpirun, some name it both, and that is often the source of confusion when more than one MPI implementations are simultaneously available in the system paths (e.g. when installed from distro packages)