How to set environment variables on compute nodes in an MPI job

Question

I don't understand how the environment is set on compute nodes when running with MPI under a scheduler.

I do:

mpirun -np 1 --hostfile ./hostfile foo.sh

with foo.sh:

#!/usr/bin/env zsh                                                                                                  
echo $LD_LIBRARY_PATH

Then I do not recover the LD_LIBRARY_PATH I have got in an interactive shell... What are the initialization files that are executed/sourced at connection with MPI?

note: I am under zsh, and I tried to put things in .zprofile or .zshenv instead of .zshrc, but it doesn't seem to make a change... My LD_LIBRARY_PATH is set in a .profile which is sourced by a .bashrc which is sourced by the .zshrc.

Answer 1

Some MPI implementations have an -x flag for mpirun for this, e.g. OpenMPI:

-x <env>

Export the specified environment variables to the remote nodes before executing the program. Only one environment variable can be specified per -x option. Existing environment variables can be specified or new variable names specified with corresponding values. For example:

% mpirun -x DISPLAY -x OFILE=/tmp/out ...

The parser for the -x option is not very sophisticated; it does not even understand quoted values. Users are advised to set variables in the environment, and then use -x to export (not define) them.

If your's does not, you'll have to explicitly set the environment variables in your job script, e.g.

export LD_LIBRARY_PATH=...

Answer 2

You could specify the number of threads per mpi using end with following command as well.

env OMP_NUM_THREADS=n PARALLEL=n mpirun -np m program.exe < input.file > output.file &

where n and m are the number of threads and number of CPU cores.

Example:

env OMP_NUM_THREADS=2 PARALLEL=2 mpirun -np 12 program.exe < input.file > output.file &