How to set several number of threads in hybrid openMP/MPI program

Question

I wrote a hybrid openMP/MPI program and I call it like the following

mpirun -np ncores --bind-to none -x OMP_NUM_THREADS=nthreads ./program

where ncores is the number of non shared memory processes (MPI) and nthreads is the number of shared memory threads (OpenMP).

That means in each of the ncores, the program will be executed on nthreads.

I do not want to have nthreads in every core but I am rather interested in varying that number for each core. e.g. if ncores=2 I´d like to set 2 threads on core 1 and 6 threads on core 2.

Is there a way to do that?

I am using Open MPI 1.10.3

Answer 1

You has global environment setting "OMP_NUM_THREADS" for all your MPI processes. This setting is the same for every of them, and you want to have different thread number for various mpi processes, so you should not set it as global variable.

I can suggest two variants. First is to delete -x option and replace your "./program" with some bash script "./program_script.sh", which will get process id from mpirun (use mpirun -np 2 env to find name of variable with process id) and set correct value of "OMP_NUM_THREADS" env variable for mpi process with this id and then start your ./program.

Second variant is to delete -x option and set threads number directly from your MPI processes using standard omp_set_num_threads() call of OpenMP Standard:

MPI_Comm_size(MPI_COMM_WORLD,&nproc);
MPI_Comm_rank(MPI_COMM_WORLD,&rank);
if(rank == 0) {
   omp_set_num_threads(2);
}
if(rank == 1) {
   omp_set_num_threads(6);
}

Don't forget to set correct CPU binding in your mpirun (check with --report-bindings, set with --bind-to or rankfile).

And third variant (when your thread count is less or equal to cpu core count only) is to delete -x option and just set correct cpu binding from your mpirun: enable core 0 and 1 for 1st MPI process and cores 2,3,4,5,6,8 for your 2nd MPI process. Most OpenMP libraries will sense allowed cpu set and start by default one thread per CPU.