Running a helloworld.py program on a grid using slurm

Question

This is a long question, so I will give a summary first:

I am new in parallel programming and grid systems. I want to run the first example in http://jeremybejarano.zzl.org/MPIwithPython/introMPI.html in a grid I have an account. There is a C example provided by maintainers. I can run that. But with python version, I get all zeros for ranks. What may be the problem?

Long version of the question:

I have this code written in python 2.7 (+numpy+matplotlib+mayavi) that takes a lot of time to run. Since I have an account in some grid, I want to move the code there, and spend less time for waiting trials to finish.

Unfortunately, I am new to parallelism, grids, etc. I also do not have admin rights in the grid.

Some documentation is provided. The system uses SLURM. You prepare an sbatch file, and send the job by sbatch filename. There are this helloworld program example written in C:

#include <stdio.h>
#include <mpi.h>


int main (argc, argv)
     int argc;
     char *argv[];
{
  int rank, size;

  MPI_Init (&argc, &argv);      /* starts MPI */
  MPI_Comm_rank (MPI_COMM_WORLD, &rank);        /* get current process id */
  MPI_Comm_size (MPI_COMM_WORLD, &size);        /* get number of processes */
  printf( "Hello world from process %d of %d\n", rank, size );
  MPI_Finalize();
  return 0;
}

and the slurm file to run it that the admins provide:

#!/bin/bash
#SBATCH -M linux
#SBATCH -p mid1
#SBATCH -A username
#SBATCH -J mid1-test
#SBATCH -N 1 
#SBATCH -n 4 
#SBATCH --time=2-00:00:00 
#SBATCH --workdir=/truba_scratch/username/test
#SBATCH --output=slurm-%j.out
#SBATCH --error=slurm-%j.err
#SBATCH --mail-type=ALL
#SBATCH [email protected]


. /usr/share/Modules/init/sh
module load somehostithink/library/openmpi-1.4.3/gcc
export OMP_NUM_THREADS=1
echo "SLURM_NODELIST $SLURM_NODELIST"

mpirun helloworld

exit

I can submit by sbatch helloworld.slurm. At the end I see "hello worlds" from 0 to 3. E.g. rank takes different values for each process. Nice!

The problem is, there is no example program written in Python. The python in the system is old: 2.6.x. So I downloaded anaconda distribution and installed it in the user space. I tried to adapt the example helloworld.slurm above. I want to run the helloworld example here: http://jeremybejarano.zzl.org/MPIwithPython/introMPI.html . I can submit the job, but I get helloworlds with all the same rank, as you can see from output files. E.g. this does not seem to run on different processes.

Note: I get the same err with the c version, but it still runs and produces different ranks.

helloworld python version:

from mpi4py import MPI
comm = MPI.COMM_WORLD
rank = comm.Get_rank()
print "hello world from process ", rank

slurm file for python hello world (main.py)

#!/bin/bash
#SBATCH -M linux
#SBATCH -p mid1
#SBATCH -A username
#SBATCH -J mid1-test
#SBATCH -N 1
#SBATCH -n 4
#SBATCH --time=2-00:00:00
#SBATCH --workdir=/scratch/username/test
#SBATCH --output=slurm-%j.out
#SBATCH --error=slurm-%j.err
#SBATCH --mail-type=ALL
#SBATCH [email protected]

. /usr/share/Modules/init/sh
module load somehost/library/openmpi-1.4.3/gcc
export OMP_NUM_THREADS=1
echo "SLURM_NODELIST $SLURM_NODELIST"

mpirun /scratch/username/anaconda/bin/python /scratch/username/test/main.py

exit

The error file produced:

slurmd[shomehostithink]: task/cgroup: plugin not compiled with hwloc support, skipping affinity.

The output file produced:

SLURM_NODELIST hostidithink
hello world from process  0
hello world from process  0
hello world from process  0
hello world from process  0

So, what might be the cause of problem? How can I solve it?

I obviously sent a message to the admin, but he did not respond yet.

Answer 1

Under Linux, Anaconda's mpi4py is bundled with mpich (and it uses OpenMPI with OS X.) Since you are using an OpenMPI mpirun, the singleton effect could be explained by that. You have two options:

1. recompile Anaconda from source to use openmpi-1.4.3/gcc
2. try and find Anaconda's mpirun program and use that one, with something like /scratch/username/anaconda/bin/mpirun /scratch/username/anaconda/bin/python /scratch/username/test/main.py

The error you encounter with Slurm is stating that the cgroup Slurm plugin was not compiled with hwloc support, and hence, task affinity (pinning processes to cores) is not supported. It should not be the cause of the singleton problem.