Using Python multiprocessing in Slurm, and which combination of ntasks or ncpus I need

Question

I'm trying to run a python script on a slurm cluster, and I'm using python's built-in multiprocessing module.

I'm using quite a simple set up, where for testing purpose, the example is:

len(arg_list)
Out[2]: 5

threads = multiprocessing.Pool(5)
output = threads.map(func, arg_list)

So func is applied 5 times in parallel on 5 arguments in arg_list. What I want to know is how to allocate the correct amount of cpu's/tasks in slurm for this to work as expected. This is what the relevant part of my slurm batch script looks like:

#!/bin/bash

# Runtime and memory
#SBATCH --time=90:00:00
#SBATCH --mem-per-cpu=2G

# For parallel jobs
#SBATCH --cpus-per-task=10
##SBATCH --nodes=2
#SBATCH --ntasks=1
##SBATCH --ntasks-per-node=4  

#### Your shell commands below this line ####

srun ./script_wrapper.py 'test'

As you can see, at the moment I have ntasks=1 and cpus-per-task=10. Note that the main bulk of func contains a scipy routine which tends to run on two cores (i.e uses 200% cpu usage, which is why I want 10 cpus and not 5).

Is this the correct way to allocate resources for my purposes, because at the moment the job takes a lot longer than expected (more like it's running in a single thread).

Do I need to set ntasks=5 instead? Because my impression from online documentation was that ntasks=5 would instead call srun ./script_wrapper.py 'test' five times instead, which is not what I want. Am I right in that assumption?

Also, is there a way to easily check stuff like CPU usage and all the process id's of the python tasks called by multiprocessing.Pool? At the moment I'm trying with sacct -u <user> --format=JobID,JobName,MaxRSS,Elapsed,AveCPU, but the AveCPU and MaxRSS fields always come up empty for some reason (?) and while I see the first script as a process, I don't see the 5 others that should be called by multiprocessing. Example:

       JobID    JobName     MaxRSS    Elapsed     AveCPU 
------------ ---------- ---------- ---------- ---------- 
16260892             GP              00:13:07            
16260892.0   script_wr+              00:13:07            

Answer 1

Your Slurm task allocation looks correct to me. Python's multiprocessing will only run on a single machine, and it looks to me like you're allocating the 10 CPUs on one node correctly. What might be causing the problem is that multiprocessing's Pool.map by default works on "chunks" of the input list rather than one element at a time. It does this to minimise overhead when tasks are short. To force multiprocessing to work on one element of the list at a time, set the chunksize parameter of the map to 1, e.g.

threads.map(func, arglist, 1)

See the multiprocessing documentation for more information.

Because you say that you're using a multithreaded version of SciPy, you may also want to check the relevant threading level for the underlying library. For instance, if your SciPy has been built against Intel Math Kernel Library, try setting the OMP_NUM_THREADS andMKL_NUM_THREADS environment variables to make sure it's using no more than 2 threads per process and making full use (and not over-use) of your allocated SLURM resources.

EDIT: sacct is only going to give you running times for any processes that were launched directly by srun, and not for any subprocesses. Hence in your case you'll only have the one process from the single srun command. To monitor the subprocesses you may have to look into monitoring tools that operate at a system level rather than through Slurm.