How to install python package/module in slurm

Question

This might be a very basic question, but this is my first time working with a slurm-cluster, and I don't want to mess anything up (the administrator is on vacation).

I have a python script that uses "import torch". When I run "sbatch myscript.sh", I get the output "ImportError: No module named 'torch'". I used pip inside the node to download the torch package, but after download I still get the importError.

How do I make my imports work? Should I download the sourcecode for the packages from github and upload them to my home directory on slurm? The guide I was given by the administrator didn't include information for this scenario, what am I missing?

Answer 1

You should import all these packages in your bash code instead. Here is an example:

#!/bin/bash
#SBATCH --account=def-someuser
#SBATCH --mem-per-cpu=1.5G      # increase as needed
#SBATCH --time=1:00:00

module load python/3.6
virtualenv --no-download $SLURM_TMPDIR/env
source $SLURM_TMPDIR/env/bin/activate
pip install --no-index --upgrade pip

pip install --no-index -r requirements.txt
python ...

You can put all dependencies inside the requirements file and install them all at once in the virtualenv. For more information take a look at this page.