Selecting from multiple SLURM GPU resources

Question

I'm submitting jobs to a cluster via SLURM scheduler, and let's say I have access to 5 types of GPUs in my cluster. They are GPUs of type A,B,C,D,E. I would like to submit a job that requests the use of GPUs of type A or B or C but NOT of type D or E. So I need some type of OR logic with the --gres flag.

As a concrete example, here is what it looks like when I request a gpu of a single type (in this case, an RTX 2080): qlogin -p gpu --gres=gpu:rtx2080:1 --mem=8g -c 2 I'd like to do this but allowing SLURM to pick from a list of allowed GPU types

Answer 1

Slurm does not have that option at this time.

One workaround is for the system administrator to setup features of the node with the GPU type to allow a request such as:

qlogin -p gpu --gres=gpu:1 --constraint="rtx2080|rtx3090" --mem=8g -c 2

(assuming qlogin uses the same options as sbatch)

If that is not possible, you can submit as many job as there are types of GPU that you want, all with the same --job-name=<SOME_NAME> and the --dependency=singleton option. Then you use whichever job starts first and cancel the other with

scancel --jobname <SOME_NAME> --state=PENDING

The --dependency option makes sure only one job is started at a time.