Invalid job array specification in slurm

Question

I am submitting a toy array job in slurm. My command line is

$ sbatch -p development -t 0:30:0 -n 1 -a 1-2 j1

where j1 is script:

#!/bin/bash
echo job id is $SLURM_JOB_ID
echo array job id is $SLURM_ARRAY_JOB_ID
echo task id id $SLURM_ARRAY_TASK_ID

When I submit this, I get an error:

--> Verifying valid submit host (login1)...OK
--> Verifying valid jobname...OK
--> Enforcing max jobs per user...OK
--> Verifying availability of your home dir (/home1/03400/myname)...OK
--> Verifying availability of your work dir (/work/03400/myname)...OK
--> Verifying availability of your scratch dir (/scratch/03400/myname)...OK
--> Verifying valid ssh keys...OK
--> Verifying access to desired queue (development)...OK
--> Verifying job request is within current queue limits...OK
--> Checking available allocation (PRJ-1234)...OK
sbatch: error: Batch job submission failed: Invalid job array specification

The same job works fine without the array specification:

$ sbatch -p development -t 0:30:0 -n 1 j1

Answer 1

This post is a bit old, but in case it happens for other people, I have had the same issue but the accepted answer did not suggest what was the problem in my case.

This error (sbatch: error: Batch job submission failed: Invalid job array specification) can also be raised when the array size is too large.

From https://slurm.schedmd.com/slurm.conf.html

MaxArraySize

The maximum job array size. The maximum job array task index value will be one less than MaxArraySize to allow for an index value of zero. Configure MaxArraySize to 0 in order to disable job array use. The value may not exceed 4000001. The value of MaxJobCount should be much larger than MaxArraySize. The default value is 1001.

To check the value, the slurm.conf file should be accessible by all slurm users (still according to 1) and may be found somewhere near /etc/slurm.conf (see https://slurm.schedmd.com/slurm.conf.html#lbAM, in my case I found it at path /etc/slurm/slurm.conf).

Answer 2

The syntax for your array specification is correct. But the printout you paste is not standard Slurm, I guess you are working on Stampede ; they have their own sbatch wrapper.

What you could do is use the -vvv option to sbatch to see exactly what Slurm sees:

$ sbatch -vvv -p development -t 0:30:0 -n 1 -a 1-2 j1 |& grep array

This should return

sbatch: array             : 1-2

and if it does not it means the information is somehow lost somewhere.

What you can try is remove the array specification from the submission command line and insert it in the submission script, like this:

$ sbatch -p development -t 0:30:0 -n 1 j1

with j1 being

#!/bin/bash
#SBATCH -a 1-2
echo job id is $SLURM_JOB_ID
echo array job id is $SLURM_ARRAY_JOB_ID
echo task id id $SLURM_ARRAY_TASK_ID

The next step is to contact the system administrators with the information you will get from running the above tests and ask for help.