Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Is there a "one-liner" for submitting many jobs to SLURM (similar to LSF)?

Tags:

slurm

sbatch

lsf

Can I submit "one-liners" to SLURM?

Using bsub from LSF and the standard Linux utility xargs, I can easily submit a separate job for uncompressing all of the files in a directory:

ls *.gz | sed 's/.gz$//g' | xargs -I {} bsub 'gunzip -c {}.gz > {}'


Using SLURM, I thought that srun or sbatch would be work, but to no avail:

ls *.gz | sed 's/.gz$//g' | xargs -I {}  srun 'gunzip -c {}.gz > {}'
gzip: srun: error: compute-node-01: task 0: Exited with exit code 1
stdin: unexpected end of file

ls *.gz | sed 's/.gz$//g' | xargs -I {}  sbatch 'gunzip -c {}.gz > {}'
sbatch: error: Unable to open file gunzip -c naive_S1_L001_R1_001.fastq.gz > naive_S1_L001_R1_001.fastq

I have seen bsub from LSF listed as equivalent to sbatch from SLURM, but so far it seems that they are only equivalent for submitting script files:

                  SLURM                    LSF
                  --------------------     ------------------
Job Submission    sbatch [script_file]     bsub [script_file]

Is there any other way to submit "one-liner" jobs with SLURM?

like image 868
Christopher Bottoms Avatar asked Apr 22 '15 22:04

Christopher Bottoms


People also ask

How do I submit multiple SLURM jobs?

Slurm wrapThe wrap feature of sbatch can be used to submit multiple jobs at once. Sbatch will wrap the specified command string in a simple "sh" shell script, and submit that script to the slurm controller.

How do you submit jobs on SLURM?

There are two ways of submitting a job to SLURM: Submit via a SLURM job script - create a bash script that includes directives to the SLURM scheduler. Submit via command-line options - provide directives to SLURM via command-line arguments.


1 Answers

Try using the wrap option of sbatch. Something like the following:

ls *.gz | sed 's/.gz$//g' | xargs -I {}  sbatch --wrap="gunzip -c {}.gz > {}"

From the man page for `sbatch`:
--wrap=<command string>
       Sbatch will wrap the specified command string in  a  simple  "sh"  shell
       script,  and submit that script to the slurm controller.  When --wrap is
       used, a script name and arguments may not be specified  on  the  command
       line; instead the sbatch-generated wrapper script is used.
like image 105
Carles Fenoy Avatar answered Oct 06 '22 23:10

Carles Fenoy