Can I submit "one-liners" to SLURM?
Using bsub
from LSF and the standard Linux utility xargs
, I can easily submit a separate job for uncompressing all of the files in a directory:
ls *.gz | sed 's/.gz$//g' | xargs -I {} bsub 'gunzip -c {}.gz > {}'
Using SLURM, I thought that srun
or sbatch
would be work, but to no avail:
ls *.gz | sed 's/.gz$//g' | xargs -I {} srun 'gunzip -c {}.gz > {}'
gzip: srun: error: compute-node-01: task 0: Exited with exit code 1
stdin: unexpected end of file
ls *.gz | sed 's/.gz$//g' | xargs -I {} sbatch 'gunzip -c {}.gz > {}'
sbatch: error: Unable to open file gunzip -c naive_S1_L001_R1_001.fastq.gz > naive_S1_L001_R1_001.fastq
I have seen bsub
from LSF listed as equivalent to sbatch
from SLURM, but so far it seems that they are only equivalent for submitting script files:
SLURM LSF
-------------------- ------------------
Job Submission sbatch [script_file] bsub [script_file]
Is there any other way to submit "one-liner" jobs with SLURM?
Slurm wrapThe wrap feature of sbatch can be used to submit multiple jobs at once. Sbatch will wrap the specified command string in a simple "sh" shell script, and submit that script to the slurm controller.
There are two ways of submitting a job to SLURM: Submit via a SLURM job script - create a bash script that includes directives to the SLURM scheduler. Submit via command-line options - provide directives to SLURM via command-line arguments.
Try using the wrap option of sbatch
. Something like the following:
ls *.gz | sed 's/.gz$//g' | xargs -I {} sbatch --wrap="gunzip -c {}.gz > {}"
--wrap=<command string>
Sbatch will wrap the specified command string in a simple "sh" shell
script, and submit that script to the slurm controller. When --wrap is
used, a script name and arguments may not be specified on the command
line; instead the sbatch-generated wrapper script is used.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With