I am trying to adjust some bash scripts to make them run on a (pbs) cluster.
The individual tasks are performed by several script thats are started by a main script.
So far this main scripts starts multiple scripts in background (by appending &
) making them run in parallel on one multi core machine.
I want to substitute these calls by qsub
s to distribute load accross the cluster nodes.
However, some jobs depend on others to be finished before they can start.
So far, this was achieved by wait
statements in the main script.
But what is the best way to do this using the grid engine?
I already found this question as well as the -W after:jobid[:jobid...]
documentation in the qsub
man page but I hope there is a better way.
We are talking about several thousend jobs to run in parallel first and another set of the same size to run simultatiously after the last one of these finished.
This would mean I had to queue a lot of jobs depending on a lot of jobs.
I could bring this down by using a dummy job in between, doing nothing but depending on the first group of jobs, on which the second group could depend. This would decrease the number of dependencies from millions to thousands but still: It feeles wrong and I am not even sure if such a long command line would be accepted by the shell.
qwait -u <user>
)?qwait [-p <PID>]
)?Of course it would be possible to write something like this using qstat
and sleep
in a while
loop, but I guess this use case is important enough to have a built in solution and I was just incapable to figure that one out.
What would you recommend / use in such a situation?
Addendum I:
Since it was requested in a comment:
$ qsub --version
version: 2.4.8
Maybe also helpful to determine the exact pbs system:
$ qsub --help
usage: qsub [-a date_time] [-A account_string] [-b secs]
[-c [ none | { enabled | periodic | shutdown |
depth=<int> | dir=<path> | interval=<minutes>}... ]
[-C directive_prefix] [-d path] [-D path]
[-e path] [-h] [-I] [-j oe] [-k {oe}] [-l resource_list] [-m n|{abe}]
[-M user_list] [-N jobname] [-o path] [-p priority] [-P proxy_user] [-q queue]
[-r y|n] [-S path] [-t number_to_submit] [-T type] [-u user_list] [-w] path
[-W otherattributes=value...] [-v variable_list] [-V] [-x] [-X] [-z] [script]
Since the comments point to job arrays so far I searched the qsub
man page with the following results:
[...]
DESCRIPTION
[...]
In addition to the above, the following environment variables will be available to the batch job.
[...]
PBS_ARRAYID
each member of a job array is assigned a unique identifier (see -t)
[...]
OPTIONS
[...]
-t array_request
Specifies the task ids of a job array. Single task arrays are allowed.
The array_request argument is an integer id or a range of integers. Multiple ids or id ranges can be combined in a comman delimeted list. Examples : -t 1-100 or -t 1,10,50-100
[...]
Addendum II:
I have tried the torque solution given by Dmitri Chubarov but it does not work as described.
Without the job arrray it works as expected:
testuser@headnode ~ $ qsub -W depend=afterok:`qsub ./test1.sh` ./test2 && qstat
2553.testserver.domain
Job id Name User Time Use S Queue
----------------------- ---------------- --------------- -------- - -----
2552.testserver Test1 testuser 0 Q testqueue
2553.testserver Test2 testuser 0 H testqueue
testuser@headnode ~ $ qstat
Job id Name User Time Use S Queue
----------------------- ---------------- --------------- -------- - -----
2552.testserver Test1 testuser 0 R testqueue
2553.testserver Test2 testuser 0 H testqueue
testuser@headnode ~ $ qstat
Job id Name User Time Use S Queue
----------------------- ---------------- --------------- -------- - -----
2553.testserver Test2 testuser 0 R testqueue
However, using job arrays the second job won't start:
testuser@headnode ~ $ qsub -W depend=afterok:`qsub -t 1-2 ./test1.sh` ./test2 && qstat
2555.testserver.domain
Job id Name User Time Use S Queue
----------------------- ---------------- --------------- -------- - -----
2554-1.testserver Test1-1 testuser 0 Q testqueue
2554-2.testserver Test1-1 testuser 0 Q testqueue
2555.testserver Test2 testuser 0 H testqueue
testuser@headnode ~ $ qstat
Job id Name User Time Use S Queue
----------------------- ---------------- --------------- -------- - -----
2554-1.testserver Test1-1 testuser 0 R testqueue
2554-2.testserver Test1-2 testuser 0 R testqueue
2555.testserver Test2 testuser 0 H testqueue
testuser@headnode ~ $ qstat
Job id Name User Time Use S Queue
----------------------- ---------------- --------------- -------- - -----
2555.testserver Test2 testuser 0 H testqueue
I guess this is due to the lack of array indication in the job id that is returned by the first qsub
:
testuser@headnode ~ $ qsub -t 1-2 ./test1.sh
2556.testserver.domain
As you can see there is no ...[]
indicating this being a job array.
Also, in the qsub
output there are no ...[]
s but ...-1
and ...-2
indicating the array.
So the remaining question is how to format -W depend=afterok:...
to make a job depend on a specified job array.
Submitting the Job To do that, we use the 'qsub' command and give it the name of the . pbs script. Since the script contains the PBS directives for everything our job needs, we don't need to specify any other commandline options.
The PBS queueing system integrates all computers in the CIP-F room into one compute cluster. PBS schedules and executes jobs across all of the cluster nodes. Jobs can be submitted into the queueing systems from any node of the cluster. Important PBS commands are: qstat.
#PBS -l walltime=01:00:00 Sets the maximum wall-clock time during which this job can. run. ( walltime=hh:mm:ss) #PBS -l mem=n{mb|gb} Sets the maximum amount of memory allocated to the job.
The qsub command is used to submit jobs to the queue. job, as previously mentioned, is a program or task that may be assigned to run on a cluster system. qsub command is itself simple, however, it to actually run your desired program may be a bit tricky. is because qsub, when used as designed, will only run scripts.
Filling in following the solution suggested by Jonathan in the comments.
There are several resource managers based on the original Portable Batch System: OpenPBS, TORQUE and PBS Professional. The systems had diverged significantly and use different command syntax for newer features such as job arrays.
Job arrays are a convenient way to submit multiple similar jobs based on the same job script. Quoting from the manual:
Sometimes users will want to submit large numbers of jobs based on the same job script. Rather than using a script to repeatedly call qsub, a feature known as job arrays now exists to allow the creation of multiple jobs with one qsub command.
To submit a job array PBS provides the following syntax:
qsub -t 0-10,13,15 script.sh
this submits jobs with ids from 0,1,2,...,10,13,15.
Within the script the variable PBS_ARRAYID
carries the id of the job within the array and can be used to pick the necessary configuration.
Job array have their specific dependency options.
TORQUE resource manager that is probably used in the OP. There additional dependency options are provided that can be seen in the following example:
$ qsub -t 1-1000 script.sh
1234[].pbsserver.domainname
$ qsub -t 1001-2000 -W depend=afterokarray:1234[] script.sh
1235[].pbsserver.domainname
This will result in the following qstat
output
1234[] script.sh user 0 R queue
1235[] script.sh user 0 H queue
Tested on torque version 3.0.4
The full afterokarray syntax is in the qsub(1)
manual.
In PBS Professional dependencies can work uniformly on ordinary jobs and array jobs. Here is an example:
$ qsub -J 1-1000 -ry script.sh
1234[].pbsserver.domainname
$ qsub -J 1001-2000 -ry -W depend=afterok:1234[] script.sh
1235[].pbsserver.domainname
This will result in the following qstat
output
1234[] script.sh user 0 B queue
1235[] script.sh user 0 H queue
Array dependencies became available in Torque since version 2.5.3. Job arrays from version 2.5 are not compatible with job arrays in versions 2.3 or 2.4. In particular the []
syntax was introduced in Torque since version 2.5.
For torque versions prior to 2.5 a different solution may work that is based on submitting dummy delimeter jobs between batches of jobs to be separated.
It uses three dependency types: on
,before
, and after
.
Consider the following example
$ DELIM=`qsub -Wdepend=on:1000 dummy.sh `
$ qsub -Wdepend=beforeany:$DELIM script.sh
1001.pbsserver.domainname
... another 998 jobs ...
$ qsub -Wdepend=beforeany:$DELIM script.sh
2000.pbsserver.domainname
$ qsub -Wdepend=after:$DELIM script.sh
2001.pbsserver.domainname
...
This will result in the queue state like this
1000 dummy.sh user 0 H queue
1001 script.sh user 0 R queue
...
2000 script.sh user 0 R queue
2001 script.sh user 0 H queue
...
That is the job #2001 will run only after the previous 1000 jobs terminate. Probably the rudimentary job array facilities available in TORQUE 2.4 can be used as well to submit the script job.
This solution will also work for TORQUE version 2.5 and higher.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With