How to know when PBS batch jobs are complete

Question

I have a BASH script that submits multiple serial jobs to the PBS queueing system. Once the jobs are submitted the script ends. The jobs then run on a cluster and when they are all finished I can move on to the next step. A typical workflow might involve several of these steps.

My question:

Is there a way for my script not to exit upon completion of the submission, but rather to sleep until ALL jobs submitted by that script have completed on the cluster, only then exiting?

Answer 1

You are trying to establish a workflow, correct? The best way to do what you're attempting to accomplish would be to use job dependencies. Essentially, what you are trying to do is submit X number of jobs, and then submit more jobs that depend on the first set of jobs, and you can do this with job dependencies. There are different ways to do dependencies that you can read about in the previous link, but here's an example of submitting 3 jobs and then submitting 3 more that won't execute until after the first 3 have exited.

#first batch
jobid1=`qsub ...`
jobid2=`qsub ...`
jobid3=`qsub ...`

#next batch
depend_str="-W after:${jobid1} -W after:${jobid2} -W after:${jobid3}"
qsub ... $depend_str
qsub ... $depend_str
qsub ... $depend_str