How to use PBS Job Dependencies
PBS job dependencies allow you to prevent a job from running until another job has completed one of several actions (started running, completed, etc.). The format of a PBS dependency directive is
#PBS -W depend=dependency_expression
where dependency_expression is of the form
Type is one of the dependency directive types below and jobid[:jobid[:jobid...] is a list of at least one job ID upon which the dependency exists.
|after||This job may be scheduled after jobs jobid have started|
|afterok||This job may be scheduled after jobs jobid have completed with no errors|
|afternotok||This job may be scheduled after jobs jobid have completed with errors|
|afterany||This job may be scheduled after jobs jobid have completed with or without errors|
|before||After this job begins, jobs jobid may be scheduled|
|beforeok||After this job completes without errors, jobs jobid may be scheduled|
|beforenotok||After this job completes with errors, jobs jobid may be scheduled|
|beforeany||After this job completes with or without errors, jobs jobid may be scheduled|
There are also job array dependencies which are similar to the dependencies above but which cause the job to be scheduled only after an event has occurred for all of the jobs in some already-existing job array: afterstartarray, afterokarray, afternotok, afteranyarray, beforestartarray, beforeokarray, beforenotok, beforeanyarray. Job array dependencies can take an optional job count which will be used in place of all of the jobs in the array — for example, “-W depend=afterokarray:12345678” will wait until five of the jobs in job array 12345678 complete successfully before scheduling the new job.
If you wanted to script job depenencies you could generate each of your PBS scripts without the #PBS -W line. These scripts, say, program1.pbs, program2.pbs, and program3.pbs, should look like any normal PBS script that you use. You would then need to write a shell script, say, submit_dependencies.sh that contains
#!/bin/bash # Submit the first job and save the JobID as JOBONE JOBONE=$(qsub program1.pbs) # Submit the second job, use JOBONE as depend, save JobID JOBTWO=$(qsub -W depend=afterok:$JOBONE program2.pbs) # Submit last job using JOBTWO as depend, do not need to save the JobID qsub -W depend=afterok:$JOBTWO program3.pbs
You could then execute it with
$ bash ./submit_dependencies.sh
That shell script will submit each of your jobs with the proper dependency such that they will run one after the other until all of the jobs have completely without error or one of the jobs fails with an error.
For more information regarding job dependencies, please see the qsub manual page.