How to use PBS Job Dependencies

PBS job dependencies allow you to prevent a job from running until another job has completed one of several actions (started running, completed, etc.). The format of a PBS dependency directive is

#PBS -W depend=dependency_expression

where dependency_expression is of the form

type:jobid[:jobid[:jobid...]

Type is one of the dependency directive types below and jobid[:jobid[:jobid...] is a list of at least one job ID upon which the dependency exists.

Directive Description
after This job may be scheduled after jobs jobid have started
afterok This job may be scheduled after jobs jobid have completed with no errors
afternotok This job may be scheduled after jobs jobid have completed with errors
afterany This job may be scheduled after jobs jobid have completed with or without errors
before After this job begins, jobs jobid may be scheduled
beforeok After this job completes without errors, jobs jobid may be scheduled
beforenotok After this job completes with errors, jobs jobid may be scheduled
beforeany After this job completes with or without errors, jobs jobid may be scheduled

There are also job array dependencies which are similar to the dependencies above but which cause the job to be scheduled only after an event has occurred for all of the jobs in some already-existing job array: afterstartarray, afterokarray, afternotok, afteranyarray, beforestartarray, beforeokarray, beforenotok, beforeanyarray. Job array dependencies can take an optional job count which will be used in place of all of the jobs in the array — for example, “-W depend=afterokarray:12345678[5]” will wait until five of the jobs in job array 12345678[] complete successfully before scheduling the new job.

If you wanted to script job depenencies you could generate each of your PBS scripts without the #PBS -W line. These scripts, say, program1.pbs, program2.pbs, and program3.pbs, should look like any normal PBS script that you use. You would then need to write a shell script, say, submit_dependencies.sh that contains

#!/bin/bash
# Submit the first job and save the JobID as JOBONE
JOBONE=$(qsub program1.pbs)
# Submit the second job, use JOBONE as depend, save JobID
JOBTWO=$(qsub -W depend=afterok:$JOBONE program2.pbs)
# Submit last job using JOBTWO as depend, do not need to save the JobID
qsub -W depend=afterok:$JOBTWO program3.pbs

You could then execute it with

$ bash ./submit_dependencies.sh

That shell script will submit each of your jobs with the proper dependency such that they will run one after the other until all of the jobs have completely without error or one of the jobs fails with an error.

For more information regarding job dependencies, please see the qsub manual page.