Submitting jobs using Torque PBS


Torque PBS (just PBS, hereafter) is the Portable Batch System, and it controls the running of jobs on the Flux cluster. PBS queues, starts, controls, and stops jobs. It also has utilities to report on the status of jobs and nodes. PBS and Moab, the scheduling software, are the core software that keep jobs running on Flux.


PBS is available in the core software modules and is loaded automatically by the system at login. The PBS commands should automatically load when you log in. If, for some reason, you remove the module, or otherwise clear your modules, you can reload it with

$ module load torque

PBS overview

PBS is a batch job manager, where a job is some set of computing tasks to be performed. For most people, the primary uses will be to put jobs into the queue to be run, to check on the status of jobs, and to delete jobs. Most of the time, you will write a PBS script, which is just a text file – a shell script with PBS directives that the shell will interpret as comments – that contains information about the job and the commands that do the desired work.

You will find it convenient to name the PBS scripts in a consistent way, and some find that using the .pbs extension clearly identifies, and makes it easy to list, PBS scripts. We will use that convention for our examples. Before we get to the contents of a PBS script, we will show the three primary PBS commands, after which we will look at the PBS directives.

Submitting a PBS script

Suppose you have PBS script called test.pbs that you wish to run. The command qsub test.pbs will submit it to be run. The output will be a JobID, which is used if you need information about the job or wish to delete it. If you are having trouble with a job, it is always a good idea to include the JobID.

$ qsub test.pbs
Checking on job status

You only need to specify the numeric part of the JobID To get information about its status within the PBS system. For example,

$ qstat 1234567
Deleting a job

To delete a job, use

$ qdel 1234567

A PBS script template

There are many PBS directives you can use to specify the characteristics of a job and how you and the PBS system will interact. A PBS directive is on a line beginning with #PBS. We will show an idealized template for a PBS script to illustrate some of the most commonly used PBS directives, then we will explain them. The example PBS script, call it test.pbs contains these lines.

####  PBS preamble

#PBS -N PBS_test_script
#PBS -m abe

#PBS -A example_flux
#PBS -l qos=flux
#PBS -q flux

#PBS -l nodes=4:ppn=2,pmem=2gb
#PBS -l walltime=1:15:00
#PBS -j oe

####  End PBS preamble

if [ -s "$PBS_NODEFILE" ] ; then
    echo "Running on"

if [ -d "$PBS_O_WORKDIR" ] ; then
    echo "Running from $PBS_O_WORKDIR"

#  Put your job commands after this line
echo "Hello, world."

The lines that are not PBS directives but begin with the # character are comments and are ignored by PBS and by the shell. Once the first non-comment line is reached, and in the template above, the line that begins if, PBS stops looking for directives. It is, therefore, important to put all PBS directives before any commands.

You may find that grouping the PBS directives into blocks of related characteristics helps when reviewing a file for completeness and accuracy.

Roughly speaking, the three blocks, in order, are the attributes that control how you interact with the job, how the job gets paid for and routed, and what the resource characteristics of the job are.

All PBS directives start with #PBS, and each directive in a PBS script corresponds to a command line option to the qsub command. The #PBS -N PBS_test_script directive – which sets the job name – corresponds to adding -N PBS_test_script as an option to the qsub command. You can override the directives in the PBS script on the command line, or supplement them. So for example, you could use

$ qsub -N second_test test.pbs

and the name second_test would override the name set in test.pbs. The name should contain only letters, digits, the underscore, or the dash characters.

The next two directives (we will omit the #PBS portion for the remainder of this section), the -M and -m, control how PBS communicates job status to you by e-mail. Specify your own e-mail address after the -M ( is just a placeholder).

The three letters after the -m directive specify under what conditions PBS should send a notification: b is when a job begins, e is when a job ends, and a is when a job aborts. You can also specify n for none to suppress all e-mail from PBS.

The second block contains directives that have, roughly, to do with how your job is paid for and which sets of resources it runs against. The -A directive specifies the account to use. This is set up for the person paying for the use. The -l qos option will always be flux unless you receive specific instructions to the contrary.

The -q specifies which queue to use. In general, the queue will match the account suffix, so if the account is default_flux account, the queue would be specified using -q flux; similarly, if this were a large-memory account, default_fluxm, the queue should be specified -q fluxm; etc.

The exception to this rule is if you will be using software that is restricted to on-campus use, and you submit jobs from the node, in which case, the queue would be specified as -q flux-oncampus. Jobs can submitted to the flux-oncampus queue only from the flux-campus-login node.

The last block contains the directives that determine the environment in which your job runs and what resources are allocated to it. The most commonly changed are the processor count and layout, the memory, and the maximum amount of time the job can run.

Let us save the most complicated options for last, and review these in reverse order. The -V option instructs PBS to set the environment on all of the compute nodes assigned to a job to be the same as the environment that was in effect when you submitted the job. This is very important to include to make sure that all the paths to programs and libraries, and any important environment variables, are set everywhere. Many obscure error messages are due to this option not being used.

PBS will normally create a separate file for output and for errors. These are named job_name.oXXXXXXXX and job_name.eXXXXXXXX where job_name is that name you specify with the -N option, XXXXXXXX is the numeric JobID PBS assigned the job, and the o and e represent output and error, respectively. If there are errors, or if your program writes informative information to the error stream, then it can be helpful to combined the output and error stream so that the error messages appear in context with the output. It can be very difficult otherwise to determine exactly where in the course of a job an error occurred. This is what the -j oeoption does: it joins the output and error streams into the output stream (specifying it as eo would combine them into the error stream).

There are many options that can be specified with -l (dash ‘ell’). One of the simplest is walltime which specifies the maximum clock time that your job will be allowed to run. Time is specified as day, hours, minutes, seconds, dd:hh:mm:ss. So, 15:00 requests 15 minutes (this should be the minimum that you request for a single job), 2:30:00 would request two and one-half hours, and 7:00:00:00 would request one week. Longer times are harder to schedule than shorter times, but it is better to ask for too much time than too little, so your job will finish.

Finally, we get to the most complex of the -l options: selecting the number of nodes, processors, and memory. In the example above, the request is for nodes=4:ppn=2,pmem=2gb, which translates as “assign me 4 machines, each with 2 processors, and each processor should have 2 GB of memory”.

It does not specify that exactly; instead, it specifies a sort of “worst-case” scenario. It really says that you would take up to four physical machines, each of which has at least two processors each. You could end up with eight processors on one machine with that request.