Launcher is a utility developed at the Texas Advanced Computing Center (TACC) to allow
running of many serial or multi-threaded applications as a single, large batch submission.

Using Launcher

Launcher is available on Flux as a software module. To enable it, load it as you
would any other.

module load launcher

This will add the installation directory to your $PATH as well as set the following
environment variables:

  • LAUNCHER_DIR – installation directory. Only necessary for Launcher’s internal use.
  • LAUNCHER_PLUGIN_DIR – plugin directory for integration with various resource managers.
    Only necessary for Launcher’s internal use.
  • LAUNCHER_RMI – resource manager integration plugin to use.
  • LAUNCHER_SCHED – scheduling method to use. Can be overridden by the user.

For more information on software modules, see the module documentation here.

Submitting a Launcher Job

We will ultimately be using a job submission script as normal. For information on job
submission syntax, see the Torque documentation here.

First, set up your job file. This should be a plain text file with the commands that you
need to execute, one command per line. For example:

python myscript.py input1
python myscript.py input2
python myscript.py input3
...

Assume the name of this file is $HOME/example-job-file. Then, in your job submission script,
you would need to set the variable LAUNCHER_JOB_FILE to this value:

export LAUNCHER_JOB_FILE=$HOME/example-job-script

The last line of your submission script would be to call Launcher’s paramrun exectuable:

paramrun

This executable has been added to your path by the module. Tying everything together, your
job submission script for this run would look like this:

#!/bin/bash
#PBS -N Parametric
#PBS -j oe
#PBS -l nodes=2:ppn=2,pmem=4gb,walltime=00:10:00
#PBS -V
#PBS -A test_flux
#PBS -q flux

export LAUNCHER_JOB_FILE=$HOME/example-job-file

paramrun

A Note About Job Layouts

PBS/Torque offers a variety of ways to specify the layout of your jobs using the -l option. It is
recommended to use the nodes=X:ppn=Y syntax when using Launcher. Alternatively, you could use
procs=X,tpn=Y (where tpn divides evenly into procs). It is NOT
recommended to use procs=X alone, as this could result in a strange geometry. Launcher tries to match
the geometry of your resource request, and having an uneven geometry that could result from using only procs
could leave some cores idle, causing your job to run unnecessarily slowly
. For more information on job layouts, see
this link.

Scheduling Methods

From the TACC documentation:

The launcher has three available behaviors for scheduling jobs, available by setting the
environment variable $LAUNCHER_SCHED: (descriptions below assume k = task, p = num. procs,
n = num. jobs)

  • dynamic (default) – each task k executes the first available unclaimed line
  • interleaved – each task k executes every (k+p)th line
  • block – each task k executes lines [ k(n/p) + 1, (k+1)(n/p) ]

Please note that only the “block” and “interleaved” methods are currently supported by ARC-TS.
As such, $LAUNCHER_SCHED is set to “block” by default in the module. Feel free to set this
to “interleaved” if you think it would be better for your workflow, but “dynamic” will not
behave in the way that you would expect. We plan to support this feature in the future to
offer flexibility.

To choose a different scheduling method, you could set $LAUNCHER_SCHED in your job script
sometime before calling paramrun:

export LAUNCHER_SCHED="interleaved"

Referencing Launcher

From the TACC documentation:

If you are using Launcher, please remember to make a reference to it when publishing results.
The file paper/paper.bib contains the BibTeX-formatted citation list. Please reference
entry Wilson:2014:LSF:2616498.2616533 (i.e., in LaTeX: \cite{Wilson:2014:LSF:2616498.2616534}).

To access the citation list from within the ARC-TS module, you can copy it to a directory of your
choice ($HOME, for example):

cp $LAUNCHER_DIR/paper/paper.bib ~

Examples

Working examples showing how to use Launcher are available in /scratch/data/examples/launcher. There are two example directories at that location. To use either, start by loading the Launcher module.

$ module load launcher

Example 1

This is a simple example that writes many “Hello, world!” messages in parallel.

  1. Copy the example directory to your home directory and cd into it.
    $ cp -R /scratch/data/examples/launcher/example1 ~
    $ cd ~/example1
  2. Edit the file helloworldshort.pbs. You only need to add a working Flux account to the PBS script (the -A option).
  3. Submit the job.
    $ qsub helloworldshort.pbs

This job should finish quickly, and an output file should appear in your current directory with the results printed.

Example 2

This is a slightly more complicated example that perhaps demonstrates more realistic usage.

  1. Copy the example directory to your home directory and cd into it.
    $ cp -R /scratch/data/examples/launcher/example2 ~
    $ cd ~/example2
  2. Run the create-jobfile.sh script. It is important to run this script inside of your copied example directory.
    $ ./create-jobfile.sh
  3. Edit the file process-data.pbs. You only need to add a working Flux account to the PBS script (the -A option).
  4. Submit the job.
    $ qsub process-data.pbs

This job should also finish relatively quickly.