1. Get Duo
You must use Duo authentication to log on to Beta. Get more details and enroll here.
2. Get a Flux user account
All Flux users have access to Beta. You must establish an account on Flux by filling out this form.
3. Get an SSH Client & Connect to Beta
You must be on campus or on the VPN to connect to Beta. If you are trying to log in from off campus, or using an unauthenticated wireless network such as MGuest, you have a couple of options:
- Install VPN software on your computer
- SSH to login.itd.umich.edu and continue with the Linux instructions
Mac or Linux:
Open Terminal and type:
Windows (using PuTTY):
Download and install PuTTY here.
Launch PuTTY and enter beta.arc-ts.umich.edu as the host name, then click open.
All Operating Systems:
4. Get files
You can use SFTP (best for simple transfers of small files) or Globus (best for large files or a commonly used endpoint) to transfer data to your /home directory.
SFTP: Mac or Windows using FileZilla
- Open FileZilla and click the “Site Manager” button
- Create a New Site, which you can name “Beta” or something similar
- Select the “SFTP (SSH File Transfer Protocol)” option
- In the Host field, type beta.arc-ts.umich.edu
- Select “Interactive” for Logon Type
- In the User field, type your uniqname
- Click “Connect”
- Enter your Kerberos password
- Select your Duo method (1-3) and complete authentication
- Drag and drop files between the two systems
- Click “Disconnect” when finished
On Windows, you can also use WinSCP with similar settings, available alongside PuTTY here.
SFTP: Mac or Linux using Terminal
To copy a single file, type:
scp localfile email@example.com:~/remotefile
To copy an entire directory, type:
scp -r localdir firstname.lastname@example.org:~/remotedir
These commands can also be reversed in order to copy files from Beta to your machine:
scp -r email@example.com:~/remotedir localdir
You will need to authenticate via Duo to complete the file transfer.
Globus: Windows, Mac, or Linux
Globus is a reliable high performance parallel file transfer service provided by many HPC sites around the world. It enables easy transfer of files from one system to another, as long as they are Globus endpoints.
- The Globus endpoint for Beta is “umich#beta”.
How to use Globus
Globus Online is a web front end to the Globus transfer service. Globus Online accounts are free and you can create an account with your University identity.
- Set up your Globus account and learn how to transfer files using the Globus documentation. Select “University of Michigan” from the dropdown box to get started.
- Once you are ready to transfer files, enter “umich#beta” as one of your endpoints.
Globus Connect Personal
Globus Online also allows for simple installation of a Globus endpoint for Windows, Mac, and Linux desktops and laptops.
- Follow the Globus instructions to download the Globus Connect Personal installer and set up an endpoint on your desktop or laptop.
Batch File Copies
A non-standard use of Globus Online is that you can use it to copy files form one location to another on the same cluster. To do this use the same endpoint (umich#beta as an example) for both the sending and receiving machines. Setup the transfer and Globus will make sure the rest happens. The service will email you when the copy is finished.
Command Line Globus
There are Command line tools for Globus that are intended for advanced users. If you wish to use these, contact HPC support.
5. Submit a job
This is a simple guide to get your jobs up and running. For more advanced Slurm features, see the Slurm User Guide for Beta. If you are familiar with using the resource manager Torque, you may find the migrating from Torque to Slurm guide useful.
The sbatch command is used to submit a batch script to Slurm. It is designed to reject the job at submission time if there are requests or constraints that Slurm cannot fulfill as specified. This gives the user the opportunity to examine the job request and resubmit it with the necessary corrections. To submit a batch script simply run the following from a shared file system; those include your home directory, /scratch, and any directory under /nfs that you can normally use in a job on Flux. Do not submit jobs from /tmp or any of its subdirectories.
$ sbatch myJob.sh
The batch job script is composed of three main components:
- The interpreter used to execute the script
- #SBATCH directives that convey submission options
- The application(s) to execute along with its input arguments and options
#!/bin/bash # The interpreter used to execute the script #“#SBATCH” directives that convey submission options: #SBATCH --job-name=example_job #SBATCH --firstname.lastname@example.org #SBATCH --mail-type=BEGIN,END #SBATCH --cpus-per-task=1 #SBATCH --nodes=1 #SBATCH --ntasks-per-node=1 #SBATCH --mem-per-cpu=1000m #SBATCH --time=10:00 #SBATCH -A test #SBATCH -p standard #SBATCH --output=%x-%j.log # The application(s) to execute along with its input arguments and options: /bin/hostname sleep 60
Common Job Submission Options
|Option||Slurm Command (sbatch)||Beta Usage|
Available partitions: standard, gpu (GPU jobs only), largemem (RAM-intensive jobs only), debug
|Wall time limit||--time=<hh:mm:ss>||--time=02:00:00|
|Process count per node||--ntasks-per-node=<count>||--ntasks-per-node=1|
|Core count (per process)||--cpus-per-task=<cores>||--cpus-per-task=1|
|Minimum memory per processor||--mem-per-cpu=<memory>||--mem-per-cpu=1000m|
|Standard output/error file||--output=<file path> (path must exist)||--output=%x-%j.log
Will be in the directory from which you run
%x = job name (from –job-name)
%j = job ID
|Request event notification||
Note: multiple mail-type requests may be specified in a comma separated list:
|Email address||--mail-user=<email address>||--email@example.com|
Please note that if your job is set to utilize more than one node, make sure your code is MPI enabled in order to run across these nodes and you must use srun rather then mpirun or mpiexec. More advanced job submission options can be found in the Slurm User Guide for Beta.
An interactive job is a job that returns a command line prompt (instead of running a script) when the job runs. Interactive jobs are useful when debugging or interacting with an application. The srun command is used to submit an interactive job to Slurm. When the job starts, a command line prompt will appear on one of the compute nodes assigned to the job. From here commands can be executed using the resources allocated on the local node.
[user@beta-login ~]$ srun --pty /bin/bash srun: job 309 queued and waiting for resources srun: job 309 has been allocated resources [user@bn01 ~]$ hostname bn01.stage.arc-ts.umich.edu [user@bn01 ~]$
Jobs submitted with srun –pty /bin/bash will be assigned the cluster default values of 1 CPU and 1024MB of memory. If additional resources are required, they can be requested as options to the srun command. The following example job is assigned 2 nodes with 4 CPUS and 4GB of memory each:
[user@beta-login ~]$ srun --nodes=2 --ntasks-per-node=4 --mem-per-cpu=1GB --cpus-per-task=1 --pty /bin/bash srun: job 894 queued and waiting for resources srun: job 894 has been allocated resources [user@bn01 ~]$ srun hostname bn01.stage.arc-ts.umich.edu bn02.stage.arc-ts.umich.edu bn01.stage.arc-ts.umich.edu bn01.stage.arc-ts.umich.edu bn01.stage.arc-ts.umich.edu bn02.stage.arc-ts.umich.edu bn02.stage.arc-ts.umich.edu bn02.stage.arc-ts.umich.edu
In the above example srun is used within the job from the first compute node to run a command once for every task in the job on the assigned resources. srun can be used to run on a subset of the resources assigned to the job. See the srun man page for more details.
GPU and Large Memory Jobs
Jobs can request GPUs with the job submission options --partition=gpu and --gres=gpu:<count>. GPUs can be requested in both Batch and Interactive jobs.
Similarly, jobs can request nodes with large amounts of RAM with --partition=largemem. The largemem (and debug) partition’s nodes on Beta have the same configuration as standard nodes, so this partition is just for testing. Great Lakes will have high-RAM nodes on this partition.
Most of a job’s specifications can be seen by invoking scontrol show job <jobID>. More details about the job can be written to a file by using scontrol write batch_script <jobID> output.txt. If no output file is specified, the script will be written to slurm<jobID>.sh.
A job’s record remains in Slurm’s memory for 30 minutes after it completes. scontrol show job will return “Invalid job id specified” for a job that completed more than 30 minutes ago. At that point, one must invoke the sacct command to retrieve the job’s record from the Slurm database.