Advanced batch computing on the Flux cluster

By | | No Comments

Overview

This course will cover some more advanced topics in cluster computing on the U-M Flux Cluster. Topics to be covered include a review of common parallel programming models and basic use of Flux; dependent and array scheduling; advanced troubleshooting and analysis using checkjob, qstat, and other tools; and parallel debugging and profiling of C and Fortran code, including logging, gdb (line-oriented debugging), ddt (GUI-based debugging) and map (GUI-based profiling) of MPI and OpenMP programs. We will issue you a temporary allocation to use for the course, or you can use your existing Flux allocations, if any.

Prerequisites

Introduction to Batch Computing on Flux or equivalent. This course assumes familiarity with the Linux command line, text editing on Linux, and a basic understanding of Flux including how to submit and track jobs.

click here to register

Instructors

Dr. Charles J Antonelli
Advocacy and Research Support
LSA Information Technology

Charles is a High Performance Computing Consultant in the Advocacy and Research Support Group of LSAIT at the University of Michigan, where he is responsible for high performance computing support and education, and is an advocate to the Departments of History and Communications. Prior to this, he built a parallel data ingestion component of a novel earth science data assimilation system, a secure packet vault, and worked on the No. 5 ESS Switch at Bell Labs in the 80s. He has taught courses in operating systems, distributed file systems, C++ programming, security, and database application design.

Dr. Alexander Gaenko
Technical Services
Advanced Research Computing

Mark Champe
Advocacy and Research Support
LSA Information Technology

Materials

Course Preparation

In order to participate successfully in the workshop exercises, you must have a Flux user account, a Flux allocation (one is created for each workshop), and be enrolled in Duo. The user account allows you to log in to the cluster, create, compile, and test applications, and prepare Flux jobs for submission. The allocation allows you to submit those jobs, executing those applications in parallel on the cluster and charging their resource use against the allocation. Duo is required to help authenticate you to the cluster.


Flux user account

A single Flux user account can be used to prepare and submit jobs using various allocations. If you already already possess a user account, you can use it for this course, you can skip to “Flux allocation” below. If not, please visit https://arc-ts.umich.edu/fluxform to obtain one. A user account is free to members of the University community. Please note that obtaining an account requires human processing, so be sure to do this at least two business days before class begins.


Flux allocation

We create an allocation for the workshop so you can run jobs on the cluster during the workshop and for one day after for those who would like additional practice. The workshop allocation is quite limited and is intended only to run examples to help you cement the details of job submission and management. If you already have an existing Flux allocation, you can use that, though if there are any issues with that allocation, we will ask you to use the workshop allocation.


Duo Authentication

Duo two-factor authentication is required to log in to the cluster. When logging in, you will need to type your UMICH password as well as authenticate through Duo in order to access Flux.

If you need to enroll in Duo, follow the instructions at Getting Started: How to Enroll in Duo.

Please enroll in Duo before you come to class.

Laptop Preparation

You do not need to bring your own laptop to class. The classroom contains Windows computers, which require your uniqname and UMICH password to login, and that have all necessary software pre-loaded.

If you want to use a laptop for the course, you are welcome to do so:  please see our web page on Preparing your laptop to use Flux. However, if there are problems connecting your laptop, you will be asked to switch to the provided computer for the class. We cannot stop to debug connection issues with personal or departmental laptops during the class.

More help

Please use our class mailing list hpc201course@umich.edu for questions, comments, or to seek further assistance.

Advanced batch computing on the Flux cluster

By | | No Comments

Overview

This course will cover some more advanced topics in cluster computing on the U-M Flux Cluster. Topics to be covered include a review of common parallel programming models and basic use of Flux; dependent and array scheduling; advanced troubleshooting and analysis using checkjob, qstat, and other tools; and parallel debugging and profiling of C and Fortran code, including logging, gdb (line-oriented debugging), ddt (GUI-based debugging) and map (GUI-based profiling) of MPI and OpenMP programs. We will issue you a temporary allocation to use for the course, or you can use your existing Flux allocations, if any.

Prerequisites

Introduction to Batch Computing on Flux or equivalent. This course assumes familiarity with the Linux command line, text editing on Linux, and a basic understanding of Flux including how to submit and track jobs.

click here to register

Instructors

Dr. Charles J Antonelli
Advocacy and Research Support
LSA Information Technology

Charles is a High Performance Computing Consultant in the Advocacy and Research Support Group of LSAIT at the University of Michigan, where he is responsible for high performance computing support and education, and is an advocate to the Departments of History and Communications. Prior to this, he built a parallel data ingestion component of a novel earth science data assimilation system, a secure packet vault, and worked on the No. 5 ESS Switch at Bell Labs in the 80s. He has taught courses in operating systems, distributed file systems, C++ programming, security, and database application design.

Dr. Alexander Gaenko
Technical Services
Advanced Research Computing

Mark Champe
Advocacy and Research Support
LSA Information Technology

Materials

Course Preparation

In order to participate successfully in the workshop exercises, you must have a Flux user account, a Flux allocation (one is created for each workshop), and be enrolled in Duo. The user account allows you to log in to the cluster, create, compile, and test applications, and prepare Flux jobs for submission. The allocation allows you to submit those jobs, executing those applications in parallel on the cluster and charging their resource use against the allocation. Duo is required to help authenticate you to the cluster.


Flux user account

A single Flux user account can be used to prepare and submit jobs using various allocations. If you already already possess a user account, you can use it for this course, you can skip to “Flux allocation” below. If not, please visit https://arc-ts.umich.edu/fluxform to obtain one. A user account is free to members of the University community. Please note that obtaining an account requires human processing, so be sure to do this at least two business days before class begins.


Flux allocation

We create an allocation for the workshop so you can run jobs on the cluster during the workshop and for one day after for those who would like additional practice. The workshop allocation is quite limited and is intended only to run examples to help you cement the details of job submission and management. If you already have an existing Flux allocation, you can use that, though if there are any issues with that allocation, we will ask you to use the workshop allocation.


Duo Authentication

Duo two-factor authentication is required to log in to the cluster. When logging in, you will need to type your UMICH password as well as authenticate through Duo in order to access Flux.

If you need to enroll in Duo, follow the instructions at Getting Started: How to Enroll in Duo.

Please enroll in Duo before you come to class.

Laptop Preparation

You do not need to bring your own laptop to class. The classroom contains Windows computers, which require your uniqname and UMICH password to login, and that have all necessary software pre-loaded.

If you want to use a laptop for the course, you are welcome to do so:  please see our web page on Preparing your laptop to use Flux. However, if there are problems connecting your laptop, you will be asked to switch to the provided computer for the class. We cannot stop to debug connection issues with personal or departmental laptops during the class.

More help

Please use our class mailing list hpc201course@umich.edu for questions, comments, or to seek further assistance.

Introduction to the Flux cluster and batch computing

By | | No Comments

Overview

This workshop will provide a brief overview of the components of the Flux Cluster. The main body of the workshop will cover the resource manager and scheduler, creating submissions scripts to run jobs and the options available in them, and hands-on experience. By the end of the workshop, every participant should have created a submission script, submitted a job, tracked its progress, and collected its output. Participants will have several working examples from which to build their own submissions scripts in their own home directories.

Prerequisites

Obtain a user account on Flux. If you do not have a Flux user account, click here to go to the account application page at: https://arc-ts.umich.edu/fluxform/

Register for Duo authentication. See below for details.

This course assumes familiarity with the Linux command line as might be got from the CSCAR/ARC-TS workshop Introduction to the Linux Command Line. In particular, participants should understand how files and folders work, be able to create text files using the nano editor, be able to create and remove files and folders, and understand what input and output redirection are and how to use them.

click here to register

Instructor

Bennet Fauber
Scientific Applications Analyst
Advanced Research Computing — Technology Services

Bennet Fauber is a member of the technical team that administers the Flux cluster.

Materials

In-class handouts

Supplementary materials

Course Preparation

In order to participate successfully in the workshop exercises, you must have a Flux user account and be registered for the Duo authentication service. The user account allows you to log in to the cluster, create, compile, and test applications, and prepare Flux jobs for submission. Information about registering for Duo is at the ITS Two Factor Authentication web page.

Flux user account

A single Flux user account can be used to prepare and submit jobs using various allocations. If you already already possess a user account, you can use it for this course, you can skip to “Flux allocation” below. If not, please visit https://arc-ts.umich.edu/fluxform to obtain one. A user account is free to members of the University community. Please note that obtaining an account requires human processing, so be sure to do this at least two business days before class begins.

Flux allocation

We create an allocation for the workshop so you can run jobs on the cluster during the workshop and for one day after for those who would like additional practice. The workshop allocation is quite limited and is intended only to run examples to help you cement the details of job submission and management. If you already have an existing Flux allocation, you can use that, though if there are any issues with that allocation, we will ask you to use the workshop allocation.

Laptop Preparation

The classroom contains Windows computers, which require your uniqname and UMICH password to login, and which have all necessary software preloaded.

If you want to use a laptop for the course, you are welcome to do so. However, if there are problems connecting your laptop, you will be asked to switch to the provided computer for the class. We will try to help resolve issues after the workshop.

If you want to try to prepare your laptop for use in the class, please see our web page on Preparing your laptop to use Flux. If you have a Windows computer, we recommend that you install PuTTY/WinSCP from the Blue Disc web site. Macintosh computers typically have the minimum requirements installed as part of the system.

More help

Please use our class mailing list hpc101course@umich.edu for questions, comments, or to seek further assistance.

Introduction to the Flux cluster and batch computing

By | | No Comments

Overview

This workshop will provide a brief overview of the components of the Flux Cluster. The main body of the workshop will cover the resource manager and scheduler, creating submissions scripts to run jobs and the options available in them, and hands-on experience. By the end of the workshop, every participant should have created a submission script, submitted a job, tracked its progress, and collected its output. Participants will have several working examples from which to build their own submissions scripts in their own home directories.

Prerequisites

Obtain a user account on Flux. If you do not have a Flux user account, click here to go to the account application page at: https://arc-ts.umich.edu/fluxform/

Register for Duo authentication. See below for details.

This course assumes familiarity with the Linux command line as might be got from the CSCAR/ARC-TS workshop Introduction to the Linux Command Line. In particular, participants should understand how files and folders work, be able to create text files using the nano editor, be able to create and remove files and folders, and understand what input and output redirection are and how to use them.

click here to register

Instructor

Bennet Fauber
Scientific Applications Analyst
Advanced Research Computing — Technology Services

Bennet Fauber is a member of the technical team that administers the Flux cluster.

Materials

In-class handouts

Supplementary materials

Course Preparation

In order to participate successfully in the workshop exercises, you must have a Flux user account and be registered for the Duo authentication service. The user account allows you to log in to the cluster, create, compile, and test applications, and prepare Flux jobs for submission. Information about registering for Duo is at the ITS Two Factor Authentication web page.

Flux user account

A single Flux user account can be used to prepare and submit jobs using various allocations. If you already already possess a user account, you can use it for this course, you can skip to “Flux allocation” below. If not, please visit https://arc-ts.umich.edu/fluxform to obtain one. A user account is free to members of the University community. Please note that obtaining an account requires human processing, so be sure to do this at least two business days before class begins.

Flux allocation

We create an allocation for the workshop so you can run jobs on the cluster during the workshop and for one day after for those who would like additional practice. The workshop allocation is quite limited and is intended only to run examples to help you cement the details of job submission and management. If you already have an existing Flux allocation, you can use that, though if there are any issues with that allocation, we will ask you to use the workshop allocation.

Laptop Preparation

The classroom contains Windows computers, which require your uniqname and UMICH password to login, and which have all necessary software preloaded.

If you want to use a laptop for the course, you are welcome to do so. However, if there are problems connecting your laptop, you will be asked to switch to the provided computer for the class. We will try to help resolve issues after the workshop.

If you want to try to prepare your laptop for use in the class, please see our web page on Preparing your laptop to use Flux. If you have a Windows computer, we recommend that you install PuTTY/WinSCP from the Blue Disc web site. Macintosh computers typically have the minimum requirements installed as part of the system.

More help

Please use our class mailing list hpc101course@umich.edu for questions, comments, or to seek further assistance.

Introduction to the Linux Command Line

By | | No Comments

This course will familiarize the student with the basics of accessing and interacting with Linux computers using the GNU/Linux operating system’s Bash shell, also generically referred to as “the command line”. Topics include: a brief overview of Linux, the Bash shell, navigating the file system, basic commands, shell redirection, permissions, processes, and the command environment. The workshop will also provide a quick introduction to nano a simple text editor that will be used in subsequent workshops to edit files.

click here to register

Instructor

Kenneth Weiss
IT Project Senior Manager
Medical School Information Services (MSIS)

Ken is a High Performance Computing Consultant in the Computational Research Consulting Division of MSIS at the University of Michigan. He works with a team of IT specialists to provide high performance computing support and training for the Medical School. Prior to this, he spent 21 years managing research computing, including an HPC cluster, for Dr. Charles Sing in the Human Genetics Department.

Materials

Course Preparation

You must register at least three full days prior to the event so that we have time to insure you have proper UM credentials for the workshop. This allows enough time for you to get your account adjusted by ITS in case you do not have access to the Linux systems.

HPC training workshops begin Thursday, Sept. 21

By | Educational, Events, General Interest, HPC, News | No Comments

series of training workshops in high performance computing will be held Sept. 21 through Oct. 31, 2017, presented by CSCAR in conjunction with Advanced Research Computing – Technology Services (ARC-TS). All sessions are held at East Hall, Room B254, 530 Church St.

Introduction to the Linux command Line
This course will familiarize the student with the basics of accessing and interacting with Linux computers using the GNU/Linux operating system’s Bash shell, also known as the “command line.”
Dates: (Please sign up for only one)
• Thursday, Sept. 21, 9 a.m. – noon (full descriptionregistration)
• Thursday, Sept. 28, 9 a.m. – noon (full description | registration)
Location:
East Hall, Room B250, 530 Church St.

Introduction to the Flux cluster and batch computing
This workshop will provide a brief overview of the components of the Flux cluster, including the resource manager and scheduler, and will offer students hands-on experience.
Dates: (Please sign up for only one)
• Thursday, Sept. 28, 1 – 4 p.m. (full description | registration)
• Monday, Oct. 2, 9 a.m. – noon (full description | registration)
Location:
East Hall, Room B254, 530 Church St.

Advanced batch computing on the Flux cluster
This course will cover advanced areas of cluster computing on the Flux cluster, including common parallel programming models, dependent and array scheduling, among other topics.
Dates: (Please sign up for only one)
• Tuesday, Oct. 10, 1 – 5 p.m. (full description | registration)
• Thursday, Oct. 12, 9 a.m. – noon (full description | registration)
Location:
East Hall, Room B254, 530 Church St.

Hadoop Workshop
Learn how to process large amounts (up to terabytes) of data using SQL and/or simple programming models available in Python, Scala, and Java.
Date:
• Tuesday, Oct. 31, 1 – 5 p.m. (full description | registration)
Location:
East Hall, Room B254, 530 Church St.

Summer HPC maintenance

By | | No Comments

To accommodate equipment repairs, and upgrades to software, hardware, and operating systems, Flux, Armis, ConFlux, Flux Hadoop, and their storage systems (/home and /scratch) will be unavailable starting at 6 a.m. Saturday, July 29, returning to service on Wednesday, August 2.  

During this time, the following updates are planned:

  • Annual power maintenance at the Modular Data Center.  All systems will be powered off. (Flux/Armis/Flux Hadoop)
  • Campus network hardware and software updates (Flux/Armis/Flux Hadoop)
  • InfiniBand networking updates (firmware and software) (Flux/Armis/ConFlux)
  • Operating system and software updates (All clusters).
  • Resource manager and job scheduling software updates (Flux/Armis).
  • Migrate NFS volumes, including /home, from Value Storage to Turbo (Flux)
  • Update hardware and software of the Lustre file systems that provide /scratch (Flux)

For Flux HPC jobs, you can use the command “maxwalltime” to discover the amount of time remaining until the beginning of the maintenance. Jobs requesting more walltime than remains before the maintenance will be queued and started after the maintenance is completed.

All Flux, Armis, ConFlux, and Flux Hadoop filesystems will be unavailable during the maintenance. We encourage you to copy any data that might be needed during that time from Flux prior to the start of the maintenance.

We will post status updates on our Twitter feed ( https://twitter.com/arcts_um ) throughout the course of the maintenance and send an email to all HPC and Hadoop users when the maintenance has been completed.

ARC-TS Town Hall on Next Generation HPC Cluster

By | | No Comments

The University of Michigan is beginning the process of building our next generation HPC platform, “Big House.”  Flux, the shared HPC cluster, has reached the end of its useful life. Flux has served us well for more than five years, but as we move forward with replacement, we want to make sure we’re meeting the needs of the research community.

ARC-TS will be holding a series of town halls to take input from faculty and researchers on the next HPC platform to be built by the University.  These town halls are open to anyone.

Your input will help to ensure that U-M is on course for providing HPC, so we hope you will make time to attend one of these sessions. If you cannot attend, please email hpc-support@umich.edu with any input you want to share.

ARC-TS Town Hall on Next Generation HPC Cluster

By | | No Comments

The University of Michigan is beginning the process of building our next generation HPC platform, “Big House.”  Flux, the shared HPC cluster, has reached the end of its useful life. Flux has served us well for more than five years, but as we move forward with replacement, we want to make sure we’re meeting the needs of the research community.

ARC-TS will be holding a series of town halls to take input from faculty and researchers on the next HPC platform to be built by the University.  These town halls are open to anyone.

Your input will help to ensure that U-M is on course for providing HPC, so we hope you will make time to attend one of these sessions. If you cannot attend, please email hpc-support@umich.edu with any input you want to share.

ARC-TS Town Hall on Next Generation HPC Cluster

By | | No Comments

The University of Michigan is beginning the process of building our next generation HPC platform, “Big House.”  Flux, the shared HPC cluster, has reached the end of its useful life. Flux has served us well for more than five years, but as we move forward with replacement, we want to make sure we’re meeting the needs of the research community.

ARC-TS will be holding a series of town halls to take input from faculty and researchers on the next HPC platform to be built by the University.  These town halls are open to anyone.

Your input will help to ensure that U-M is on course for providing HPC, so we hope you will make time to attend one of these sessions. If you cannot attend, please email hpc-support@umich.edu with any input you want to share.