Explore ARCExplore ARC

Protected: Beta

By | Uncategorized

This content is password protected. To view it please enter your password below:

Great Lakes

By | Systems and Services

Advanced Research Computing – Technology Services (ARC-TS) is starting the process of creating a new, campus-wide computing cluster, “Great Lakes,” that will serve the broad needs of researchers across the University. Over time, Great Lakes will replace Flux, the shared research computing cluster that currently serves over 300 research projects and 2,500 active users.

The Great Lakes cluster will be available to all researchers on campus for simulation, modeling, machine learning, data science, genomics, and more. The platform will provide a balanced combination of computing power, I/O performance, storage capability, and accelerators.

ARC-TS is in the process of procuring the cluster. A “Beta” cluster will be available to help researchers learn the new system before Great Lakes is deployed in the first half of 2019.

Based on extensive input from faculty and other stakeholders across campus, the new Great Lakes cluster will be designed to deliver similar services and capabilities as Flux, including the ability to accommodate faculty purchases of hardware (as in the current Flux Operating Environment service), access to GPUs and large-memory nodes, and improved support for emerging uses such as machine learning and genomics. Great Lakes will consist of approximately 20,000 cores.

For technical support, please email hpc-support@umich.edu.

Current Status

April 18, 2018: Reviewing proposals from potential vendors.

Order Service

Great Lakes will be available in the first half of 2019. This page will provide updates on the progress of the project.

Please contact hpc-support@umich.edu with any questions.

Yottabyte Research Cloud

By | Systems and Services

yb-logoThe Yottabyte Research Cloud is a partnership between ARC and Yottabyte that provides U-M researchers with high performance, secure and flexible computing environments enabling the analysis of sensitive data sets restricted by federal privacy laws, proprietary access agreements, or confidentiality requirements.

The system is built on Yottabyte’s composable, software-defined infrastructure platform, called Cloud Composer and represents U-M’s first use of software-defined infrastructure for research, allowing the on-the-fly personalized configuration of any-scale computing resources.

Cloud Composer software inventories the physical CPU, RAM and storage components of Cloud Blox appliances into definable and configurable virtual resource groups that may be used to build multi-tenant, multi-site infrastructure as a service.

See the Sept. 2016 press release for more information.

The YRBC platform can accommodate sensitive institutional data classified up to High — including CUI — as identified in the Sensitive Data Guide.

Capabilities

The Yottabyte Research Cloud supports several existing and planned platforms for researchers at the University of Michigan:

  • Data Pipeline Tools, which include databases, message buses, data processing and storage solutions. This platform is suitable for sensitive institutional data classified up to High — including CUI, and data that is not classified as sensitive.
  • Research Database Hosting, an environment that can house research-focused data stored in a number of different database engines.
  • Glovebox, a virtual desktop service for researchers who have sensitive institutional data classified up to High — including CUI — and require higher security. (planned)
  • Virtual desktops for research. This service is similar to Glovebox but is suitable for data that is not classified as sensitive. (planned)
  • Docker Container Service. This service can take any research application that can be containerized for deployment. This service will be suitable forfor sensitive institutional data classified up to High — including CUI, and data that is not classified as sensitive. (planned)

Researchers who need to use Hadoop or Spark for data-intensive work should explore ARC-TS’s separate Hadoop cluster.

Contact arcts-support@umich.edu for more information.

Hardware

The system deploys 40 high performance Hyperconverged YottaBlox nodes (H2400i-E5), each consisting of two, Intel Xeon E5-2680V4 CPU (1,120 cores total), 512GB DDR4 2400MHz RAM (20,480GB total), dual port 40GbE network adapters (80 total) and (2) 800GB NVMe SSD DC P3700 drives (64TB); and 20 storage YottaBlox nodes (S2400i-E5-HDD), each consisting of two, Intel Xeon E5-2620V4 CPU (320 cores total), 128 GB DDR4 2133MHz RAM (2,560 GB total), quad port 10GbE network adapters (80 total),  (2) 800 GB DC S3610 SSD (32 TB total) and 12 x 6 TB 7200 RPM (1,440TB total).

Access

These tools are offered to all researchers at the University of Michigan free of charge, provided that certain usage restrictions are not exceeded. Large-scale users who outgrow the no-cost allotment may purchase additional YBRC resources. All interested parties should contact arcts-support@umich.edu.

Sensitive Data

The U-M Research Ethics and Compliance webpage on Controlled Unclassified Information provides details on handling this type of data. The U-M Sensitive Data Guide to IT Services is a comprehensive guide to sensitive data.

Order Service

The Yottabyte Research Cloud is a pilot program available to all U-M researchers.

Access to Yottabyte Research Cloud resources involves a single email to us at arcts-support@umich.edu. Please include:

  • Your name or your advisor’s name
  • Your unit
  • What you would like to use YBRC for
  • Whether you plan to use restricted data.

Someone from your unit IT staff or an ARC-TS IT staff member will reach out to you and arrange details to determine the best path to make your request work within the Yottabyte Cloud environment.

General Questions

What is the Yottabyte Research Cloud?

The Yottabyte Research Cloud (YBRC) is the University’s private cloud environment for research.   It’s a collection of processors, memory, storage, and networking that can be subdivided into smaller units and allocated to research projects on an as-needed basis to be accessed by virtual machines and containers.

How do I get access to Yottabyte Research Cloud Resources?

Access to Yottabyte Research Cloud resources involves a single email to us at arcts-support@umich.edu. Please include:

  • Your name or your advisor’s name
  • Your unit
  • What you would like to use YBRC for
  • Whether you plan to use restricted data.

Someone from your unit IT staff or an ARC-TS IT staff member will reach out to you and arrange details to determine the best path to make your request work within the Yottabyte Cloud environment.  

What class of problems is Yottabyte Research Cloud designed to solve?

Yottabyte Research Cloud resources are aimed squarely at research and the teaching and training of students involved in research.  Primarily, Yottabyte resources are for sponsored research.  Yottabyte Research Cloud is not for administrative or clinical use (business of the university or the hospital).  Clinical research is acceptable as long as it is sponsored research.  

How large is the Yottabyte Research Cloud?

In total, Yottabyte Research Cloud (YBRC) has 960 processing cores for each Yottabyte cluster, 7.5 Terabytes, and roughly 330 TB of scratch storage available in Maize and Blue each.   

What does Maize Yottabyte Research Cloud and Blue Yottabyte Research Cloud stand for?

Yottabyte resources are divided up between two clusters of computing and storage.    Maize YBRC is for restricted data analyses and storage, and Blue YBRC is for unrestricted data analyses and storage.

What can I do with the Yottabyte Research Cloud?

The initial offering of YBRC is focused on a few different types of use cases:  

  1. Database hosting and data ingestion of streaming data from an external source into a database. We can host many types of databases within Yottabyte, including most structured and unstructured databases.  Examples include MariaDB, PostgreSQL, and MongoDB.
  2. Hosting for applications that you can’t host locally in your lab or you would like to connect to our HPC and data science clusters, such as Material Studio, Galaxy, and SAS Studio.
  3. Hosting of Virtual Desktops and Servers for restricted data use cases, such as statistical analysis of health data, or an analytical project for Controlled Unsecured Information (CUI).  Most people in this case may need a powerful workstation for SAS, Stata or R analyses, for example, or some other application.  

Are these the only things I can do with resources in the Yottabyte Research Cloud?

No!  Contact us at arcts-support@umich.edu if you want to learn whether or not your idea can be done within YBRC!  

How do I get help if I have an issue with something in Yottabyte?

The best way to get help is to send an email to arcts-support@umich.edu with a brief description of the issues that you are seeing.  

What are the support hours for the Yottabyte Research Cloud?

Yottabyte is supported between the hours of 9am to 5pm Monday through Friday.  Response times for support outside of these hours will be longer.

Usage Questions

What’s the biggest machine I can build within Yottabyte Research Cloud?

Because of the way that YBRC divides up resources, the largest Virtual Machine within the cluster is 16 processing cores, and 128 GB of RAM.  

How many Yottabyte Research Cloud resources am I able to access at no cost?

ARC-TS policy is to limit no-cost individual allocations to 100 cores, so that access is always open to multiple research groups.

What if I need more than the no-cost maximum?

If you need to use more than 100 cores of YBRC, we recommend that you purchase YBRC physical infrastructure of your own and add it to the cluster.  Physical infrastructure can be purchased in 96 physical core chunks, which can be oversubscribed as memory allows.  For every block purchased, the researcher will also receive 4 years of hardware and OS support for that block in the case of failure.  For a cost estimate of buying your own blocks of infrastructure and adding to the cluster, please email arcts-support@umich.edu.  

What is ‘scratch’ storage?

Scratch storage for Yottabyte Research Cloud is the storage area network that OS storage and active data storage on the local virtual machines that are not actively being backed up or replicated to a separate infrastructure.  Like the scratch storage on Flux, we don’t recommend storing any data solely on the local disk of any machines.  Make sure that you have backups on other machines, like Turbo, Locker, or some other service.  

HIPAA Compliance Questions

What can I do inside of an HIPAA network enclave?

For researchers with restricted data with a HIPAA classification, we provide a small menu of Linux and Windows workstations to be installed within your enclave.  We do not delegate administrative rights for those workstations to researchers or research staff.  We may delegate administrative rights for workstations and services in your enclaves to IT staff in your unit who have successfully completed the HIPAA IT training coursework given by ITS or HITS, and are familiar with desktop and virtual machine environments.  

Machines in the HIPAA network enclaves are encircled by a deny first firewall that prevents most traffic from entering the enclaves.  Researchers can still visit external-to-campus websites from within a HIPAA network enclave.  Researchers within a HIPAA network enclave can use storage services such as Turbo and MiStorage Silver (via CIFS) to host data for longer-term storage.

What are a researcher and research group responsibilities when they have HIPAA data within YBRC?

All researchers, staff, and students that use YBRC when analyzing restricted data have a shared responsibility in keeping their restricted data secure.

  • Researchers need to be aware of the personnel in their labs who have access to the data in their enclaves.  
    • Each lab should have a process for adding and removing users from enclaves that includes removing departed lab members from access to restricted data as soon as possible after they have left the lab.
    • Each lab should review who has access to their data and enclaves on a twice yearly basis via checking the memberships of their M-Community and Active Directory groups to ensure that people have been removed as requested.  
  • Each lab user must store their restricted data in a specific directory, as discussed during their introductory meeting with YBRC staff.  They must keep the data only in this directory over the life of the data on the system.  

CUI Compliance Questions

What can I do inside of a Secure Enclave Service CUI enclave?

Staff will work with researchers using CUI-classified data to determine the types of analysis that can be conducted on YBRC resources that comply with relevant regulations.

What are a researcher and research group responsibilities when they have CUI data within YBRC?

All researchers, staff, and students that use YBRC when analyzing restricted data have a shared responsibility in keeping their restricted data secure.

  • Researchers need to be aware of the personnel in their labs who have access to the data in their enclaves.  
    • Each lab should have a process for adding and removing users from enclaves that includes removing departed lab members from access to restricted data as soon as possible after they have left the lab.
    • Each lab should review who has access to their data and enclaves on a twice yearly basis via checking the memberships of their M-Community and Active Directory groups to ensure that people have been removed as requested.  
  • Each lab user must store their restricted data in a specific directory, as discussed during their introductory meeting with YBRC staff.  They must keep the data only in this directory over the life of the data on the system.  

ConFlux

By | Systems and Services

conflux1-300x260ConFlux is a cluster that seamlessly combines the computing power of HPC with the analytical power of data science. The next generation of computational physics requires HPC applications (running on external clusters) to interconnect with large data sets at run time. ConFlux provides low latency communications for in- and out- of-core data, cross-platform storage, as well as high throughput interconnects and massive memory allocations. The file-system and scheduler natively handle extreme-scale machine learning and traditional HPC modules in a tightly integrated work flow — rather than in segregated operations — leading to significantly lower latencies, fewer algorithmic barriers and less data movement.

The ConFlux cluster is built with ~58 IBM Power8 CPU two-socket “Firestone” S822LC compute nodes providing 20 cores in each.  Seventeen Power8 CPU two-socket “Garrison” S822LC compute nodes provide an additional 20 cores and host four NVIDIA Pascal GPUs connected via NVIDIA’s NVLink technology to the Power8 system bus.  Each GPU based node has a local high-speed NVMe flash memory for random access.

All compute and storage is connected via a 100 Gb/s InfiniBand fabric. The IBM and NVLink connectivity, combined with IBM CAPI Technology provide an unprecedented data transfer throughput required for the data-driven computational physics researchers will be conducting.

ConFlux is funded by a National Science Foundation grant; the Principal Investigator is Karthik Duraisamy, Assistant Professor of Aerospace Engineering and Director of the Center for Data-Driven Computational Physics (CDDCP). ConFlux and the CDDCP are under the auspices of the Michigan Institute for Computational Discovery and Engineering.

Order Service

A portion of the cycles on ConFlux will be available through a competitive application process. More information will be posted as it becomes available.

Close up of Armis cluster

Armis (HIPAA-aligned HPC Cluster)

By | Systems and Services

The Armis HPC cluster, in conjunction with Turbo Research Storage, provides a secure, scalable, and distributed computing environment that aligns with HIPAA privacy standards. The HPC environment is composed of a task-managing administrative nodes, standard Linux-based 2 and 4 socket server class hardware in a secure data center, connected by both a high-speed ethernet (1 Gbps) and InfiniBand network (40Gbps), and a secure parallel filesystem for temporary data, provided by HIPAA-aligned Turbo Research Storage. Armis is currently provided as a pilot service.

Order Service

Armis is currently offered as a pilot program. To request an Armis account, please fill out this form.

Please see the Terms of Usage for more information.

Related Events

There are no upcoming events at this time.

XSEDE

XSEDE

By | Systems and Services

xsedebanner

The Extreme Scale Engineering and Discovery Environment (XSEDE) is an NSF-funded service that provides computing resources to institutions across the country. XSEDE is an open scientific discovery infrastructure combining leadership class resources at eleven partner sites to create an integrated, persistent computational resource.

ARC-TS participates in the XSEDE Campus Champion program, facilitating access to the organization’s resources. Contact Brock Palen for more information.

For general information on XSEDE, visit the XSEDE home page.

Order Service

Visit the XSEDE User Portal for information on getting an XSEDE allocation.

Related Event

August 28 @ 2:00 pm - 4:30 pm

Geostatistical modeling

Geostatistics deals with continuous variation over space and emphasizes the idea of spatial correlation via covariance. It is widely used for spatial interpolation. We will use ArcGIS and R to…

September 13 @ 9:00 am - 2:00 pm

Data-Intensive Social Science Challenge Symposium

Data-intensive social science is one of the research focus areas that MIDAS supports with its Challenge Awards. Our long-term goal is to support this research area more broadly, using the…

Flux HPC Cluster

By | Systems and Services

The Flux HPC ClusterFlux is the shared, Linux-based high-performance computing (HPC) cluster available to all researchers at the University of Michigan.

Flux consists of approximately 27,000 cores – including 1,372 compute nodes composed of multiple CPU cores, with at least 4 GB of RAM per core, interconnected with InfiniBand networking.

Please see the following pages for more information on Flux:

For technical support, please email hpc-support@umich.edu.

Unit-specific Flux Allocations

Flux Operating Environment

The Flux Operating Environment (FOE) supports researchers with grants that require the purchase of computing hardware. FOE allows researchers to place their own hardware within the Flux cluster.

For more information, visit our FOE page.

Flux On Demand

Flux on Demand (FOD) allows users to run jobs as needed without committing to a month-long allocation. FOD may be the right choice for users with sporadic workloads that don’t result in consistent sets of jobs run over the course of a month. FOD jobs have access to 3,900 Standard Flux processors.

To create a Flux On Demand allocation, email hpc-support@umich.edu with the list of users who should have access to the account. See the ARC-TS Computing Resources Rates page for details on the costs of Flux On Demand.

Large Memory Flux

Flux has 360 cores with larger amounts of RAM — about 25GB per core, or 1TB in a 40-core node. Large Memory Flux is designed for researchers with codes requiring large amounts of RAM or cores in a single system.

For information on determining the size of a Flux allocation, please see our pages on How Flux WorksSizing a Flux Order, and Managing a Flux Project.

GPUs

Flux has 24 K20x GPUs connected to 3 compute nodes,  24 K40 GPUs connected to 6 nodes, and 12 TITANV GPUs connected to 3 nodes. These are available for researchers who have applications that can benefit from the acceleration provided by GPU co-processors. In addition, the software library on Flux has several programs that can benefit from these accelerators.

Each GPU allocation comes with 2 compute cores and 8GB of CPU RAM.

FluxG GPU Specifications

GPU Model NVidia K20X NVidia K40 NVidia TITANV
Number and Type of GPU one Kepler GK110 Kepler GK110B GV100
Peak double precision floating point perf. 1.31 Tflops 1.43 Tflops 7.5 Tflops
Peak single precision floating point perf. 3.95 Tflops 4.29 Tflops 15 Tflops
Tensor Performance (Deep Learning) 110 Tflops
Memory bandwidth (ECC off) 250 GB/sec 288 GB/sec 652.8 GB/sec
Memory size (GDDR5) 6 GB 12 GB 12 GB
CUDA cores 2688 2880 5120 (single precision)

If you have questions, please send email to hpc-support@umich.edu.

Order Service

For information on determining the size of a Flux allocation, please see our pages on How Flux Works, Sizing a Flux Order, and Managing a Flux Project.

To order:

1. Fill out the ARC-TS HPC account request form.

2. Email hpc-support@umich.edu with the following information:

  • the number of cores needed
  • the start date and number of months for the allocation
  • the shortcode for the funding source
  • the list of people who should have access to the allocation
  • the list of people who can change the user list and augment or end the allocations.

For information on costs, visit our Rates page.

Related Events

There are no upcoming events at this time.