Computation on Armis is comprised of nodes containing 12-24 cores with at least 4 GB of RAM per core. As demand increases, Armis can dynamically grow up to 14,000 cores. All compute nodes are interconnected with InfiniBand networking. The current pilot offering is as follows:
- 10 compute nodes, each configured with 16 cores (two eight-core 2.60 GHz Intel Xeon E5-2670 processors) and 64 GB RAM.
The compute nodes are all interconnected with InfinBand networking. The InfiniBand fabric is based on the Mellanox quad-data rate (QDR) platform in the Voltaire GridDirector 4700, which provides 40 Gbps of bandwidth and sub-5μs latency per host. Five Grid Director 4700 switches are connected to each other with 240 Gbps of bandwidth each.
In addition to the InfiniBand networking, there is a gigabit Ethernet network that also connects all of the nodes. This is used for node management and NFS file system access.
The high-speed home and scratch file system are provided by the ARC-TS Turbo Research Storage. Turbo is a high-capacity, fast, reliable, and secure data storage service that allows investigators across U-M to connect their data to the computing resources necessary for their research, including U-M’s Armis HPC cluster. Turbo supports storage of sensitive data.
Computing jobs on Armis are managed through a combination of the Moab Scheduler, the Terascale Open-Source Resource and QUEue Manager (Torque), and the Gold Allocation Manager from Adaptive Computing.
There are three layers of software on Armis.
The Armis cluster runs Red Hat Enterprise Linux version 6.5. We update the operating system on Armis as Red Hat releases new versions and our library of third-party applications offers support. Due to the need to support several types of drivers (AFS and Lustre file system drivers, InfiniBand network drivers and NVIDIA GPU drivers) and dozens of third party applications, we are cautious in upgrading and can lag Red Hat’s releases by months.
Compilers and Parallel and Scientific Libraries
Armis supports the Gnu Compiler Collection and the Intel Compilers for C and Fortran. The Armis cluster’s parallel library is OpenMPI version 1.6.5 which is supported with both GCC and Intel compilers. Armis supports the Intel Math Kernel Library (MKL) for math libraries.
Software installed on Armis must be compatible with these compilers and libraries.
Armis supports a wide range of application software.