Explore ARCExplore ARC

Overview

The Flux Hadoop cluster will be retired from service on July 1, 2019. Persistent data that is stored in the Hadoop Distributed File System (HDFS) will be destroyed and unavailable after July 1. If you have persistent data that must be preserved and need assistance making arrangements to find a storage location for this data, or you would like to move it to the Cavium Hadoop cluster please contact hpc-support@umich.edu as
soon as possible.

The Cavium Hadoop cluster is replacing the Flux Hadoop cluster. You may request access to the Cavium Hadoop cluster by contacting hpc-support@umich.edu. The Cavium Hadoop user guide is here: https://arc-ts.umich.edu/cavium/user-guide/. Our Cavium Hadoop 101 training slides are located here: https://tinyurl.com/y2flygjn.

Please contact us if you have any questions or issues.

Hadoop is designed to scale up from single servers to thousands of machines, each offering local computation and storage. Rather than rely on hardware to deliver high-availability, the library itself is designed to detect and handle failures at the application layer, so delivering a highly-available service on top of a cluster of computers, each of which may be prone to failures. (From hadoop.apache.org)

Flux Hadoop is a technology preview and available at no cost. It may have less technical support than the other Flux services. The Flux Hadoop cluster consists of 12 nodes offering 100TB of HDFS space, and is based on the Kerberos-enabled Hortonworks Data Platform 2.6.3.0. Spark 1 and 2.x are available (and SparkR), Hive2 with Hive on Tez, as well as Anaconda Python 2 and 3.

The software available is:

Leave a Reply