Jan 10 2018
0

Overview

By | | No Comments

The Flux Hadoop cluster will be retired from service on July 1, 2019. Persistent data that is stored in the Hadoop Distributed File System (HDFS) will be destroyed and unavailable after July 1. If you have persistent data that must be preserved and need assistance making arrangements to find a storage location for this data, or you would like to move it to the Cavium Hadoop cluster please contact hpc-support@umich.edu as soon as possible.

The Cavium Hadoop cluster is replacing the Flux Hadoop cluster. You may request access to the Cavium Hadoop cluster by contacting hpc-support@umich.edu.

Please contact us if you have any questions or issues.

Hadoop is designed to scale up from single servers to thousands of machines, each offering local computation and storage. Rather than rely on hardware to deliver high-availability, the library itself is designed to detect and handle failures at the application layer, so delivering a highly-available service on top of a cluster of computers, each of which may be prone to failures. (From hadoop.apache.org)

Flux Hadoop is a technology preview and available at no cost. It may have less technical support than the other Flux services. The Flux Hadoop cluster consists of 12 nodes offering 100TB of HDFS space, and is based on the Kerberos-enabled Hortonworks Data Platform 2.6.3.0. Spark 1 and 2.x are available (and SparkR), Hive2 with Hive on Tez, as well as Anaconda Python 2 and 3.

The software available is:

Sep 05 2017
0

Creating an Account

By | | No Comments

Using Flux Hadoop requires a Flux user account but does not require an active Flux allocation. Information on getting a Flux account can be found on the Flux User Guide page. You only need to pay attention to steps 1, 2, and 4 in order to use Flux-Hadoop. Please note that in step 4 rather than logging into flux-login.arc-ts.umich.edu, you need to log into flux-hadoop-login.arc-ts.umich.edu. This video will help teach you some basic Linux navigation commands if needed.

When you create an account, you will automatically be added to our default queue. If you are using our Hadoop cluster for a class or another specific purpose, or you need a significant allocation, please open a ticket with us so you can be added to the appropriate queue. In many of the examples in this user guide, there is a “–queue <your_queue>” flag. Please fill in the name of your queue when running this examples, or simply “default” if you do not have one.