Spark Shell

By September 6, 2017

Spark has an easy-to-use interactive shell that can be used to learn API and also analyze data interactively. Below is a simple example written in Scala:

spark-shell --master yarn-client
scala> val textFile = spark.read.textFile("test.txt")
scala> textFile.count()
scala> textFile.first()
scala> textFile.filter(line => line.contains("words")).count()

Leave a Reply