Sahara/SparkPlugin
Introduction
Spark is an in-memory implementation of MapReduce written in Scala.
This blueprint proposes a Savanna provisioning plugin for Spark that can launch and resize Spark clusters and run EDP jobs.
Requirements
Support for version 0.8.0 of Spark and later is planned, since it has relaxed dependencies on Hadoop and HDFS library versions. Spark in standalone mode is targeted, there will be no no support for Mesos or YARN modes for now.
Documentation
Notes about the changes to savanna-image-elements: Savanna/SparkImageBuilder
Notes on using the Spark plugin: Savanna/SparkPluginNotes
Status
We are running unit and integration tests on the plugin, that is almost finished. In January we plan to publish the code for feedback and review.
Development is done by: Do Huy-Hoang and Vo Thanh Phuc (Master students at Eurecom), Daniele Venzano (Research Engineer at Eurecom), under the supervision of Prof. Pietro Michiardi (at eurecom). This work is partially supported by the BigFoot project, a EC-funded research project.
