Sahara/SparkPlugin
Introduction
Spark is an in-memory implementation of MapReduce written in Scala.
This blueprint proposes a Sahara provisioning plugin for Spark that can launch and resize Spark clusters and run EDP jobs.
In the first iteration no support for scaling and EDP will be available, but those features are planned and will be integrated later.
Requirements
Spark version 0.9.1 is supported, in standalone mode. There will be no no support for Mesos or YARN modes for now.
At first this plugin will support only Cloudera CDH HDFS as data layer. A DIB element is provided as well to generate disk images compatible with this plugin.
Documentation
- How to use the Spark plugin: Sahara/SparkPluginNotes
- Notes about the changes to sahara-image-elements: Sahara/SparkImageBuilder
Status
Development is done by: Do Huy-Hoang and Vo Thanh Phuc (Master students at Eurecom), Daniele Venzano (Research Engineer at Eurecom), under the supervision of Prof. Pietro Michiardi (at eurecom). This work is partially supported by the BigFoot project, a EC-funded research project.
