Difference between revisions of "Sahara/SparkPlugin"
m (Sergey Lukjanov moved page Savanna/SparkPlugin to Sahara/SparkPlugin: Savanna project was renamed due to the trademark issues.) |
|||
Line 1: | Line 1: | ||
== Introduction == | == Introduction == | ||
− | [http://spark | + | [http://spark.apache.org/ Spark] is an in-memory implementation of MapReduce written in Scala.<br/> |
− | [https://blueprints.launchpad.net/ | + | [https://blueprints.launchpad.net/sahara/+spec/spark-plugin This blueprint] proposes a Sahara provisioning plugin for Spark that can launch and resize Spark clusters and run EDP jobs. |
+ | |||
+ | In the first iteration no support for scaling and EDP will be available, but those features are planned and will be integrated later. | ||
== Requirements == | == Requirements == | ||
− | + | Spark version 0.9.1 is supported, in ''standalone'' mode. There will be no no support for Mesos or YARN modes for now. | |
+ | |||
+ | At first this plugin will support only Cloudera CDH HDFS as data layer. A DIB element is provided as well to generate disk images compatible with this plugin. | ||
== Documentation == | == Documentation == | ||
− | Notes about the changes to | + | Notes about the changes to sahara-image-elements: [[Sahara/SparkImageBuilder]]<br/> |
− | Notes on using the Spark plugin: [[ | + | Notes on using the Spark plugin: [[Sahara/SparkPluginNotes]] |
== Status == | == Status == | ||
− | |||
Development is done by: Do Huy-Hoang and Vo Thanh Phuc (Master students at Eurecom), Daniele Venzano (Research Engineer at Eurecom), under the supervision of Prof. Pietro Michiardi (at eurecom). This work is partially supported by the BigFoot project, a EC-funded research project. | Development is done by: Do Huy-Hoang and Vo Thanh Phuc (Master students at Eurecom), Daniele Venzano (Research Engineer at Eurecom), under the supervision of Prof. Pietro Michiardi (at eurecom). This work is partially supported by the BigFoot project, a EC-funded research project. | ||
== Related Resources == | == Related Resources == | ||
− | * [[ | + | * [[Sahara/PluggableProvisioning/PluginAPI]] |
− | * [https://blueprints.launchpad.net/ | + | * [https://blueprints.launchpad.net/sahara/+spec/spark-plugin Blueprint] |
Revision as of 16:08, 26 May 2014
Introduction
Spark is an in-memory implementation of MapReduce written in Scala.
This blueprint proposes a Sahara provisioning plugin for Spark that can launch and resize Spark clusters and run EDP jobs.
In the first iteration no support for scaling and EDP will be available, but those features are planned and will be integrated later.
Requirements
Spark version 0.9.1 is supported, in standalone mode. There will be no no support for Mesos or YARN modes for now.
At first this plugin will support only Cloudera CDH HDFS as data layer. A DIB element is provided as well to generate disk images compatible with this plugin.
Documentation
Notes about the changes to sahara-image-elements: Sahara/SparkImageBuilder
Notes on using the Spark plugin: Sahara/SparkPluginNotes
Status
Development is done by: Do Huy-Hoang and Vo Thanh Phuc (Master students at Eurecom), Daniele Venzano (Research Engineer at Eurecom), under the supervision of Prof. Pietro Michiardi (at eurecom). This work is partially supported by the BigFoot project, a EC-funded research project.