Jump to: navigation, search

Difference between revisions of "Sahara/PluggableProvisioning"

Line 25: Line 25:
 
===API for IR and VM Manager===
 
===API for IR and VM Manager===
 
Please visit  [[Savanna/PluggableProvisioning/IRAndVMManagerAPI]]
 
Please visit  [[Savanna/PluggableProvisioning/IRAndVMManagerAPI]]
 +
 +
 +
===Savanna and Plugins===
 +
It is very important to determine zones of responsibility between Savanna and Plugins:
 +
*Savanna:
 +
**provides resources and infrastructure (pre-configured vms, dns, etc.);
 +
**cluster topologies, nodes and storage placement;
 +
**cluster/hadoop/tooling configurations and state storage;
 +
*Plugins:
 +
**cluster monitoring;
 +
**additional tools installation and management (Pig, Hive, etc.);
 +
**final cluster configuration and hadoop management;
 +
**add/remove nodes to/from cluster (It is opaque for Savanna).
 +
 +
=====Savanna - plugins interoperability:=====
 +
 +
#User launches cluster (adds/removes nodes) using Savanna API;
 +
#User determines which provisioning plugin should be used;
 +
#If User wants to run cluster from Configuration File then plugin validates file’s correctness and creates Cluster object;
 +
#Savanna parses user’s changes and runs common validations on it; Besides this Savanna runs plugin-specific validation;
 +
#Savanna creates (modifies) cluster object in DB, returns response to user and starts background job that will provision and launch cluster;
 +
#User receives response with info about created (modified) cluster from Savanna API;
 +
#Savanna calls in background “launch cluster” (add/remove nodes) method if the provisioning plugin;
 +
#Plugin receives cluster configuration and can start vms from tagged images optionally using VM Manager and Image Registry;
 +
#VM Manager provides helpers for ssh/scp/etc to vms;
 +
#Plugin should configure and start 3rd party vendor management tool at the management vm and this tool will control Hadoop cluster;
 +
#Plugin can update cluster status and info to expose information about it.

Revision as of 15:07, 21 May 2013

Overview

Savanna Pluggable Provisioning Mechanism aims to deploy Hadoop clusters and integrate them with 3rd party vendor management tools like Apache Ambari and Cloudera Management Console. Additionally all functionality related to provisioning and managing vanilla Hadoop will be extracted to separate Vanilla Hadoop plugin.

Components and their description

Savanna Pluggable Mechanism consists of the following main components:

  1. Savanna (Controller)
  2. Plugins

And there are two more components which are used by Controller and Plugins:

  1. Image Registry (IR);
  2. VM Manager;

Image Registry

Savanna in general and in particular each plugin need to have special prepared images in Glance.
E.g. if user wants to deploy the cluster from scratch then only one image is needed - Ubuntu, for example. So there should be an ability to get the exact Ubuntu image from Glance. In more general case there should be an ability to determine which image belongs to specific plugin. Beside this, some plugins operte with special ‘management’ node type which requires special management images.
IR provides a mechanism to resolve this issue: associate image with one or several tags and set some additional info about image, for example:

  1. _savanna_tag_<tag-name>: True
  2. _savanna_description: “short description”
  3. _savanna_os: “ubuntu-12.04-x86_64”
  4. _savanna_hadoop: “hadoop-1.1.1”

Information about images is persisted in PostgreSQL. The image has the same id in Glance and in IR.

VM Manager

This component is just a pack of low-level helpers to help plugin interact with vms.

API for IR and VM Manager

Please visit Savanna/PluggableProvisioning/IRAndVMManagerAPI


Savanna and Plugins

It is very important to determine zones of responsibility between Savanna and Plugins:

  • Savanna:
    • provides resources and infrastructure (pre-configured vms, dns, etc.);
    • cluster topologies, nodes and storage placement;
    • cluster/hadoop/tooling configurations and state storage;
  • Plugins:
    • cluster monitoring;
    • additional tools installation and management (Pig, Hive, etc.);
    • final cluster configuration and hadoop management;
    • add/remove nodes to/from cluster (It is opaque for Savanna).
Savanna - plugins interoperability:
  1. User launches cluster (adds/removes nodes) using Savanna API;
  2. User determines which provisioning plugin should be used;
  3. If User wants to run cluster from Configuration File then plugin validates file’s correctness and creates Cluster object;
  4. Savanna parses user’s changes and runs common validations on it; Besides this Savanna runs plugin-specific validation;
  5. Savanna creates (modifies) cluster object in DB, returns response to user and starts background job that will provision and launch cluster;
  6. User receives response with info about created (modified) cluster from Savanna API;
  7. Savanna calls in background “launch cluster” (add/remove nodes) method if the provisioning plugin;
  8. Plugin receives cluster configuration and can start vms from tagged images optionally using VM Manager and Image Registry;
  9. VM Manager provides helpers for ssh/scp/etc to vms;
  10. Plugin should configure and start 3rd party vendor management tool at the management vm and this tool will control Hadoop cluster;
  11. Plugin can update cluster status and info to expose information about it.