Sahara/PluggableProvisioning/PluginAPI

Plugin API
get_versions
 * Returns all versions of Hadoop that could be used with the plugin. It is responsibility of the plugin to make sure that all required images for each hadoop version are available, as well as configs and whatever else that plugin needs to create the Hadoop cluster.
 * Returns: list of strings - Hadoop versions
 * Example return value: (“Apache Hadoop 1.1.1”, “CDH 3”, “HDP 1.2”)

get_configs(hadoop_version)
 * Lists all configs supported by plugin with descriptions, defaults and targets for which this config is applicable.
 * Returns: list of configs
 * Example return value: ((“jobtracker heap size”, "JobTracker heap size, in MB", "int", “512”, true, “service:mapreduce”, "node"))

get_node_processes(hadoop_version)
 * Returns all supported services and node processes for a given Hadoop version. Each node process belongs to a single service and that relationship is reflected in the returned dict object. See example for details.
 * Returns: dictionary having entries (service -> list of processes)
 * Example return value: {"mapreduce": ["tasktracker", "jobtracker"], "hdfs": ["datanode", "namenode"]}

convert(cluster, file)
 * Provides plugin with ability to create cluster based on plugin-specific config. Savanna expects plugin to fill in all the required fields. See “Cluster Lifecycle for Config File Mode” section below for clarification.

validate_cluster(cluster)
 * Validates a given cluster object. Returns empty list if it is correct. Otherwise for each incorrect field the method should return validation_error with meaningful content.
 * Returns: list of validation_errors
 * Example return value: ((“heap size”), (“Heap size is a required field and must be specified”), (“mapred.task.timeout”, “The parameter must be int”))

update_infra(cluster)
 * Plugin has a chance to change cluster description here. Specifically, plugin
 * must specify image for VMs
 * could change VMs specs in any way it needs.
 * For instance, plugin can ask for additional VMs for the management tool.
 * Returns: None

configure_cluster(cluster)
 * Configures cluster on provisioned by savanna VMs. In this function plugin should perform all actions like adjusting OS, installing required packages (including Hadoop, if needed), configuring Hadoop, etc.
 * Returns: None

start_cluster(cluster)
 * Start already configured cluster. This method is guaranteed to be called only on cluster which was already prepared with configure_cluster(...) call.
 * Returns: None

scale_cluster(cluster, remaining_vm_specs, new_vm_specs, delete_vm_specs)
 * To be changed
 * Scales cluster - adds/removes nodes to/from active cluster. This function should configure new nodes and attach them to cluster. Additionally here plugin can make some cleanup on VMs that will be deleted.
 * Returns: None

on_terminate_cluster(cluster)
 * When user terminates cluster, Savanna simply shuts down all the cluster VMs. This method is guaranteed to be invoked before that, allowing plugin to do some clean-up.
 * Returns: None

Object Model
Here is a description of all the objects involved in the API. Notes:
 * cluster and node_group have ‘extra’ field allowing plugin to persist any complementary info about the cluster.
 * node_process is just a process that runs at some node in cluster. Example list of node processes:
 * 1) jobtracker
 * 2) namenode
 * 3) tasktracker
 * 4) datanode
 * Each plugin may have different names for the same processes.

сonfig
That object represents a single parameter in Hadoop or VM configuration. Plugin can add fields to this object for its own purposes: it is guarantied that controller returns the config object back to plugin untouched.

user_input
Value provided by user for a specific config.

instance
An instance created for the cluster

node_group
Specifies group of nodes within a cluster.

cluster
Contains all relevant info about cluster. This object is is provided to the plugin for both cluster creation and scaling. The “Cluster Lifecycle” section below further specifies which fields are filled at which moment

validation_error
Describes what is wrong with one of the values provided by user.