Jump to: navigation, search

Difference between revisions of "Sahara/PluggableProvisioning/PluginAPI"

(Created page with "==Plugin API== '''''get_versions()''''' :Returns all versions of Hadoop that could be used with the plugin. It is responsibility of the plugin to make sure that all required i...")
 
Line 8: Line 8:
 
:''Returns'': list of configs<br/>Example Return Value: ((“heap size”, “512”, true, “jt”))
 
:''Returns'': list of configs<br/>Example Return Value: ((“heap size”, “512”, true, “jt”))
  
−
'''''get_supported_node_process(hadoop_version)'''''
+
'''''get_node_processes(hadoop_version)'''''
 
:Lists of all supported node_processes for a given Hadoop version.
 
:Lists of all supported node_processes for a given Hadoop version.
 
:''Returns'': list of strings - node processes<br/>Example Return Value: (“jt”, “nn”, “tt”, “dt”)
 
:''Returns'': list of strings - node processes<br/>Example Return Value: (“jt”, “nn”, “tt”, “dt”)
Line 42: Line 42:
 
:When user terminates cluster, Savanna simply shuts down all the cluster VMs. This method is guaranteed to be invoked before that, allowing plugin to do some clean-up.
 
:When user terminates cluster, Savanna simply shuts down all the cluster VMs. This method is guaranteed to be invoked before that, allowing plugin to do some clean-up.
 
:''Returns'': None
 
:''Returns'': None
 +
 +
== Object Model ==
 +
 +
Here is a description of all the objects involved in the API.
 +
Notes:
 +
* all fields are strings unless specified otherwise.
 +
* cluster and node_group have ‘extra’ field allowing plugin to persist any complementary info about the cluster.
 +
* node_process is just a process that runs at some node in cluster.
 +
:Here is an example list of node processes:
 +
#jobtracker
 +
#namenode
 +
#tasktracker
 +
#datanode
 +
:Each plugin may has different names for the same processes.
 +
 +
==== cluster_confs ====
 +
 +
{| class="wikitable"
 +
|-
 +
! Name !! Type !! Constraints  !! Description
 +
|-
 +
| name ||  ||  ||
 +
|-
 +
| description ||  ||  ||
 +
|-
 +
| type || enum ||  || possible values are: 'string', 'integer', 'boolean'
 +
|-
 +
| default_value ||  ||  ||
 +
|-
 +
| is_optional ||  ||  ||
 +
|-
 +
| applicable_target ||  ||  || The target could be either a node_process, ‘node’ or ‘cluster’
 +
 +
|}
 +
 +
 +
====user_input====
 +
Value provided by user for a specific config.
 +
 +
{| class="wikitable"
 +
|-
 +
! Name !! Type !! Constraints  !! Description
 +
|-
 +
| config ||  ||  || a config object for which this user_input is provided
 +
|-
 +
| value ||  ||  || a value provided by the user for the config
 +
|}
 +
 +
 +
====instance====
 +
An instance created for the cluster
 +
 +
{| class="wikitable"
 +
|-
 +
! Name !! Type !! Constraints  !! Description
 +
|-
 +
| id ||  ||  ||
 +
|-
 +
| ip ||  ||  ||
 +
|-
 +
| credentials ||  ||  ||
 +
|-
 +
| extra ||  ||  ||
 +
|}
 +
 +
 +
====node_group====
 +
Specifies group of nodes within a cluster.
 +
 +
{| class="wikitable"
 +
|-
 +
! Name !! Type !! Constraints  !! Description
 +
|-
 +
| name  ||  ||  || Helps uniquely identify the node_group
 +
|-
 +
| image ||  ||  || Image’s id from Glance. All instances from this node_group are launched from this image
 +
|-
 +
| flavor ||  ||  || OpenStack’s flavor which is applied to all instances from this node_group
 +
|-
 +
| node_processes || list of strings  ||  || List of Hadoop processes which run on all nodes from this node_group
 +
|-
 +
| node_configs || list of user_inputs ||  || List of configs provided by user. The configs either are for Hadoop processes running on this group, or they relate to the plugin-specific node configuration
 +
|-
 +
| anti_affinity_group ||  ||  || That parameter will be used to control node placement for DN nodes. TBD
 +
|-
 +
| count ||  ||  || Number of instances in that group
 +
|-
 +
| instances || list of instances ||  || list of created instances
 +
|-
 +
| vm_requirements ||  ||  ||
 +
|
 +
| extra || || ||
 +
|}
 +
 +
 +
====cluster====
 +
Contains all relevant info about cluster. This object is is provided to the plugin for both cluster creation and scaling. The “Cluster Lifecycle” section below further specifies which fields are filled at which moment
 +
 +
{| class="wikitable"
 +
|-
 +
! Name !! Type !! Constraints  !! Description
 +
|-
 +
| name ||  ||  ||
 +
|-
 +
| plugin ||  ||  ||
 +
|-
 +
| hadoop_version ||  ||  ||
 +
|-
 +
| default_image ||  ||  || Image specified by user. Plugin could use it as a base image for cluster nodes
 +
|-
 +
| cluster_configs || list of user_inputs ||  || List of cluster-wide configs
 +
|-
 +
| node_groups || list of node_groups ||  ||
 +
|-
 +
| extra ||  ||  ||
 +
|}
 +
 +
 +
====validation_error====
 +
Describes what is wrong with one of the values provided by user.
 +
 +
{| class="wikitable"
 +
|-
 +
! Name !! Type !! Constraints  !! Description
 +
|-
 +
| config_name ||  ||  ||
 +
|-
 +
| error_message ||  ||  ||
 +
|}

Revision as of 14:21, 21 May 2013

Plugin API

get_versions()

Returns all versions of Hadoop that could be used with the plugin. It is responsibility of the plugin to make sure that all required images for each hadoop version are available, as well as configs and whatever else that plugin needs to create the Hadoop cluster.
Returns: list of strings - Hadoop versions
Example Return Value: (“Apache Hadoop 1.1.1”, “CDH 3”, “HDP 1.2”)

get_configs(hadoop_version)

Lists all configs supported by plugin with descriptions, defaults and targets for which this config is applicable.
Returns: list of configs
Example Return Value: ((“heap size”, “512”, true, “jt”))

get_node_processes(hadoop_version)

Lists of all supported node_processes for a given Hadoop version.
Returns: list of strings - node processes
Example Return Value: (“jt”, “nn”, “tt”, “dt”)

convert(cluster, file)

Provides plugin with ability to create cluster based on plugin-specific config. Savanna expects plugin to fill in all the required fields. See “Cluster Lifecycle for Config File Mode” section below for clarification.

validate_cluster(cluster)

Validates a given cluster object. Returns empty list if it is correct. Otherwise for each incorrect field the method should return validation_error with meaningful content.
Returns: list of validation_errors
Example Return Value: ((“heap size”), (“Heap size is a required field and must be specified”), (“mapred.task.timeout”, “The parameter must be int”))

update_infra(cluster)'

Plugin has a chance to change cluster description here. Specifically, plugin
  • must specify image for VMs
  • could change VMs specs in any way it needs.
For instance, plugin can ask for additional VMs for the management tool.
Returns: None

configure_cluster(cluster)

Configures cluster on provisioned by savanna VMs. In this function plugin should perform all actions like adjusting OS, installing required packages (including Hadoop, if needed), configuring Hadoop, etc.
Returns: None

start_cluster(cluster)

Start already configured cluster. This method is guaranteed to be called only on cluster which was already prepared with configure_cluster(...) call.
Returns: None

scale_cluster(cluster, remaining_vm_specs, new_vm_specs, delete_vm_specs)

To be changed
Scales cluster - adds/removes nodes to/from active cluster. This function should configure new nodes and attach them to cluster. Additionally here plugin can make some cleanup on VMs that will be deleted.
Returns: None

on_terminate_cluster(cluster)

When user terminates cluster, Savanna simply shuts down all the cluster VMs. This method is guaranteed to be invoked before that, allowing plugin to do some clean-up.
Returns: None

Object Model

Here is a description of all the objects involved in the API. Notes:

  • all fields are strings unless specified otherwise.
  • cluster and node_group have ‘extra’ field allowing plugin to persist any complementary info about the cluster.
  • node_process is just a process that runs at some node in cluster.
Here is an example list of node processes:
  1. jobtracker
  2. namenode
  3. tasktracker
  4. datanode
Each plugin may has different names for the same processes.

cluster_confs

Name Type Constraints Description
name
description
type enum possible values are: 'string', 'integer', 'boolean'
default_value
is_optional
applicable_target The target could be either a node_process, ‘node’ or ‘cluster’


user_input

Value provided by user for a specific config.

Name Type Constraints Description
config a config object for which this user_input is provided
value a value provided by the user for the config


instance

An instance created for the cluster

Name Type Constraints Description
id
ip
credentials
extra


node_group

Specifies group of nodes within a cluster.

Name Type Constraints Description
name Helps uniquely identify the node_group
image Image’s id from Glance. All instances from this node_group are launched from this image
flavor OpenStack’s flavor which is applied to all instances from this node_group
node_processes list of strings List of Hadoop processes which run on all nodes from this node_group
node_configs list of user_inputs List of configs provided by user. The configs either are for Hadoop processes running on this group, or they relate to the plugin-specific node configuration
anti_affinity_group That parameter will be used to control node placement for DN nodes. TBD
count Number of instances in that group
instances list of instances list of created instances
vm_requirements extra


cluster

Contains all relevant info about cluster. This object is is provided to the plugin for both cluster creation and scaling. The “Cluster Lifecycle” section below further specifies which fields are filled at which moment

Name Type Constraints Description
name
plugin
hadoop_version
default_image Image specified by user. Plugin could use it as a base image for cluster nodes
cluster_configs list of user_inputs List of cluster-wide configs
node_groups list of node_groups
extra


validation_error

Describes what is wrong with one of the values provided by user.

Name Type Constraints Description
config_name
error_message
Retrieved from "https://wiki.openstack.org/w/index.php?title=Sahara/PluggableProvisioning/PluginAPI&oldid=23020"