Sahara/PluggableProvisioning/PluginAPI - Revision history

Sergey Lukjanov: Sergey Lukjanov moved page Savanna/PluggableProvisioning/PluginAPI to Sahara/PluggableProvisioning/PluginAPI: Savanna project was renamed due to the trademark issues.

2014-03-07T15:41:34Z

Sergey Lukjanov moved page Savanna/PluggableProvisioning/PluginAPI to Sahara/PluggableProvisioning/PluginAPI: Savanna project was renamed due to the trademark issues.

Sergey Lukjanov: /* сonfig */

2013-06-05T14:02:34Z

‎сonfig

Dmitrymex at 09:48, 30 May 2013

2013-05-30T09:48:00Z

Dmitrymex at 09:43, 27 May 2013

2013-05-27T09:43:30Z

Nadya Privalova: /* user_input */

2013-05-23T10:10:16Z

‎user_input

Nadya Privalova: /* сonfig */

2013-05-23T10:09:06Z

‎сonfig

Nadya Privalova: /* Config */

2013-05-23T10:07:55Z

‎Config

Dmitrymex at 14:53, 21 May 2013

2013-05-21T14:53:06Z

Dmitrymex at 14:21, 21 May 2013

2013-05-21T14:21:52Z

Nadya Privalova: Created page with "==Plugin API== '''''get_versions()''''' :Returns all versions of Hadoop that could be used with the plugin. It is responsibility of the plugin to make sure that all required i..."

2013-05-21T12:58:57Z

Created page with "==Plugin API== '''''get_versions()''''' :Returns all versions of Hadoop that could be used with the plugin. It is responsibility of the plugin to make sure that all required i..."

New page

==Plugin API==
'''''get_versions()'''''
:Returns all versions of Hadoop that could be used with the plugin. It is responsibility of the plugin to make sure that all required images for each hadoop version are available, as well as configs and whatever else that plugin needs to create the Hadoop cluster.
:''Returns'': list of strings - Hadoop versions<br />Example Return Value: (“Apache Hadoop 1.1.1”, “CDH 3”, “HDP 1.2”)

'''''get_configs(hadoop_version)'''''
:Lists all configs supported by plugin with descriptions, defaults and targets for which this config is applicable.
:''Returns'': list of configs<br/>Example Return Value: ((“heap size”, “512”, true, “jt”))

'''''get_supported_node_process(hadoop_version)'''''
:Lists of all supported node_processes for a given Hadoop version.
:''Returns'': list of strings - node processes<br/>Example Return Value: (“jt”, “nn”, “tt”, “dt”)

'''''convert(cluster, file)'''''
:Provides plugin with ability to create cluster based on plugin-specific config. Savanna expects plugin to fill in all the required fields. See “Cluster Lifecycle for Config File Mode” section below for clarification.

'''''validate_cluster(cluster)'''''
:Validates a given cluster object. Returns empty list if it is correct. Otherwise for each incorrect field the method should return validation_error with meaningful content.
:''Returns'': list of validation_errors<br/>Example Return Value: ((“heap size”), (“Heap size is a required field and must be specified”), (“mapred.task.timeout”, “The parameter must be int”))

'''''update_infra(cluster)''''
:Plugin has a chance to change cluster description here. Specifically, plugin
*must specify image for VMs
*could change VMs specs in any way it needs.
:For instance, plugin can ask for additional VMs for the management tool.
:''Returns'': None

'''''configure_cluster(cluster)'''''
:Configures cluster on provisioned by savanna VMs. In this function plugin should perform all actions like adjusting OS, installing required packages (including Hadoop, if needed), configuring Hadoop, etc.
:''Returns'': None

'''''start_cluster(cluster)'''''
:Start already configured cluster. This method is guaranteed to be called only on cluster which was already prepared with configure_cluster(...) call.
:''Returns'': None

'''''scale_cluster(cluster, remaining_vm_specs, new_vm_specs, delete_vm_specs)'''''
:To be changed
:Scales cluster - adds/removes nodes to/from active cluster. This function should configure new nodes and attach them to cluster. Additionally here plugin can make some cleanup on VMs that will be deleted.
:''Returns'': None

'''''on_terminate_cluster(cluster)'''''
:When user terminates cluster, Savanna simply shuts down all the cluster VMs. This method is guaranteed to be invoked before that, allowing plugin to do some clean-up.
:''Returns'': None

@@ Line 72: / Line 72: @@
 | description || string ||  ||
 |-
-| type || enum ||  || possible values are: 'string', 'integer', 'boolean'
+| config_type || enum ||  || possible values are: 'string', 'integer', 'boolean', 'enum'
 |-
 | default_value || same as 'type' ||  ||
@@ Line 82: / Line 84: @@
 | scope || string || || could be either 'node' or 'cluster'
 |}
 ====user_input====

← Older revision		Revision as of 09:48, 30 May 2013
Line 61:		Line 61:

	==== сonfig ====		==== сonfig ====
		+
		+	That object represents a single parameter in Hadoop or VM configuration. Plugin can add fields to this object for its own purposes: it is guarantied that controller returns the config object back to plugin untouched.

	{\| class="wikitable"		{\| class="wikitable"

@@ Line 82: / Line 82: @@
 ! Name !! Type !! Constraints  !! Description
 |-
-| config || object ||  || a config object for which this user_input is provided
+| config || config ||  || a config object for which this user_input is provided
 |-
 | value || string ||  || a value provided by the user for the config
 |}
 ====instance====

@@ Line 55: / Line 55: @@
 * Each plugin may have different names for the same processes.
-==== Config ====
+==== сonfig ====
 {| class="wikitable"

@@ Line 55: / Line 55: @@
 * Each plugin may have different names for the same processes.
-==== cluster_confs ====
+==== Config ====
 {| class="wikitable"
@@ Line 74: / Line 74: @@
 |}
 ====user_input====

← Older revision	Revision as of 15:41, 7 March 2014
(No difference)

@@ Line 2: / Line 2: @@
 '''''get_versions()'''''
 :Returns all versions of Hadoop that could be used with the plugin. It is responsibility of the plugin to make sure that all required images for each hadoop version are available, as well as configs and whatever else that plugin needs to create the Hadoop cluster.
-:''Returns'': list of strings - Hadoop versions<br />Example Return Value: (“Apache Hadoop 1.1.1”, “CDH 3”, “HDP 1.2”)
+:''Returns'': list of strings - Hadoop versions
 '''''get_configs(hadoop_version)'''''
 :Lists all configs supported by plugin with descriptions, defaults and targets for which this config is applicable.
-:''Returns'': list of configs<br/>Example Return Value: ((“heap size”, “512”, true, “jt”))
+:''Returns'': list of configs
 '''''get_node_processes(hadoop_version)'''''
-:Lists of all supported node_processes for a given Hadoop version.
+:Returns all supported services and node processes for a given Hadoop version. Each node process belongs to a single service and that relationship is reflected in the returned dict object. See example for details.
-:''Returns'': list of strings - node processes<br/>Example Return Value: (“jt”, “nn”, “tt”, “dt”)
+:''Returns'': dictionary having entries (service -> list of processes)
 '''''convert(cluster, file)'''''
@@ Line 17: / Line 20: @@
 '''''validate_cluster(cluster)'''''
 :Validates a given cluster object. Returns empty list if it is correct. Otherwise for each incorrect field the method should return validation_error with meaningful content.
-:''Returns'': list of validation_errors<br/>Example Return Value: ((“heap size”), (“Heap size is a required field and must be specified”), (“mapred.task.timeout”, “The parameter must be int”))
+:''Returns'': list of validation_errors
-'''''update_infra(cluster)''''
+'''''update_infra(cluster)'''''
 :Plugin has a chance to change cluster description here. Specifically, plugin
-*must specify image for VMs
+:*must specify image for VMs
-*could change VMs specs in any way it needs.
+:*could change VMs specs in any way it needs.
 :For instance, plugin can ask for additional VMs for the management tool.
 :''Returns'': None
@@ Line 54: / Line 58: @@
 #datanode
 * Each plugin may have different names for the same processes.
 ==== сonfig ====
@@ Line 71: / Line 76: @@
 | is_optional || boolean ||  ||
 |-
-| applicable_target || string ||  || The target could be either a node_process, ‘node’ or ‘cluster’
+| applicable_target || string ||  || The target could be either a service returned by '''get_node_processes(...)''' call in form of 'service:<service name>', or 'general'
+|-
 |}
 ====user_input====
@@ Line 86: / Line 93: @@
 | value || string ||  || a value provided by the user for the config
 |}
 ====instance====
@@ Line 119: / Line 127: @@
 | node_processes || list of strings  ||  || list of node processes which run on all nodes of this node_group
 |-
-| node_configs || list of user_inputs ||  || list of configs provided by user. The configs either are for node processes running on this group or they are for general node configuration
+| node_configs || list of user_inputs ||  || list of configs provided by user. 'general' configs target the whole node. Configs targeting a service affect processes which belong to these services. For example if config targets 'service:mapreduce', it affects process 'jobtracker'.
 |-
 | anti_affinity_group ||  ||  || that parameter will be used to control node placement for DN nodes. TBD
@@ Line 163: / Line 171: @@
 ! Name !! Type !! Constraints  !! Description
 |-
-| config_name || string ||  ||
+| config || config ||  ||
 |-
 | error_message || string ||  ||
 |}

@@ Line 47: / Line 47: @@
 Here is a description of all the objects involved in the API.
 Notes:
 * cluster and node_group have ‘extra’ field allowing plugin to persist any complementary info about the cluster.
-* node_process is just a process that runs at some node in cluster.
+* node_process is just a process that runs at some node in cluster. Example list of node processes:
 #jobtracker
 #namenode
 #tasktracker
 #datanode
-:Each plugin may has different names for the same processes.
+* Each plugin may have different names for the same processes.
 ==== cluster_confs ====
@@ Line 63: / Line 61: @@
 ! Name !! Type !! Constraints  !! Description
 |-
-| name ||  ||  ||
+| name || string  ||  ||
 |-
-| description ||  ||  ||
+| description || string ||  ||
 |-
 | type || enum ||  || possible values are: 'string', 'integer', 'boolean'
 |-
-| default_value ||  ||  ||
+| default_value || same as 'type' ||  ||
 |-
-| is_optional ||  ||  ||
+| is_optional || boolean ||  ||
 |-
-| applicable_target ||  ||  || The target could be either a node_process, ‘node’ or ‘cluster’
+| applicable_target || string ||  || The target could be either a node_process, ‘node’ or ‘cluster’
 |}
@@ Line 85: / Line 83: @@
 ! Name !! Type !! Constraints  !! Description
 |-
-| config ||  ||  || a config object for which this user_input is provided
+| config || object ||  || a config object for which this user_input is provided
 |-
-| value ||  ||  || a value provided by the user for the config
+| value || string ||  || a value provided by the user for the config
 |}
@@ Line 98: / Line 96: @@
 ! Name !! Type !! Constraints  !! Description
 |-
-| id ||  ||  ||
+| id || string ||  ||
 |-
-| ip ||  ||  ||
+| ip || string ||  ||
 |-
 | credentials ||  ||  ||
@@ Line 115: / Line 113: @@
 ! Name !! Type !! Constraints  !! Description
 |-
-| name  ||  ||  || Helps uniquely identify the node_group
+| name  || string ||  || helps uniquely identify the node_group
 |-
-| image ||  ||  || Image’s id from Glance. All instances from this node_group are launched from this image
+| image || string ||  || image ID from Glance. All instances from this node_group are launched from this image
 |-
-| flavor ||  ||  || OpenStack’s flavor which is applied to all instances from this node_group
+| flavor || string ||  || OpenStack’s flavor which is applied to all instances from this node_group
 |-
-| node_processes || list of strings  ||  || List of Hadoop processes which run on all nodes from this node_group
+| node_processes || list of strings  ||  || list of node processes which run on all nodes of this node_group
 |-
-| node_configs || list of user_inputs ||  || List of configs provided by user. The configs either are for Hadoop processes running on this group, or they relate to the plugin-specific node configuration
+| node_configs || list of user_inputs ||  || list of configs provided by user. The configs either are for node processes running on this group or they are for general node configuration
 |-
-| anti_affinity_group ||  ||  || That parameter will be used to control node placement for DN nodes. TBD
+| anti_affinity_group ||  ||  || that parameter will be used to control node placement for DN nodes. TBD
 |-
-| count ||  ||  || Number of instances in that group
+| count || int ||  || number of instances in that group
 |-
 | instances || list of instances ||  || list of created instances
 |-
 | vm_requirements ||  ||  ||
+|-
 | extra || || ||
 |}
@@ Line 144: / Line 142: @@
 ! Name !! Type !! Constraints  !! Description
 |-
-| name ||  ||  ||
+| name || string ||  ||
 |-
-| plugin ||  ||  ||
+| plugin || string ||  ||
 |-
-| hadoop_version ||  ||  ||
+| hadoop_version || string ||  ||
 |-
-| default_image ||  ||  || Image specified by user. Plugin could use it as a base image for cluster nodes
+| default_image || string ||  || Glance image ID specified by user. Plugin could use it as a base image for cluster nodes
 |-
-| cluster_configs || list of user_inputs ||  || List of cluster-wide configs
+| cluster_configs || list of user_inputs ||  || list of cluster-wide configs
 |-
 | node_groups || list of node_groups ||  ||
@@ Line 167: / Line 165: @@
 ! Name !! Type !! Constraints  !! Description
 |-
-| config_name ||  ||  ||
+| config_name || string ||  ||
 |-
-| error_message ||  ||  ||
+| error_message || string ||  ||
 |}