Jump to: navigation, search

Difference between revisions of "Heat/DSL"

m (Provider)
m (Heat Orchestration Templates (HOT))
 
(99 intermediate revisions by 6 users not shown)
Line 1: Line 1:
== DRAFT ONLY - WORK IN PROGRESS ==
+
== Proposal Only ==
Please recognize that this is only a '''proposed''' DSL to be considered for development.
+
 
 +
Please recognize that this is only a proposed DSL to be considered for development. This specification is NOT YET IMPLEMENTED.
 +
 
 +
An evolution trying to simplify this DSL further is at [[Heat/DSL2]], and further evolution with real templates at [https://gist.github.com/5408410.git gist-5408410], and open questions in an [https://etherpad.openstack.org/heat-dsl-questions etherpad].
 +
 
 +
See also:
 +
*  [https://blueprints.launchpad.net/heat/+spec/open-api-dsl Launchpad Blueprint for Open API/DSL]
 +
* [[Heat/Vision]]
 +
* [[Heat/Vocabulary]]
  
 
== Overview ==
 
== Overview ==
 
This is a DSL to be used with HEAT to allow a simple REST API to allow a more rich set of capabilities to be exposed. It is designed to be declarative in nature, meaning that it will identify ''what'' to deploy and orchestrate rather than explicitly ''how'' to deploy or orchestrate it. The syntax is intended to be simple, and human readable.
 
This is a DSL to be used with HEAT to allow a simple REST API to allow a more rich set of capabilities to be exposed. It is designed to be declarative in nature, meaning that it will identify ''what'' to deploy and orchestrate rather than explicitly ''how'' to deploy or orchestrate it. The syntax is intended to be simple, and human readable.
  
An Architect writes a <code>Heat Blueprint</code> for how an app can be deployed.
+
Anyone, but generally a subject-matter expert (SME), writes a <code>Heat Blueprint</code> for how an app can be deployed.
 
#  The Heat Blueprint contains <code>Components</code>, <code>Relationships</code> between these components, and <code>Options</code> and <code>Constraints</code> on how the app works.
 
#  The Heat Blueprint contains <code>Components</code>, <code>Relationships</code> between these components, and <code>Options</code> and <code>Constraints</code> on how the app works.
 
#  A User defines <code>Environments</code> where they want to deploy apps (ex. a laptop, an OpenStack Cloud, a Rackspace US Cloud account)
 
#  A User defines <code>Environments</code> where they want to deploy apps (ex. a laptop, an OpenStack Cloud, a Rackspace US Cloud account)
 
# The User picks a blueprint (ex. a Scalable WordPress blueprint) and deploys it to an environment of their choice. That's a <code>Deployment</code> and results in a fully built and running, multi-component app.
 
# The User picks a blueprint (ex. a Scalable WordPress blueprint) and deploys it to an environment of their choice. That's a <code>Deployment</code> and results in a fully built and running, multi-component app.
 
# Heat knows how to add/remove servers (scaling) and can verify the app is running and perform troubleshooting (configuration management)
 
# Heat knows how to add/remove servers (scaling) and can verify the app is running and perform troubleshooting (configuration management)
 +
 +
 +
Here is a visual overview of the parts of a Blueprint.
 +
 +
Notes:
 +
* Not all possible relations are shown.
 +
* This diagram is meant as a guide, not a normative reference.
 +
 +
[[File:Heat blueprint.png|643x430px|center]]
  
 
== Roles ==
 
== Roles ==
* '''Architect''' - The person responsible for creating Heat Blueprints. He/she encodes the logic and constraints of a system or application based on their own deep knowledge and expertise with the Components of that system. For example a WordPress expert would act as the Architect to create the "Best Practice WordPress by Rackspace" Heat Blueprint.
+
* '''Cloud Service Provider''' - A service entity offering hosted cloud services on OpenStack or another cloud technology. Also known as a Vendor.
 +
* '''SME''' - The person responsible for creating Heat Blueprints. He/she encodes the logic and constraints of a system or application based on their own deep knowledge and expertise with the Components of that system. For example a WordPress expert would act as the SME to create the "Best Practice WordPress by Rackspace" Heat Blueprint.
 
* '''Tenant''' - A single customer of a Cloud Service Provider
 
* '''Tenant''' - A single customer of a Cloud Service Provider
 
* '''User''' - Deployment End-User - The person using Heat to initiate orchestrations
 
* '''User''' - Deployment End-User - The person using Heat to initiate orchestrations
* '''Cloud Service Provider''' - A service entity offering hosted cloud services on OpenStack or another cloud technology. Also known as a Vendor.
 
  
 
== DSL Design Goals ==
 
== DSL Design Goals ==
* '''Simplicity''' for the end user should be valued first ahead of all other design goals.
+
* '''Simplicity''' for the end user should be valued first ahead of all other design goals. Make it easy to read, troubleshoot, debug, etc.
* '''Human Readability'''. Artifacts presented to the orchestration API should be human readable for simplified troubleshooting and development. See Simplicity.
 
 
* '''Flexibility''' should allow the DSL to be used for a reasonable variety of use cases, but should not aim to support *all* possible use cases.
 
* '''Flexibility''' should allow the DSL to be used for a reasonable variety of use cases, but should not aim to support *all* possible use cases.
  
 
== Terminology ==
 
== Terminology ==
 
=== DSL ===
 
=== DSL ===
DSL = Domain Specific Language. This is the syntax used to express a configuration or instructions to exercise the features of the system through it's API, without while keeping the API itself minimalistic.
+
<code>DSL</code> is an acronym for Domain Specific Language. This is the syntax used to express a configuration or instructions to exercise the features of the system through its API, while keeping the API itself minimalistic.
  
=== Blueprint ===
+
=== Heat Orchestration Template (HOT) ===
A Heat Blueprint (not to be confused with Launchpad Blueprint) is an orchestration document that details everything that is needed to carry out an orchestration. Think of this as the functional equivalent of an AWS CloudFormation Template. It is expressed as [http://www.yaml.org YAML]. Note: [http://www.json.org JSON] is a subset of YAML. Any YAML parser will also accept JSON. YAML offers easier readability for humans. Heat Blueprints may be shared in common among multiple Tenants of different Cloud Service Providers. They can contain all of the vendor independent specifications for launching a particular service or application. For example, Rackspace may publish a "Best Practice WordPress by Rackspace" Heat Blueprint that can be used by any Tenant of any Cloud Service Provider. The Heat Blueprint is created by an Architect
+
A <code>Heat Orchestration Template</code>  is an orchestration document that details everything that is needed to carry out an orchestration. Think of this as the functional equivalent of an AWS CloudFormation Template. It is expressed as [http://www.yaml.org YAML]. Note: [http://www.json.org JSON] is a subset of YAML. Any YAML parser will also accept JSON. HOTs may be shared in common among multiple Tenants of different Cloud Service Providers. They can contain all of the vendor independent specifications for launching a particular service or application. For example, Rackspace may publish a "Best Practice WordPress by Rackspace" template that can be used by any Tenant of any Cloud Service Provider. The template is created by a subject-matter expert. See: [[Heat/DSL#Heat_Blueprints|Heat Blueprints]]
  
 
===  Environment ===
 
===  Environment ===
A logical target for a Deployment. It is unique to a given user, but independent from Region, Cell, Cloud, etc. A user may have one Environment for production, another for Staging, and perhaps another for dev/test. An arbitrary number of environments may be specified. A given Environment may mix Components from different Cloud Service Providers, perhaps Compute services come from the local OpenStack cloud, but backups happen on the Rackspace Cloud.
+
An <code>Environment</code> is a logical target for a Deployment. It is unique to a given user, but independent from Region, Cell, Cloud, etc. A user may have one Environment for ''Production'', another for ''Staging'', and perhaps another for ''Dev/Test''. An arbitrary number of Environments may be specified. A given Environment may mix Components from different Cloud Service Providers, perhaps Compute services come from the local OpenStack cloud, but backups happen on the Rackspace Cloud. See: [[Heat/DSL#Environments|Environments]]
  
 
=== Deployment ===
 
=== Deployment ===
A Deployment is comprised of a Heat Blueprint and an Environment. A Deployment may be bundled in an app.yaml file that contains the Deployment, Heat Blueprint, and Environment all together for convenience. Typically a user would refer to existing Environments and Heat Blueprints that they reference.
+
A <code>Deployment</code> is comprised of a Heat Blueprint, an Environment and any Inputs. A Deployment may be bundled in a yaml file containing either explicit definitions of or references to these artifacts. Typically a user would refer to existing Environments and Heat Blueprints that they reference. See: [[Heat/DSL#Deployments|Deployments]]
  
 
=== Component ===
 
=== Component ===
A Component is an abstract representation for capabilities or services offered by a Cloud Service Provider. The ids for Components are universal, and will be recorded in a central registry.
+
A <code>Component</code> is an abstract representation for capabilities or services offered by a Cloud Service. A universal catalog of components identifiers will be created in public repositories in a similar way as [http://jujucharms.com Juju Charms] and OpsCode [https://github.com/opscode-cookbooks cookbooks]. See: [[Heat/DSL#Components|Components]]
=== Options and Inputs ===
+
 
 +
=== Resource ===
 +
Resources represent artifacts of a running Deployment and come in two types:
 +
* A <code>Static Resource</code> is an orchestration artifact that can be used to customize a component, such as a security key-pair or user credentials to be used for compute resource access. See: [[Heat/DSL#Static_Resources|Static Resources]]
 +
* A <code>Deployment Resource</code> is an orchestration artifact that represents the instantiation of a component as part of a running <code>Deployment</code> such as a compute node, a software artifact, or load balancer. See: [[Heat/DSL#Providers|Providers]] and [[Heat/DSL#Deployments|Deployments]]
 +
 
 +
== Options and Inputs ==
 
Options can be exposed by Heat Blueprints and Components. An option is the definition of a user-selectable value that can supplied for a Heat Blueprint or Component.
 
Options can be exposed by Heat Blueprints and Components. An option is the definition of a user-selectable value that can supplied for a Heat Blueprint or Component.
  
When launching a Deployment, the values selected for options are stored as an input to the deployment under the 'inputs' key. Inputs can be applied at multiple levels in the deployment hierarchy as follows:
+
When launching a Deployment, the values selected for options are stored as an input to the deployment under the 'inputs' key. Inputs can be applied in various scopes in the deployment hierarchy as follows:
  
 
* Global inputs (apply to everything):
 
* Global inputs (apply to everything):
Line 46: Line 68:
 
* Heat Blueprint inputs (apply to a setting on the Heat Blueprint):
 
* Heat Blueprint inputs (apply to a setting on the Heat Blueprint):
 
** inputs: blueprint: domain: mydomain.com
 
** inputs: blueprint: domain: mydomain.com
 +
** See also: [[Heat/DSL#Heat_Blueprint_Options|Heat Blueprint Options]]
 
* Service inputs (apply to a particular service in the blueprint):
 
* Service inputs (apply to a particular service in the blueprint):
 
** inputs: services: "backend": use_encryption: true
 
** inputs: services: "backend": use_encryption: true
Line 63: Line 86:
 
The above setting would apply to (constrains) any setting called 'foo' under a 'compute' resource in the 'web' service of the blueprint. More precisely scoped options will override broader options. For example, a service or provider option will override a global option.
 
The above setting would apply to (constrains) any setting called 'foo' under a 'compute' resource in the 'web' service of the blueprint. More precisely scoped options will override broader options. For example, a service or provider option will override a global option.
  
== DSL Resources ==
+
== DSL Foundational Concepts ==
=== Component ===
+
=== Components ===
 
A component is the equivalent of a Chef recipe or a Juju charm. They are the primitive building blocks of an application deployment. These can be supplied as part of a deployment or looked up from the server.
 
A component is the equivalent of a Chef recipe or a Juju charm. They are the primitive building blocks of an application deployment. These can be supplied as part of a deployment or looked up from the server.
 
==== Example Component ====
 
==== Example Component ====
Line 72: Line 95:
 
   id: wordpress
 
   id: wordpress
 
   revision: 3
 
   revision: 3
   summary: "A pretty popular blog engine"
+
   summary: "A popular blog engine"
 
   provides:
 
   provides:
 
     url:
 
     url:
Line 91: Line 114:
 
   id: mysql
 
   id: mysql
 
   revision: 1
 
   revision: 1
   summary: "A pretty popular database. Note, this is a cloud database and therefore does not need a host"
+
   summary: "A popular database. Note, this is a cloud database and therefore does not need a host"
 
   provides:
 
   provides:
 
     db: mysql
 
     db: mysql
  
=== Provider ===
+
* '''name''' - A short descriptive string that identifies the component for humans. Example: mssql (not Microsoft SQL Server).
A Provider is a plug-in for Heat that understands how to operate a service or system. For Example, the Rackspace Compute Provider knows how to interact with the Rackspace Open Cloud to create compute resources. The OpsCode Chef Provider knows how to operate Chef and apply Cookbooks. Note: Provider != Cloud Service Provider.
+
* '''role''' - Allows one component to act in several roles(ex. master, slave)
* '''name''' - A short descriptive string that identifies the component for humans. Example: mssql (not Microsoft SQL Server)
 
* '''role''' - Allows the
 
 
* '''version''' - A string representing the version of this component. Example: 1.0.0
 
* '''version''' - A string representing the version of this component. Example: 1.0.0
 
* '''description''' - A short description of the component. Example: Microsoft SQL Server
 
* '''description''' - A short description of the component. Example: Microsoft SQL Server
Line 105: Line 126:
  
 
:Two options for "requires" Syntax (short form, and long form):
 
:Two options for "requires" Syntax (short form, and long form):
 +
 
: '''Short Form'''
 
: '''Short Form'''
 
:Syntax: type: interface
 
:Syntax: type: interface
 
:Example:  
 
:Example:  
  database: mysql
+
  requires:
 +
  database: mysql
 
:
 
:
 
:'''Long Form'''
 
:'''Long Form'''
Line 114: Line 137:
 
:
 
:
 
:Example for "I need a host compute resource that is an Ubuntu 12.04 or later machine":
 
:Example for "I need a host compute resource that is an Ubuntu 12.04 or later machine":
  server: compute
+
  requires:
interface: debian
+
  host:  # this is a user-supplied identifier
relation: host
+
    type: compute
 +
    interface: linux
 +
    distro: debian
 +
    relation: host
 
:
 
:
 
:Note that requirement_keys can be arbitrary to label the requirement. Example requirements from above:
 
:Note that requirement_keys can be arbitrary to label the requirement. Example requirements from above:
:server: (this is the arbitrary key - the label of this requirement) of type type: compute
+
:type: the type of required resource
:interface: debian (this says I expect to be able to do apt-get, vs yum)
+
:interface: the general interface requirement of the resource (should be a flavor of linux)
 +
:distro: an arbitrary key that states that qualifying components must provide distro:debian to satisfy this requirement
 
:relation: host (this says I am hosted on this resource. I cannot exist without it. I am down when it is down). "host" is a keyword.  
 
:relation: host (this says I am hosted on this resource. I cannot exist without it. I am down when it is down). "host" is a keyword.  
 
:constraint: - 'os': ['debian', 'redhat'] The name+key/value syntax allows for requiring more than one of the same resource (ex. log and data mysql databases) as well as adding additional constraints, etc....
 
:constraint: - 'os': ['debian', 'redhat'] The name+key/value syntax allows for requiring more than one of the same resource (ex. log and data mysql databases) as well as adding additional constraints, etc....
 +
* '''options''' - For listing the optional settings I can set on this component.
  
* '''provides''' - An array of resource_type:interface entries
+
They take the general form:
 +
 
 +
<option-name>:
 +
  default: <value>
 +
  required: < optional | required | auto-generated> (default: optional)
 +
  type: < string | integer | boolean | url> (default: string)
 +
 
 +
Example:
 +
username:
 +
  default: root
 +
  required: required
 +
  type: string
 +
 
 +
* '''provides''' - An array or hash of resource_type:interface entries
 
:Note: The array can be represented in YAML with entries preceded with dashes.  
 
:Note: The array can be represented in YAML with entries preceded with dashes.  
 
:For example, let's say we get a Hosting API that provides a "website" resource. Website could provide:
 
:For example, let's say we get a Hosting API that provides a "website" resource. Website could provide:
  
  database: mysql
+
Example:
application: php
+
components:
 +
  - &website_reference_id
 +
  id: website
 +
  provides:
 +
    - database: mysql
 +
    - application:
 +
      name: php
 +
      version: [5.0, 5.5, 6.3-BETA]
 +
 
 +
=== Static Resources and Deployment Resources ===
 +
A <code>Static Resource</code> is an artifact that can be defined up-front by the blueprint author and used to customize one or more aspects of the blueprint. A good example is defining a shared security key-pair that will be created and shared by all resources in the blueprint. They can be expressed as arbitrary key:value pairs and are defined in the Heat Blueprint. Static Resources are processed by the system and generate shared <code>Deployment Resources</code> during provisioning.
 +
 
 +
Example:
 +
  resources:
 +
    sync-keys:
 +
      type: key-pair
 +
      constrains:
 +
      - setting: private_sync_key
 +
        resource_type: application
 +
        service: drupal
 +
        attribute: private_key
 +
      - setting: public_sync_key
 +
        service: drupal
 +
        resource_type: application
 +
        attribute: public_key_ssh
 +
 
 +
=== Providers ===
 +
A Provider is a plug-in for Heat that understands how to instantiate and manage a <code>Deployment Resource</code> via one or more <code>Component</code> implementations. For Example, the Rackspace Compute Provider knows how to interact with the Rackspace Open Cloud to create compute resources. The OpsCode Chef Provider knows how to operate Chef and apply Cookbooks. Note: Provider != Cloud Service Provider.
 +
 
 +
Fields:
 +
* '''vendor''' - A short string identifying the provenance of the provider. Example: openstack
 +
* '''provides''' - A list of what is provided in a list of key:value pairs
 +
* '''constraints''' - A list of constraints about how this provider can be offered. Expressed as a list of key:value pairs
 +
 
 +
 
 +
Example:
 +
providers:
 +
- id: nova
 +
  vendor: openstack
 +
  provides:
 +
  - compute: linux
 +
  - compute: windows
 +
  constraints:
 +
  - region: ORD
  
* '''options''' - For listing the optional settings I can set on this component. Example:
+
Example in the context of an <code>Environment</code>:
 +
environment:
 +
  providers:
 +
    - id: chef-solo
 +
      vendor: opscode
  
username:
 
    default: root
 
    required: optional | required | auto-generated (default: optional)
 
    type: string (currently we support string, int, boolean) (default: string)
 
  
 
Action Items:  
 
Action Items:  
* How can we tie this to a "list" from the provider label: User Name (used for display friendliness) description: ... source_field_name: the name as it is known by the underlying provider. Ex. wordpress/database/db_user for OpsCode Chef. We need this currently since we dynamically generate component definitions and we need to be able to map things back after they've flowed through Heat.  
+
* A Provider may have algorithms or user-created mapping files to map items from the Heat namespace to the provider-specific namespace. Ex. wordpress/database/db_user for OpsCode Chef. This could be used to dynamically generate component definitions and we need to be able to map things back after they've flowed through Heat.
* Should be able to get rid of this in the future with clean, reversible mapping logic in providers. type: string regex: for validation sample: for display, to show what this looks like. help: help text. password:
 
  
=== Providers API ===
+
==== Providers API ====
 
  GET /providers
 
  GET /providers
 
  GET /tenant_id/environments/environment_id/providers/provider_id/catalog/
 
  GET /tenant_id/environments/environment_id/providers/provider_id/catalog/
  
 
Returns a list of components, grouped by resource type
 
Returns a list of components, grouped by resource type
 +
 +
See also: [[Heat/Open_API]]
  
 
=== Environments ===
 
=== Environments ===
Line 153: Line 238:
 
* '''id''' - A unique identifier provided by client (or user).  
 
* '''id''' - A unique identifier provided by client (or user).  
 
* '''name''' - String for convenience to identify the environment
 
* '''name''' - String for convenience to identify the environment
* '''providers''' - Predefined keys are names (capabilities) available from the providers. The provider is identified in Heat based on the key and vendor field.  
+
* '''providers''' - Predefined keys are names (capabilities) available from the providers. The provider is identified in Heat based on the id and vendor field.  
 +
 
  
nova:  
+
prod_compute:
    vendor: rackspace
+
  id: nova:  
 +
  vendor: rackspace
  
The above results in Heat dynamically loading heat.providers.rackspace.nova (last two are vendor.key)  
+
The above results in Heat dynamically loading heat.providers.rackspace.nova (last two are vendor.id)  
  chef-solo:
+
  chef:
 +
    id: chef-solo
 
     vendor: opscode  
 
     vendor: opscode  
     database:
+
database:
        vendor: rackspace  
+
     id: database
        constraints: region=DFW
+
    vendor: rackspace  
        catalog: ...
+
    constraints:
 +
    - region: DFW
 +
    catalog: ...
  
 
About '''constraints''': optional constraints may be applied at the provider level such as the example below to restrict the region to "DFW". The syntax follows normal constraint syntax, but can be shortened to key:value shorthand. For the normal syntax, a 'value' defines what the constraint evaluates to.
 
About '''constraints''': optional constraints may be applied at the provider level such as the example below to restrict the region to "DFW". The syntax follows normal constraint syntax, but can be shortened to key:value shorthand. For the normal syntax, a 'value' defines what the constraint evaluates to.
Line 171: Line 261:
  
 
The Environment will define what this looks like based on environment providers and constraints. A blueprint may be incompatible with an environment if the environment cannot provide the required resources OR meet the stated or inherited constraints.
 
The Environment will define what this looks like based on environment providers and constraints. A blueprint may be incompatible with an environment if the environment cannot provide the required resources OR meet the stated or inherited constraints.
 +
 +
Note that defining an Environment in a Deployment is optional and the system will default to considering all providers and their catalogs when attempting to satisfy the components requested in a Blueprint. Conflict resolution is an area for further discussion and design and should influence the request and response cycle of deployment api endpoints.
  
 
==== Example Environment ====
 
==== Example Environment ====
Line 178: Line 270:
 
   providers:
 
   providers:
 
     nova:
 
     nova:
 +
      id: nova
 
       vendor: rackspace
 
       vendor: rackspace
 
       provides:
 
       provides:
Line 184: Line 277:
 
       constraints:
 
       constraints:
 
       - region: ORD
 
       - region: ORD
     load-balancer:
+
     lb:
 +
      id: load-balancer
 
       vendor: rackspace
 
       vendor: rackspace
 
       provides:
 
       provides:
 
       - loadbalancer: http
 
       - loadbalancer: http
 
     database:
 
     database:
 +
      id: database
 
       vendor: rackspace
 
       vendor: rackspace
 
       provides:
 
       provides:
 
       - database: mysql
 
       - database: mysql
     chef-solo:
+
     chef:
 +
      id: chef-server
 
       vendor: opscode
 
       vendor: opscode
 
       provides:
 
       provides:
Line 198: Line 294:
 
       - database: mysql  # this is mysql installed on a host
 
       - database: mysql  # this is mysql installed on a host
  
=== Heat Blueprints ===
+
=== Heat Orchestration Templates (HOT) ===
These define the architecture for an application. The blueprint describes the resources needed to make an application run, how to connect, and how scale them. Heat Blueprints can have options that determine the final deployment topology and the values that go into the individual component options. The Architect determines what options to expose and with what constraints to apply on the options available to the end user.
+
These define the architecture for an application. The template describes the resources needed to make an application run and which components to connect. HOTs can have options that determine the final deployment topology and the values that go into the individual component options. The SME determines what options to expose and with what constraints to apply on the options available to the end user.
  
An application is a collection of components that provide a useful capability. An application is distinct from infrastructure. A Heat Blueprint describes a design that provides an application with certain 'ilities (scalability, availability, etc...). It defines the components, their relationships to each other, and the various constraints on the components and relationships that must be met in order for the application to work as specified.
+
An application is a collection of components that provide a useful capability. An application is distinct from infrastructure. A HOT describes a design that provides an application with certain 'ilities (scalability, availability, etc...). It defines the components, their relationships to each other, and the various constraints on the components and relationships that must be met in order for the application to work as specified.
  
When expressing services, Heat Blueprints contain:
+
When expressing services, Heat Orchestration Templates contain:
  
 
* One or more services (all under one 'services' entry)
 
* One or more services (all under one 'services' entry)
 
* An arbitrary name for service ID (each one has its own ID)
 
* An arbitrary name for service ID (each one has its own ID)
* One component per service with the expectation that we'll be able to pull in Heat Blueprints as Components in the future.
+
* One or more components per service with the expectation that we'll be able to pull in nested templates (via inclusion or reference) as Components in the future.
 
* Relations between services
 
* Relations between services
  
Line 216: Line 312:
 
* '''services''' - like tiers, but not restricted to the concept of tiers. Currently, there is one component defined per service.  
 
* '''services''' - like tiers, but not restricted to the concept of tiers. Currently, there is one component defined per service.  
  
Example Heat Blueprint:
+
Example HOT:
id: wordpress
+
<pre>
name: my_wp_blog
+
name: Simple Wordpress
description: My WordPress Blog
+
services:
 
+
  blog:
The id or key/value pairs are used find a suitable component. The example above would match against a component like this:
+
    requires:
 
+
    - compute:
component: wordpress
+
      type: linux
...
+
      constraints:
 +
        os: {option: linuxdistribution}
 +
        flavor: {option: instancetype}
 +
    components:
 +
      wordpress:
 +
        requires:
 +
        - application:
 +
            name: wordpress
 +
            role: web
 +
        relations:
 +
        - blog: db
 +
      db:
 +
        requires:
 +
        - database:
 +
            type: mysql
 +
            constraints:
 +
                name: { option: dbname }
 +
                username: { option: dbusername }
 +
                password: { option: dbpassword }
 +
                rootpw: { option: dbrootpassword }
 +
options:
 +
  instancetype:
 +
    description: Webserver instance type
 +
    type: string
 +
    default: m1.large
 +
    constraints:
 +
    - in:
 +
      values:
 +
      - m1.large
 +
      - m1.xlarge
 +
      - m2.large
 +
      - m2.xlarge
 +
      message: "Must be one of m1.large, m1.xlarge, m2.large, m2.xlarge"
 +
  dbname:
 +
    description: The wordpress database name
 +
    type: string
 +
    default: wordpress
 +
    constraints:
 +
    - len:
 +
      min: 1
 +
      max: 64
 +
      message: Must be between 1 and 64 characters
 +
    - matches:
 +
      expr: [a-zA-Z][a-zA-Z0-9]*
 +
      message: must begin with a letter and contain only alphanumeric characters
 +
  dbusername:
 +
    description: The wordpress database admin account username
 +
    type: string
 +
    default: admin
 +
    constraints:
 +
    - len:
 +
      min: 1
 +
      max: 16
 +
      message: must be between 1 and 16 characters
 +
    - matches:
 +
      expr: [a-zA-Z0-9]*
 +
      message: must contain only alphanumeric characters
 +
  dbpassword:
 +
    description: The WordPress database admin account password
 +
    type: password
 +
    default: admin
 +
    constraints:
 +
    - len:
 +
      min: 1
 +
      max: 41
 +
      message: must be between 1 and 41 characters
 +
    - matches:
 +
      expr: [a-zA-Z0-9]*
 +
      message: must contain only alphanumeric characters
 +
  dbrootpassword:
 +
    description: Root password for MySQL
 +
    type: password
 +
    default: admin
 +
    constraints:
 +
    - len:
 +
      min: 1
 +
      max: 41
 +
      message: must be between 1 and 41 characters
 +
    - matches:
 +
      expr: [a-zA-Z0-9]*
 +
      message: must contain only alphanumeric characters
 +
  linuxdistribution:
 +
    description: Distribution of choice
 +
    type: string
 +
    default: F17
 +
    constraints:
 +
    - in
 +
      values:
 +
      - F18
 +
      - F17
 +
      - U10
 +
      - RHEL-6.1
 +
      - RHEL-6.2
 +
      - RHEL-6.3
 +
</pre>
 +
In the example above, the component defined in the <code>blog</code> service would match against any component known to the system that could satisfy:
 +
<pre>
 +
application:
 +
    name: wordpress
 +
    role: web
 +
</pre>
 +
given that no other requirements were placed on the component. This requirement translates as "I need a component implementation that can provide an application named <code>wordpress</code> and that can satisfy the <code>web</code> role"
  
When using a key/value pair:
+
* '''relations''': Specify connections to other services and/or components. Examples: - wordpress app to mysql database - load balancer to webhead.
  
component:
+
* '''options''' - This section of the template identifies the parameters that can be supplied upon provisioning. It follows syntax like component options, except for constraints.  See the HOT Options section for more information.
    type: application
 
    interface: http
 
    constraints:
 
    - count: 2
 
  
Note: We have not yet defined any schema to be able to say something like "greater than one"
+
See options section for syntax for expressing more complex logic, like greater than, less than, etc....  
 
 
* '''relations''': Specify connections to other services. Examples: - wordpress app to mysql database - load balancer to webhead Two syntaxes for relations; short and long:
 
 
 
Short syntax: service_name:interface.
 
Example:
 
my_db_thang: mysql
 
Long syntax: arbitrary name, key values.
 
Example:
 
db:
 
    interface: mysql
 
    service: my_db_thang
 
 
 
* '''options''' - This is critical part of a Heat Blueprint that identifies the parameters that can be supplied upon instantiation. It follows syntax like component options, except for constraints.  See the Heat Blueprint Options section for more information.
 
 
 
* '''database_bigness''' - Option for the size of the database default: 2GB
 
 
 
Action Item: Need to define syntax/schema for expressing more complex logic, like greater than, less than, etc....  
 
 
 
* '''constrains''' - Specifies what values within components that this option constrains. Key/value to select the options, and then the option name.
 
Example:
 
service: my_database_thang
 
    setting: memory
 
service: my_database_log_thang
 
    setting: memory choice:
 
        - value: 1024
 
          name: 1gig
 
        - value: 2048
 
          name: 2GB
 
  
 
* '''resources''' - static resources to be created at planning time and shared across the blueprint. For example, users and keys:
 
* '''resources''' - static resources to be created at planning time and shared across the blueprint. For example, users and keys:
Line 275: Line 440:
  
 
==== Heat Blueprint Options ====
 
==== Heat Blueprint Options ====
 +
You can specify [[Heat/DSL#Options_and_Inputs|options and inputs]] on Heat Blueprints. The supported option fields are detailed here.
 
===== Fields =====
 
===== Fields =====
* '''label''' - the short label to use when displaying the option to the user. description: A full description of this option (what it is)  
+
* '''label''' - The short label to use when displaying an option to the user. description: A full description of this option (what it is)  
 +
* '''meta-data''' - A place where you can hang additional information, such as the version of the schema used, or other arbitrary data needed by your subsystems.
 +
** '''schema-version''' - A string that identifies the schema version used in this Heat Blueprint. Example: 0.7
 +
Example:
 +
  meta-data:
 +
    schema-version: 1.0
 +
    application-name: "Drupal"
 +
    blueprint-type: "Application"
 +
    flavor: "Single Server"
 +
 
 
* '''help''' - Detailed help on this option (how to use it)  
 
* '''help''' - Detailed help on this option (how to use it)  
 
* '''sample''' - An example of what the data will look like. This could be shown as the background of a text control.
 
* '''sample''' - An example of what the data will look like. This could be shown as the background of a text control.
Line 434: Line 609:
  
 
==== Heat Blueprint Examples ====
 
==== Heat Blueprint Examples ====
This is an example of a Heat Blueprint using <code>options</code>, blueprint meta-data and schema versioning.
+
Here is the smallest possible <code>Heat Blueprint</code> for launching a single server:
 +
blueprint:
 +
  services:
 +
    "server":
 +
      component:
 +
        resource-type: compute
 +
environment:
 +
  providers:
 +
    - id: nova:
 +
      vendor: openstack
 +
 
 +
A Hello World <code>Heat Blueprint</code>:
 +
blueprint:
 +
  name: "Hello World"
 +
  services:
 +
    "my_server":
 +
      component:
 +
        resource-type: compute
 +
        interface: linux
 +
  options: #Options are used by the user interface, but they are not required for a truly minimal blueprint
 +
    memory:
 +
      label: Server Size
 +
      default: 512
 +
      type: integer
 +
      constrains:
 +
      - resource_type: compute
 +
        setting: memory
 +
    os:
 +
      label: Operating System
 +
      default: Ubuntu 12.04
 +
      type: string
 +
      constrains:
 +
      - resource_type: compute
 +
        setting: os
 +
    region:
 +
      type: string
 +
environment:   
 +
  id: rackspace-open-cloud
 +
  name: Rackspace Open Cloud
 +
  providers:
 +
    - id: nova:
 +
      vendor: openstack
 +
 
 +
This is a fragment of a <code>Heat Blueprint</code> using <code>options</code>, blueprint <code>meta-data</code> and schema versioning.
  
 
  blueprint:
 
  blueprint:
 
   id: 0255a076c7cf4fd38c69b6727f0b37ea
 
   id: 0255a076c7cf4fd38c69b6727f0b37ea
   name: Managed Cloud WordPress w/ MySQL on VMs
+
   name: WordPress w/ MySQL on VMs
 
   meta-data:
 
   meta-data:
 
     schema-version: 0.7
 
     schema-version: 0.7
Line 444: Line 662:
 
     lb:
 
     lb:
 
       component:
 
       component:
         interface: http
+
         interface: vip
 
         type: load-balancer
 
         type: load-balancer
        constraints:
 
        - algorithm: ROUND_ROBIN
 
 
       relations:
 
       relations:
         web: http
+
         web:
        master: http
+
          service: web
    ...
+
          interface: http
 +
          attributes:
 +
            inbound: https/443
 +
            algorithm: least_connections
 
   options:
 
   options:
 
     url:
 
     url:
Line 481: Line 700:
 
         service: master
 
         service: master
 
         attribute: private_key
 
         attribute: private_key
 +
...
  
Example of a Heat Blueprint for deploying an HA installation of WordPress
+
Fragment of a <code>Heat Blueprint</code> for deploying an HA installation of WordPress:
 
  # A WordPress architecture template
 
  # A WordPress architecture template
 
  blueprint: &wp
 
  blueprint: &wp
Line 491: Line 711:
 
       component: *loadbalancer
 
       component: *loadbalancer
 
       relations:
 
       relations:
         web: http
+
         web: #arbitrary name for relationship
      exposed: true
+
          service: web
      open-ports: [80/tcp]
+
          interface: http
 +
          attributes:
 +
            inbound: https/443
 +
            algorithm: least_connections
 
     web:
 
     web:
 
       component: *wordpress_reference_id  # wordpress component above
 
       component: *wordpress_reference_id  # wordpress component above
 
       relations: {backend: mysql}
 
       relations: {backend: mysql}
 
     backend:
 
     backend:
       components: *mysql
+
       component: *mysql
 
   options:
 
   options:
 
     instance_count:
 
     instance_count:
Line 509: Line 732:
 
       constraints:
 
       constraints:
 
       - greater-than: 1 # this is an HA config
 
       - greater-than: 1 # this is an HA config
 +
...
  
 
=== Deployments ===
 
=== Deployments ===
Line 521: Line 745:
  
 
  option_foo: bar  
 
  option_foo: bar  
 +
 +
Example Heat Blueprint using an input
 
  blueprint:  
 
  blueprint:  
 
     blueprint_input_foo: bar  
 
     blueprint_input_foo: bar  
Line 584: Line 810:
 
       provider: nova
 
       provider: nova
 
       status: up
 
       status: up
       flavor: 1
+
       service: web
      image: 119
 
 
       instance:
 
       instance:
 
         id: 2098383
 
         id: 2098383
 +
        flavor: 1
 +
        image: 119
 
         private_ip: 10.10.1.1
 
         private_ip: 10.10.1.1
 
         public_ip: 2.2.2.18
 
         public_ip: 2.2.2.18
Line 597: Line 824:
 
       type: server
 
       type: server
 
       status: up
 
       status: up
 +
      service: web
 
       provider: nova
 
       provider: nova
      flavor: 1
 
      image: 119
 
 
       instance:
 
       instance:
 
         id: 2098387
 
         id: 2098387
 +
        flavor: 1
 +
        image: 119
 
         private_ip: 10.10.1.8
 
         private_ip: 10.10.1.8
 
         public_ip: 2.2.2.22
 
         public_ip: 2.2.2.22
Line 610: Line 838:
 
     '2':
 
     '2':
 
       type: load-balancer
 
       type: load-balancer
 +
      service: lb
 
       dns-name: CMDEP32ea304-lb1.rackcloudtech.com
 
       dns-name: CMDEP32ea304-lb1.rackcloudtech.com
 
       instance:
 
       instance:
Line 618: Line 847:
 
     '3':
 
     '3':
 
       type: database
 
       type: database
 +
      service: backend
 
       provider: databases
 
       provider: databases
 
       dns-name: CMDEP32ea304-db1.rackcloudtech.com
 
       dns-name: CMDEP32ea304-db1.rackcloudtech.com
      flavor: 1
 
      disk: 2
 
 
       instance:
 
       instance:
 
         id: 99958744
 
         id: 99958744
 +
        flavor: 1
 +
        disk: 2
 +
 +
== Comparison with CFN Terminology ==
 +
 +
Moved to [[Heat/Vocabulary]]
 +
 +
== API ==
 +
See also: the [[Heat/Open_API]] to support this Open DSL.

Latest revision as of 13:04, 30 April 2013

Proposal Only

Please recognize that this is only a proposed DSL to be considered for development. This specification is NOT YET IMPLEMENTED.

An evolution trying to simplify this DSL further is at Heat/DSL2, and further evolution with real templates at gist-5408410, and open questions in an etherpad.

See also:

Overview

This is a DSL to be used with HEAT to allow a simple REST API to allow a more rich set of capabilities to be exposed. It is designed to be declarative in nature, meaning that it will identify what to deploy and orchestrate rather than explicitly how to deploy or orchestrate it. The syntax is intended to be simple, and human readable.

  1. Anyone, but generally a subject-matter expert (SME), writes a Heat Blueprint for how an app can be deployed.
  2. The Heat Blueprint contains Components, Relationships between these components, and Options and Constraints on how the app works.
  3. A User defines Environments where they want to deploy apps (ex. a laptop, an OpenStack Cloud, a Rackspace US Cloud account)
  4. The User picks a blueprint (ex. a Scalable WordPress blueprint) and deploys it to an environment of their choice. That's a Deployment and results in a fully built and running, multi-component app.
  5. Heat knows how to add/remove servers (scaling) and can verify the app is running and perform troubleshooting (configuration management)


Here is a visual overview of the parts of a Blueprint.

Notes:

  • Not all possible relations are shown.
  • This diagram is meant as a guide, not a normative reference.
Heat blueprint.png

Roles

  • Cloud Service Provider - A service entity offering hosted cloud services on OpenStack or another cloud technology. Also known as a Vendor.
  • SME - The person responsible for creating Heat Blueprints. He/she encodes the logic and constraints of a system or application based on their own deep knowledge and expertise with the Components of that system. For example a WordPress expert would act as the SME to create the "Best Practice WordPress by Rackspace" Heat Blueprint.
  • Tenant - A single customer of a Cloud Service Provider
  • User - Deployment End-User - The person using Heat to initiate orchestrations

DSL Design Goals

  • Simplicity for the end user should be valued first ahead of all other design goals. Make it easy to read, troubleshoot, debug, etc.
  • Flexibility should allow the DSL to be used for a reasonable variety of use cases, but should not aim to support *all* possible use cases.

Terminology

DSL

DSL is an acronym for Domain Specific Language. This is the syntax used to express a configuration or instructions to exercise the features of the system through its API, while keeping the API itself minimalistic.

Heat Orchestration Template (HOT)

A Heat Orchestration Template is an orchestration document that details everything that is needed to carry out an orchestration. Think of this as the functional equivalent of an AWS CloudFormation Template. It is expressed as YAML. Note: JSON is a subset of YAML. Any YAML parser will also accept JSON. HOTs may be shared in common among multiple Tenants of different Cloud Service Providers. They can contain all of the vendor independent specifications for launching a particular service or application. For example, Rackspace may publish a "Best Practice WordPress by Rackspace" template that can be used by any Tenant of any Cloud Service Provider. The template is created by a subject-matter expert. See: Heat Blueprints

Environment

An Environment is a logical target for a Deployment. It is unique to a given user, but independent from Region, Cell, Cloud, etc. A user may have one Environment for Production, another for Staging, and perhaps another for Dev/Test. An arbitrary number of Environments may be specified. A given Environment may mix Components from different Cloud Service Providers, perhaps Compute services come from the local OpenStack cloud, but backups happen on the Rackspace Cloud. See: Environments

Deployment

A Deployment is comprised of a Heat Blueprint, an Environment and any Inputs. A Deployment may be bundled in a yaml file containing either explicit definitions of or references to these artifacts. Typically a user would refer to existing Environments and Heat Blueprints that they reference. See: Deployments

Component

A Component is an abstract representation for capabilities or services offered by a Cloud Service. A universal catalog of components identifiers will be created in public repositories in a similar way as Juju Charms and OpsCode cookbooks. See: Components

Resource

Resources represent artifacts of a running Deployment and come in two types:

  • A Static Resource is an orchestration artifact that can be used to customize a component, such as a security key-pair or user credentials to be used for compute resource access. See: Static Resources
  • A Deployment Resource is an orchestration artifact that represents the instantiation of a component as part of a running Deployment such as a compute node, a software artifact, or load balancer. See: Providers and Deployments

Options and Inputs

Options can be exposed by Heat Blueprints and Components. An option is the definition of a user-selectable value that can supplied for a Heat Blueprint or Component.

When launching a Deployment, the values selected for options are stored as an input to the deployment under the 'inputs' key. Inputs can be applied in various scopes in the deployment hierarchy as follows:

  • Global inputs (apply to everything):
    • inputs: domain: mydomain.com
  • Heat Blueprint inputs (apply to a setting on the Heat Blueprint):
  • Service inputs (apply to a particular service in the blueprint):
    • inputs: services: "backend": use_encryption: true
  • Provider inputs (apply to a provider and any resourcers that provider provides):
    • inputs: providers: 'legacy': region: dallas
  • Resource type inputs. These can be applied under services or providers as follows:
    • inputs: services: "backend": 'database': 'memory': 512 Mb providers: 'nova': 'compute': 'operating-system': Ubuntu 12.04 LTS

Options can be associated with one or more options using constraints. Example:

blueprint:
    options: 
        "my_setting": 
            default: 1 
            constrains: [{service: web, resource_type: compute, setting: foo}]

The above setting would apply to (constrains) any setting called 'foo' under a 'compute' resource in the 'web' service of the blueprint. More precisely scoped options will override broader options. For example, a service or provider option will override a global option.

DSL Foundational Concepts

Components

A component is the equivalent of a Chef recipe or a Juju charm. They are the primitive building blocks of an application deployment. These can be supplied as part of a deployment or looked up from the server.

Example Component

# Definitions of components used (similar to Juju charm syntax)
components:
- &wordpress_reference_id
  id: wordpress
  revision: 3
  summary: "A popular blog engine"
  provides:
    url:
      interface: http
  requires:
    db:
     interface: mysql
    server:
      relation: host
      interface: linux
  options:
    url:
      type: String
      default: wp.test.local
      description: the url to use to host your blog on

- &mysql.1
  id: mysql
  revision: 1
  summary: "A popular database. Note, this is a cloud database and therefore does not need a host"
  provides:
    db: mysql
  • name - A short descriptive string that identifies the component for humans. Example: mssql (not Microsoft SQL Server).
  • role - Allows one component to act in several roles(ex. master, slave)
  • version - A string representing the version of this component. Example: 1.0.0
  • description - A short description of the component. Example: Microsoft SQL Server
  • is - Describes the type of resource based on a closed list of OpenStack primitives (compute, database, load-balancer, application, ...)
  • requires What this resource needs in order to function. The need can be specific or general. General would be anything with a mysql interface (for WordPress). Specific would be "I need a host compute resource that is an Ubuntu 12.04 or later machine".
Two options for "requires" Syntax (short form, and long form):
Short Form
Syntax: type: interface
Example:
requires:
 database: mysql
Long Form
Syntax: requirement_key: value
Example for "I need a host compute resource that is an Ubuntu 12.04 or later machine":
requires:
  host:  # this is a user-supplied identifier
    type: compute
    interface: linux
    distro: debian
    relation: host
Note that requirement_keys can be arbitrary to label the requirement. Example requirements from above:
type: the type of required resource
interface: the general interface requirement of the resource (should be a flavor of linux)
distro: an arbitrary key that states that qualifying components must provide distro:debian to satisfy this requirement
relation: host (this says I am hosted on this resource. I cannot exist without it. I am down when it is down). "host" is a keyword.
constraint: - 'os': ['debian', 'redhat'] The name+key/value syntax allows for requiring more than one of the same resource (ex. log and data mysql databases) as well as adding additional constraints, etc....
  • options - For listing the optional settings I can set on this component.

They take the general form:

<option-name>: 
 default: <value> 
 required: < optional | required | auto-generated> (default: optional) 
 type: < string | integer | boolean | url> (default: string) 

Example:

username: 
 default: root 
 required: required
 type: string
  • provides - An array or hash of resource_type:interface entries
Note: The array can be represented in YAML with entries preceded with dashes.
For example, let's say we get a Hosting API that provides a "website" resource. Website could provide:

Example:

components:
- &website_reference_id
  id: website
  provides:
    - database: mysql
    - application:
      name: php
      version: [5.0, 5.5, 6.3-BETA]

Static Resources and Deployment Resources

A Static Resource is an artifact that can be defined up-front by the blueprint author and used to customize one or more aspects of the blueprint. A good example is defining a shared security key-pair that will be created and shared by all resources in the blueprint. They can be expressed as arbitrary key:value pairs and are defined in the Heat Blueprint. Static Resources are processed by the system and generate shared Deployment Resources during provisioning.

Example:

 resources:
   sync-keys:
     type: key-pair
     constrains:
     - setting: private_sync_key
       resource_type: application
       service: drupal
       attribute: private_key
     - setting: public_sync_key
       service: drupal
       resource_type: application
       attribute: public_key_ssh

Providers

A Provider is a plug-in for Heat that understands how to instantiate and manage a Deployment Resource via one or more Component implementations. For Example, the Rackspace Compute Provider knows how to interact with the Rackspace Open Cloud to create compute resources. The OpsCode Chef Provider knows how to operate Chef and apply Cookbooks. Note: Provider != Cloud Service Provider.

Fields:

  • vendor - A short string identifying the provenance of the provider. Example: openstack
  • provides - A list of what is provided in a list of key:value pairs
  • constraints - A list of constraints about how this provider can be offered. Expressed as a list of key:value pairs


Example:

providers:
- id: nova
  vendor: openstack
  provides:
  - compute: linux
  - compute: windows
  constraints:
  - region: ORD

Example in the context of an Environment:

environment:
  providers:
    - id: chef-solo
      vendor: opscode


Action Items:

  • A Provider may have algorithms or user-created mapping files to map items from the Heat namespace to the provider-specific namespace. Ex. wordpress/database/db_user for OpsCode Chef. This could be used to dynamically generate component definitions and we need to be able to map things back after they've flowed through Heat.

Providers API

GET /providers
GET /tenant_id/environments/environment_id/providers/provider_id/catalog/

Returns a list of components, grouped by resource type

See also: Heat/Open_API

Environments

Environments provide a 'context' and optional constraints as well as a list of providers. For example, if I have an environment with a Cloud Databases provider 'constrained' to region LON where Cloud Databases don't have 32GB instances, then the catalog won't have 32GB instances.

  • id - A unique identifier provided by client (or user).
  • name - String for convenience to identify the environment
  • providers - Predefined keys are names (capabilities) available from the providers. The provider is identified in Heat based on the id and vendor field.


prod_compute:

 id: nova: 
 vendor: rackspace

The above results in Heat dynamically loading heat.providers.rackspace.nova (last two are vendor.id)

chef:
    id: chef-solo
    vendor: opscode 
database: 
    id: database
    vendor: rackspace 
    constraints:
    - region: DFW
    catalog: ...

About constraints: optional constraints may be applied at the provider level such as the example below to restrict the region to "DFW". The syntax follows normal constraint syntax, but can be shortened to key:value shorthand. For the normal syntax, a 'value' defines what the constraint evaluates to.

About catalog: - a way to inject a catalog into the provider (two uses for this are #1 testing, #2 only want to show 1GB instances). So if a catalog is provided, the provider will use that. Otherwise, it could log on and query the underlying service (list images, list flavors, get cookbooks, etc...). You'll see this in app.yaml.

The Environment will define what this looks like based on environment providers and constraints. A blueprint may be incompatible with an environment if the environment cannot provide the required resources OR meet the stated or inherited constraints.

Note that defining an Environment in a Deployment is optional and the system will default to considering all providers and their catalogs when attempting to satisfy the components requested in a Blueprint. Conflict resolution is an area for further discussion and design and should influence the request and response cycle of deployment api endpoints.

Example Environment

# Environment
environment: &environment_1000_stag
  name: Rackspace Cloud US - staging
  providers:
    nova:
      id: nova
      vendor: rackspace
      provides:
      - compute: linux
      - compute: windows
      constraints:
      - region: ORD
    lb:
      id: load-balancer
      vendor: rackspace
      provides:
      - loadbalancer: http
    database:
      id: database
      vendor: rackspace
      provides:
      - database: mysql
    chef:
      id: chef-server
      vendor: opscode
      provides:
      - application: http  # see catalog for list of apps like wordpress, drupal, etc...
      - database: mysql  # this is mysql installed on a host

Heat Orchestration Templates (HOT)

These define the architecture for an application. The template describes the resources needed to make an application run and which components to connect. HOTs can have options that determine the final deployment topology and the values that go into the individual component options. The SME determines what options to expose and with what constraints to apply on the options available to the end user.

An application is a collection of components that provide a useful capability. An application is distinct from infrastructure. A HOT describes a design that provides an application with certain 'ilities (scalability, availability, etc...). It defines the components, their relationships to each other, and the various constraints on the components and relationships that must be met in order for the application to work as specified.

When expressing services, Heat Orchestration Templates contain:

  • One or more services (all under one 'services' entry)
  • An arbitrary name for service ID (each one has its own ID)
  • One or more components per service with the expectation that we'll be able to pull in nested templates (via inclusion or reference) as Components in the future.
  • Relations between services

Attributes:

  • id - unique identifier provided by the user/client
  • name - a short string naming the service
  • description: a more detailed description of the service
  • services - like tiers, but not restricted to the concept of tiers. Currently, there is one component defined per service.

Example HOT:

name: Simple Wordpress
services:
  blog:
    requires:
    - compute:
      type: linux
      constraints:
        os: {option: linuxdistribution}
        flavor: {option: instancetype}
    components:
      wordpress:
        requires:
        - application:
            name: wordpress
            role: web
        relations:
        - blog: db
      db:
        requires:
        - database:
            type: mysql
            constraints:
                name: { option: dbname }
                username: { option: dbusername }
                password: { option: dbpassword }
                rootpw: { option: dbrootpassword }
options:
  instancetype:
    description: Webserver instance type
    type: string
    default: m1.large
    constraints:
    - in:
      values:
      - m1.large
      - m1.xlarge
      - m2.large
      - m2.xlarge
      message: "Must be one of m1.large, m1.xlarge, m2.large, m2.xlarge"
  dbname:
    description: The wordpress database name
    type: string
    default: wordpress
    constraints:
    - len:
      min: 1
      max: 64
      message: Must be between 1 and 64 characters
    - matches: 
      expr: [a-zA-Z][a-zA-Z0-9]*
      message: must begin with a letter and contain only alphanumeric characters
  dbusername:
    description: The wordpress database admin account username
    type: string
    default: admin
    constraints:
    - len:
      min: 1
      max: 16
      message: must be between 1 and 16 characters
    - matches:
      expr: [a-zA-Z0-9]*
      message: must contain only alphanumeric characters
  dbpassword:
    description: The WordPress database admin account password
    type: password
    default: admin
    constraints:
    - len:
      min: 1
      max: 41
      message: must be between 1 and 41 characters
    - matches:
      expr: [a-zA-Z0-9]*
      message: must contain only alphanumeric characters
  dbrootpassword:
    description: Root password for MySQL
    type: password
    default: admin
    constraints:
    - len:
      min: 1
      max: 41
      message: must be between 1 and 41 characters
    - matches:
      expr: [a-zA-Z0-9]*
      message: must contain only alphanumeric characters
  linuxdistribution:
    description: Distribution of choice
    type: string
    default: F17
    constraints:
    - in
      values:
      - F18
      - F17
      - U10
      - RHEL-6.1
      - RHEL-6.2
      - RHEL-6.3

In the example above, the component defined in the blog service would match against any component known to the system that could satisfy:

application:
    name: wordpress
    role: web

given that no other requirements were placed on the component. This requirement translates as "I need a component implementation that can provide an application named wordpress and that can satisfy the web role"

  • relations: Specify connections to other services and/or components. Examples: - wordpress app to mysql database - load balancer to webhead.
  • options - This section of the template identifies the parameters that can be supplied upon provisioning. It follows syntax like component options, except for constraints. See the HOT Options section for more information.

See options section for syntax for expressing more complex logic, like greater than, less than, etc....

  • resources - static resources to be created at planning time and shared across the blueprint. For example, users and keys:
my_key: 
    type: key-pair 

Note: private/public key pair will be created before deploying the workflow constrains:

service: my_database_thang 
    setting: key 
    attribute: private_key

The above example will take the private_key value from the generated keys and apply it as the value for 'key' in the my_database_thang component.

Heat Blueprint Options

You can specify options and inputs on Heat Blueprints. The supported option fields are detailed here.

Fields
  • label - The short label to use when displaying an option to the user. description: A full description of this option (what it is)
  • meta-data - A place where you can hang additional information, such as the version of the schema used, or other arbitrary data needed by your subsystems.
    • schema-version - A string that identifies the schema version used in this Heat Blueprint. Example: 0.7

Example:

 meta-data:
   schema-version: 1.0
   application-name: "Drupal"
   blueprint-type: "Application"
   flavor: "Single Server"
  • help - Detailed help on this option (how to use it)
  • sample - An example of what the data will look like. This could be shown as the background of a text control.
  • display-hints - a mapping (key, value pairs) of hints for how to order and display this option in relation to other options ....
  • group - a group name used to group options together. Examples to hint Horizon:
    • deployment: this is a deployment option and shows right under the deployment name
    • application: this is an application option and shows on the first screen of options
    • compute: this is an option that should be shown under the compute server options section
    • load-balancer: this is an option that should be shown under the load balancer options section
    • database: this is an option that should be shown under the database options section
    • dns: this is an option that should be shown under the dns options section
  • order - Indicate the relative order of this option within its group (as an integer)
  • list-type - What is the type of the entries in the list. Used to identify if it should be a specific resource type and list or attribute. The format is resource-type.list where resource-type is a known Heat resource type (compute, database, etc...) and the list is one of the lists from the provider [TODO: define these lists more precisely in the DSL or schema. Right now, they exist only in the provider catalogs]. Examples (and what we will initially support):
    • compute.memory - list of available compute image sizes
    • _compute.os - list of available compute operating systems
    • load-balancer.algorithm - list of available load balancer algorithms
      • encrypted-protocols - The subset of protocols that are encrypted so that we know when to show ssl cert controls. Ex. [https, pop3s]
      • always-accept-certificates - if a blueprint always accepts and handles the certificates (especially if the url is entered in free-form supporting any protocol)
  • default - The default value to use for the option.


NOTE: YAML will assume numbers are ints, so enclose strings in "quotation marks" as a best practice. Special values for this are =generate_password() which will generate a random password on the server. TODO: Consider adding parameters to generate_password() so the heat Blueprint author can make the password generated match their (or their application's) requirements. This would be used by the blueprint author in tandem with constraints (below) for validation on the client side.

  • type - the data type of this option. Valid types are: string, integer, boolean, password, url, region, and text (multi-line string). See later for a description of the url type which has some special attributes.
  • choice - a list of items to select from (used to display a drop-down). The entries are either plain strings or a mapping with value and name entries where value is what is passed to Heat and name is what is displayed to the user. Note: does not apply validation. If you want validation, use in a constraint. This is used for display only. Example:
choice:
- name: Ubuntu 12.04
  value: q340958723409587230459872345
- name: Ubuntu 12.10
  value: 2384729387w0tw9879t87ywt3y42
  • constrains - A list of mappings used as a way to set or limit aspects of the blueprint with the value (or parts of) the option.
  • required - Boolean. Set to true if this option must be supplied by the user.
  • constraints - An array of mappings (key/value pairs) in constraints syntax. Supported constraints are:
    • greater-than
    • less-than
    • greater-than-or-equal-to
    • less-than-or-equal-to
    • min-length - for strings
    • max-length - for strings
    • allowed-chars - Example: "ABCDEFGabcdefg01234565789!&@" - TODO: see if regex can be used here
    • required-chars - Example: "ABCDEFG" - TODO: see if regex can be used here
    • in - a list of acceptable values (these could also be used by clients to display drop-downs)
    • protocols - Used for URL types. This lists allowed protocols in the URL.
    • regex - Do not use look-forward/behind. Keep these simple so they are supported in javascript (client) and python (server). While many of the above can also be written as regex rules, both are available to Heat Blueprint authors to use the one that suits them best.
  • constraints: message - You can add a message key/value pair to any constraint. We recommend adding a message to regex constraints so it is easy to understand what they do when reviewed and so clients/servers can generate useful error messages and people reading the blueprint don't have to decipher the regexs. Ex. "must have 8-16 characters"

Browser clients may parse constraints and apply validation rules. A good practice is to have multiple simple regex constraints to allow browser clients to provide clear and useful feedback to the user for each rule they may break. For example, list lowercase, uppercase, and numeric requirements for a password as three constraints with a message unique to each.

Example Heat Blueprint with Options:

blueprint:
  options:
    database_name:
      label:  Database Name
      sample: db1
      display-hints:
        order: 1
        group: database
     default: wp_db
     description: "This is the name of the database that will be created to host your application's data"
     type: string
     constraints:
     - regex: ^(?=.*).{2,15}$
       message: must be between 2 and 15 characters long
     - regex: ^[A-Za-z0-9]*$
       message: can only contain alphanumeric characters
    database_password:
      label:  Database Password
      display-hints:
        order: 2
        group: database
     description: "This is the password to use to access the database that will host your application's data"
     type: password
     constraints:
     - regex: ^(?=.*).{8,15}$
       message: must be between 8 and 15 characters long
     - regex: ^(?=.*\d)
       message: must contain a digit
     - regex: ^(?=.*[a-z])
       message: must contain a lower case letter
     - regex: ^(?=.*[A-Z])
       message: must contain an upper case letter
Common Types
The URL Type

Options of type url provide some advanced handling of common url use cases. The option can be used simply as a string that accepts a url. In this case, the only benefit of setting the type to url is that a client application can perform certain validation to make sure the provided value is a valid URL (according to RFC 3986).

Example:

option:
  my_web_site:
    type: url

It is useful, however, to be able to handle different parts of a URL (i.e the scheme or protocol, domain, path, port, username, password, etc...) separately. They may be validated independently (e.g. make sure the protocol is http or https only). The parts may be wired up to different parts of the blueprint using constraints (e.g. use the domain part for a dns setting). The way that is supported is that the url type has attributes that can be accessed in the Heat Blueprint or other parts of Heat. These attributes are:

* scheme - this is the first part of the URL
* protocol - this is the first part of the URL as well (an alias to scheme)
* netloc - the dns name or address part
* port - this is the port if specified (e.g. the port in http://localhost:8080 is 8080)
* path - the path of the resource or file
* private_key - the private_key of a certificate to use if the protocol is an encrypted one
* public_key - the public_key of a certificate to use if the protocol is an encrypted one
* intermediate_key - the intermediate key chain of a certificate to use if the protocol is an encrypted one

These attributes can be specified in constraints:

options:
  my_url:
    label: Site Address
    type: url
    constraints:
    - protocols: [http, https]
    constrains:
    - type: load-balancer
      service: lb
      attribute: protocol  # This picks out the 'http' or 'https' part of the URL
      setting: protocol
    - type: compute
      service: web
      attribute: "private_key"  # This picks out the cert
      setting: ssl_certificate
    - type: compute
      service: web
      attribute: "intermediate_key"  # This picks up the intermediate cert
      setting: ssl_intermediate_certificate

You can constrain a list of protocols using the protocols constraint.

options:
 my_url:
    label: Site Address
    type: url
    constraints:
    - protocols: [http, https]

And there are special display-hints used to aid a client in rendering and validating the url. These are encrypted-protocols and always-accept-certificates which are documented in constraints.

When supplying the value for a url as an input, it can be supplied as a string or as a mapping with attributes.

As a string it would be my_site_address: https://mydomain.com/blog.

As a mapping, it would look be:

inputs:
  my_url:
    url: https://domain.com/path  # 'url' is a special shortcut - see note below
    private_key: |
      -----  BEGIN ...
    intermediate_key: |
      -----  BEGIN ...
    public_key: |
      -----  BEGIN ...

Note: A common use case is to supply the url and keys. A shortcut is available that accepts a key called url that can be used to supply the url without having to provide all the components of the url.

Heat Blueprint Examples

Here is the smallest possible Heat Blueprint for launching a single server:

blueprint:
  services:
    "server":
      component:
        resource-type: compute
environment:
  providers:
   - id: nova:
     vendor: openstack

A Hello World Heat Blueprint:

blueprint:
  name: "Hello World"
  services:
    "my_server":
      component:
        resource-type: compute
        interface: linux
  options: #Options are used by the user interface, but they are not required for a truly minimal blueprint
    memory:
      label: Server Size
      default: 512
      type: integer
      constrains:
      - resource_type: compute
        setting: memory
    os:
      label: Operating System
      default: Ubuntu 12.04
      type: string
      constrains:
      - resource_type: compute
        setting: os
    region:
      type: string
environment:    
  id: rackspace-open-cloud
  name: Rackspace Open Cloud
  providers:
   - id: nova:
     vendor: openstack

This is a fragment of a Heat Blueprint using options, blueprint meta-data and schema versioning.

blueprint:
  id: 0255a076c7cf4fd38c69b6727f0b37ea
  name: WordPress w/ MySQL on VMs
  meta-data:
    schema-version: 0.7
  services:
    lb:
      component:
        interface: vip
        type: load-balancer
      relations:
        web:
          service: web
          interface: http
          attributes:
            inbound: https/443
            algorithm: least_connections
  options:
    url:
      label: Site Address
      description: 'The domain you wish to host your blog on. (ex: example.com)'
      type: url
      required: true
      default: http://example.com/
      display-hints:
        group: application
        order: 1
        encrypted-protocols: [https]
        sample: http://example.com/
      constraints:
      - protocols: [http, https]
      - regex: '^([a-zA-Z]{2,}?\:\/\/[a-zA-Z0-9\-]+(?:\.[a-zA-Z0-9\-]+)*\.[a-zA-Z]{2,6}(?:\/?|(?:\/[\w\-]+)*)(?:\/?|\/\w+\.[a-zA-Z]{2,4}(?:\?[\w]+\=[\w\-]+)?)?(?:\&[\w]+\=[\w\-]+)*)$'
        message: must be a valid web address
      constrains:
      - setting: allow_insecure
        service: lb
        resource_type: load-balancer
        value: true  # turn on HTTP if protocol is HTTPS (provider handles it)
  resources:
    sync-keys:
      type: key-pair
      constrains:
      - setting: private_sync_key
        resource_type: application
        service: master
        attribute: private_key
...

Fragment of a Heat Blueprint for deploying an HA installation of WordPress:

# A WordPress architecture template
blueprint: &wp
  id: "6fcc7f31-08f8-4664-90e3-58fffc71f773"
  name: Multi-server Wordpress
  services:
    lb:
      component: *loadbalancer
      relations:
        web: #arbitrary name for relationship
          service: web
          interface: http
          attributes:
            inbound: https/443
            algorithm: least_connections
    web:
      component: *wordpress_reference_id  # wordpress component above
      relations: {backend: mysql}
    backend:
      component: *mysql
  options:
    instance_count:
      type: number
      label: Number of Instances
      description: The number of instances for the specified task.
      default: 2
      constrains:
      - {service: web, resource_type: compute, setting: count}
      constraints:
      - greater-than: 1 # this is an HA config
...

Deployments

A deployment defines and points to a running application and the infrastructure it is running on. It basically says "I took blueprint X and deployed it to environment Y using the following options". It combines a blueprint, an environment to deploy the resources to, and any additional inputs specific to this deployment.

General form:

  • id - a User defined string id
  • name: A short string identifying the Deployment
  • blueprint: can be a reference (YAML), but for now, it's a full copy of a Heat Blueprint. We'll support references using URI later (git, local references, etc...).
  • environment: same as blueprint - right now we make a full copy
  • inputs: - these are for setting levers and dials... there are different scopes: global, blueprint, service, and provider scopes. So you can set the memory size at the provider level (i.e. all servers should be 1GB). Or, all servers in service X should be 1GB and all servers in service Y should be 2GB.
option_foo: bar 

Example Heat Blueprint using an input

blueprint: 
    blueprint_input_foo: bar 
services: 
    my_db_thang: 
        service_input_foo: baz 
        compute: (filter by resource type!)
        service_resource_input_foo: boo 
   providers: 
       'nova': 
           compute: 
               os: ubuntu 

Note: More specific inputs override more general ones (see get_setting code in deployment)

Action Items

  • Determine the need for global inputs. Do we really need them? Or at least put them in a "globals" category
  • Define how we reference blueprints and environments in deployments without including full copies. Reference UUID? URL to file in git or local reference? etc.
  • Improve syntax for indicating a generated value, or remove it and let a generated value be the default. Or create system for code to be included in blueprint to be used to generate, validate values. See artifacts prototype in app.yaml.

Examples of Deployments:

 deployment:
   blueprint:
       name: Simple Wordpress
       wordpress:
         config: *wordpress  # points to wordpress component
         relations: {db: mysql}
   environment:
         name: rackcloudtech-test
         providers:
         - compute:
           vendor: rackspace
           constraints:
           - region: ORD
         - loadbalancer: &rax-lbaas
         - database: &rax-dbaas
         - common:
           vendor: rackspace
           credentials:
           - rackspace:
             ...

Example Deployment with provides and requires in components and providers as keys:

 deployment:
   environment:
     providers:
       nova:
         provides:
         - compute
         vendor: rackspace

Example Deployment

# Actual running app and the parameters supplied when deploying it
deployment:
  blueprint: *wp
  environment: *environment_1000_stag
  inputs:
    instance_count: 4
  resources:
    '0':
      type: server
      provider: nova
      status: up
      service: web
      instance:
        id: 2098383
        flavor: 1
        image: 119
        private_ip: 10.10.1.1
        public_ip: 2.2.2.18
      dns-name: srv1.stabletransit.com
      relations:
        web-backend:
          state: up
    '1':
      type: server
      status: up
      service: web
      provider: nova
      instance:
        id: 2098387
        flavor: 1
        image: 119
        private_ip: 10.10.1.8
        public_ip: 2.2.2.22
      dns-name: srv2.stabletransit.com
      relations:
        web-backend:
          state: up
    '2':
      type: load-balancer
      service: lb
      dns-name: CMDEP32ea304-lb1.rackcloudtech.com
      instance:
        id: 8668444
      relations:
        lb-web:
          state: up
    '3':
      type: database
      service: backend
      provider: databases
      dns-name: CMDEP32ea304-db1.rackcloudtech.com
      instance:
        id: 99958744
        flavor: 1
        disk: 2

Comparison with CFN Terminology

Moved to Heat/Vocabulary

API

See also: the Heat/Open_API to support this Open DSL.