Difference between revisions of "Heat/DSL"
Adrian Otto (talk | contribs) m (→Providers) |
Randall Burt (talk | contribs) m (→Heat Orchestration Templates (HOT)) |
||
(32 intermediate revisions by 6 users not shown) | |||
Line 1: | Line 1: | ||
− | == | + | == Proposal Only == |
− | Please recognize that this is only a | + | |
+ | Please recognize that this is only a proposed DSL to be considered for development. This specification is NOT YET IMPLEMENTED. | ||
+ | |||
+ | An evolution trying to simplify this DSL further is at [[Heat/DSL2]], and further evolution with real templates at [https://gist.github.com/5408410.git gist-5408410], and open questions in an [https://etherpad.openstack.org/heat-dsl-questions etherpad]. | ||
+ | |||
+ | See also: | ||
+ | * [https://blueprints.launchpad.net/heat/+spec/open-api-dsl Launchpad Blueprint for Open API/DSL] | ||
+ | * [[Heat/Vision]] | ||
+ | * [[Heat/Vocabulary]] | ||
== Overview == | == Overview == | ||
Line 34: | Line 42: | ||
<code>DSL</code> is an acronym for Domain Specific Language. This is the syntax used to express a configuration or instructions to exercise the features of the system through its API, while keeping the API itself minimalistic. | <code>DSL</code> is an acronym for Domain Specific Language. This is the syntax used to express a configuration or instructions to exercise the features of the system through its API, while keeping the API itself minimalistic. | ||
− | === Heat | + | === Heat Orchestration Template (HOT) === |
− | A <code>Heat | + | A <code>Heat Orchestration Template</code> is an orchestration document that details everything that is needed to carry out an orchestration. Think of this as the functional equivalent of an AWS CloudFormation Template. It is expressed as [http://www.yaml.org YAML]. Note: [http://www.json.org JSON] is a subset of YAML. Any YAML parser will also accept JSON. HOTs may be shared in common among multiple Tenants of different Cloud Service Providers. They can contain all of the vendor independent specifications for launching a particular service or application. For example, Rackspace may publish a "Best Practice WordPress by Rackspace" template that can be used by any Tenant of any Cloud Service Provider. The template is created by a subject-matter expert. See: [[Heat/DSL#Heat_Blueprints|Heat Blueprints]] |
=== Environment === | === Environment === | ||
Line 130: | Line 138: | ||
:Example for "I need a host compute resource that is an Ubuntu 12.04 or later machine": | :Example for "I need a host compute resource that is an Ubuntu 12.04 or later machine": | ||
requires: | requires: | ||
− | - type: compute | + | host: # this is a user-supplied identifier |
− | + | type: compute | |
− | + | interface: linux | |
− | + | distro: debian | |
+ | relation: host | ||
: | : | ||
:Note that requirement_keys can be arbitrary to label the requirement. Example requirements from above: | :Note that requirement_keys can be arbitrary to label the requirement. Example requirements from above: | ||
Line 148: | Line 157: | ||
default: <value> | default: <value> | ||
required: < optional | required | auto-generated> (default: optional) | required: < optional | required | auto-generated> (default: optional) | ||
− | type: < string | | + | type: < string | integer | boolean | url> (default: string) |
Example: | Example: | ||
Line 156: | Line 165: | ||
type: string | type: string | ||
− | * '''provides''' - An array of resource_type:interface entries | + | * '''provides''' - An array or hash of resource_type:interface entries |
:Note: The array can be represented in YAML with entries preceded with dashes. | :Note: The array can be represented in YAML with entries preceded with dashes. | ||
:For example, let's say we get a Hosting API that provides a "website" resource. Website could provide: | :For example, let's say we get a Hosting API that provides a "website" resource. Website could provide: | ||
Line 170: | Line 179: | ||
version: [5.0, 5.5, 6.3-BETA] | version: [5.0, 5.5, 6.3-BETA] | ||
− | === Static Resources === | + | === Static Resources and Deployment Resources === |
− | A <code>Static Resource</code> is an artifact that can be used to customize a | + | A <code>Static Resource</code> is an artifact that can be defined up-front by the blueprint author and used to customize one or more aspects of the blueprint. A good example is defining a shared security key-pair that will be created and shared by all resources in the blueprint. They can be expressed as arbitrary key:value pairs and are defined in the Heat Blueprint. Static Resources are processed by the system and generate shared <code>Deployment Resources</code> during provisioning. |
Example: | Example: | ||
Line 198: | Line 207: | ||
Example: | Example: | ||
providers: | providers: | ||
− | + | - id: nova | |
− | + | vendor: openstack | |
− | + | provides: | |
− | + | - compute: linux | |
− | + | - compute: windows | |
− | + | constraints: | |
− | + | - region: ORD | |
Example in the context of an <code>Environment</code>: | Example in the context of an <code>Environment</code>: | ||
environment: | environment: | ||
providers: | providers: | ||
− | chef-solo | + | - id: chef-solo |
vendor: opscode | vendor: opscode | ||
− | + | ||
− | |||
Action Items: | Action Items: | ||
− | * A Provider may have algorithms or user-created mapping files to map items from the Heat namespace to the provider-specific | + | * A Provider may have algorithms or user-created mapping files to map items from the Heat namespace to the provider-specific namespace. Ex. wordpress/database/db_user for OpsCode Chef. This could be used to dynamically generate component definitions and we need to be able to map things back after they've flowed through Heat. |
==== Providers API ==== | ==== Providers API ==== | ||
Line 230: | Line 238: | ||
* '''id''' - A unique identifier provided by client (or user). | * '''id''' - A unique identifier provided by client (or user). | ||
* '''name''' - String for convenience to identify the environment | * '''name''' - String for convenience to identify the environment | ||
− | * '''providers''' - Predefined keys are names (capabilities) available from the providers. The provider is identified in Heat based on the | + | * '''providers''' - Predefined keys are names (capabilities) available from the providers. The provider is identified in Heat based on the id and vendor field. |
− | + | prod_compute: | |
− | + | id: nova: | |
+ | vendor: rackspace | ||
− | The above results in Heat dynamically loading heat.providers.rackspace.nova (last two are vendor. | + | The above results in Heat dynamically loading heat.providers.rackspace.nova (last two are vendor.id) |
− | chef-solo | + | chef: |
+ | id: chef-solo | ||
vendor: opscode | vendor: opscode | ||
− | database | + | database: |
− | + | id: database | |
− | + | vendor: rackspace | |
− | + | constraints: | |
+ | - region: DFW | ||
+ | catalog: ... | ||
About '''constraints''': optional constraints may be applied at the provider level such as the example below to restrict the region to "DFW". The syntax follows normal constraint syntax, but can be shortened to key:value shorthand. For the normal syntax, a 'value' defines what the constraint evaluates to. | About '''constraints''': optional constraints may be applied at the provider level such as the example below to restrict the region to "DFW". The syntax follows normal constraint syntax, but can be shortened to key:value shorthand. For the normal syntax, a 'value' defines what the constraint evaluates to. | ||
Line 258: | Line 270: | ||
providers: | providers: | ||
nova: | nova: | ||
+ | id: nova | ||
vendor: rackspace | vendor: rackspace | ||
provides: | provides: | ||
Line 264: | Line 277: | ||
constraints: | constraints: | ||
- region: ORD | - region: ORD | ||
− | load-balancer | + | lb: |
+ | id: load-balancer | ||
vendor: rackspace | vendor: rackspace | ||
provides: | provides: | ||
- loadbalancer: http | - loadbalancer: http | ||
database: | database: | ||
+ | id: database | ||
vendor: rackspace | vendor: rackspace | ||
provides: | provides: | ||
- database: mysql | - database: mysql | ||
− | chef- | + | chef: |
+ | id: chef-server | ||
vendor: opscode | vendor: opscode | ||
provides: | provides: | ||
Line 278: | Line 294: | ||
- database: mysql # this is mysql installed on a host | - database: mysql # this is mysql installed on a host | ||
− | === Heat | + | === Heat Orchestration Templates (HOT) === |
− | These define the architecture for an application. The | + | These define the architecture for an application. The template describes the resources needed to make an application run and which components to connect. HOTs can have options that determine the final deployment topology and the values that go into the individual component options. The SME determines what options to expose and with what constraints to apply on the options available to the end user. |
− | An application is a collection of components that provide a useful capability. An application is distinct from infrastructure. A | + | An application is a collection of components that provide a useful capability. An application is distinct from infrastructure. A HOT describes a design that provides an application with certain 'ilities (scalability, availability, etc...). It defines the components, their relationships to each other, and the various constraints on the components and relationships that must be met in order for the application to work as specified. |
− | When expressing services, Heat | + | When expressing services, Heat Orchestration Templates contain: |
* One or more services (all under one 'services' entry) | * One or more services (all under one 'services' entry) | ||
* An arbitrary name for service ID (each one has its own ID) | * An arbitrary name for service ID (each one has its own ID) | ||
− | * One | + | * One or more components per service with the expectation that we'll be able to pull in nested templates (via inclusion or reference) as Components in the future. |
* Relations between services | * Relations between services | ||
Line 296: | Line 312: | ||
* '''services''' - like tiers, but not restricted to the concept of tiers. Currently, there is one component defined per service. | * '''services''' - like tiers, but not restricted to the concept of tiers. Currently, there is one component defined per service. | ||
− | Example | + | Example HOT: |
− | + | <pre> | |
− | + | name: Simple Wordpress | |
− | + | services: | |
+ | blog: | ||
+ | requires: | ||
+ | - compute: | ||
+ | type: linux | ||
+ | constraints: | ||
+ | os: {option: linuxdistribution} | ||
+ | flavor: {option: instancetype} | ||
+ | components: | ||
+ | wordpress: | ||
+ | requires: | ||
+ | - application: | ||
+ | name: wordpress | ||
+ | role: web | ||
+ | relations: | ||
+ | - blog: db | ||
+ | db: | ||
+ | requires: | ||
+ | - database: | ||
+ | type: mysql | ||
+ | constraints: | ||
+ | name: { option: dbname } | ||
+ | username: { option: dbusername } | ||
+ | password: { option: dbpassword } | ||
+ | rootpw: { option: dbrootpassword } | ||
+ | options: | ||
+ | instancetype: | ||
+ | description: Webserver instance type | ||
+ | type: string | ||
+ | default: m1.large | ||
+ | constraints: | ||
+ | - in: | ||
+ | values: | ||
+ | - m1.large | ||
+ | - m1.xlarge | ||
+ | - m2.large | ||
+ | - m2.xlarge | ||
+ | message: "Must be one of m1.large, m1.xlarge, m2.large, m2.xlarge" | ||
+ | dbname: | ||
+ | description: The wordpress database name | ||
+ | type: string | ||
+ | default: wordpress | ||
+ | constraints: | ||
+ | - len: | ||
+ | min: 1 | ||
+ | max: 64 | ||
+ | message: Must be between 1 and 64 characters | ||
+ | - matches: | ||
+ | expr: [a-zA-Z][a-zA-Z0-9]* | ||
+ | message: must begin with a letter and contain only alphanumeric characters | ||
+ | dbusername: | ||
+ | description: The wordpress database admin account username | ||
+ | type: string | ||
+ | default: admin | ||
+ | constraints: | ||
+ | - len: | ||
+ | min: 1 | ||
+ | max: 16 | ||
+ | message: must be between 1 and 16 characters | ||
+ | - matches: | ||
+ | expr: [a-zA-Z0-9]* | ||
+ | message: must contain only alphanumeric characters | ||
+ | dbpassword: | ||
+ | description: The WordPress database admin account password | ||
+ | type: password | ||
+ | default: admin | ||
+ | constraints: | ||
+ | - len: | ||
+ | min: 1 | ||
+ | max: 41 | ||
+ | message: must be between 1 and 41 characters | ||
+ | - matches: | ||
+ | expr: [a-zA-Z0-9]* | ||
+ | message: must contain only alphanumeric characters | ||
+ | dbrootpassword: | ||
+ | description: Root password for MySQL | ||
+ | type: password | ||
+ | default: admin | ||
+ | constraints: | ||
+ | - len: | ||
+ | min: 1 | ||
+ | max: 41 | ||
+ | message: must be between 1 and 41 characters | ||
+ | - matches: | ||
+ | expr: [a-zA-Z0-9]* | ||
+ | message: must contain only alphanumeric characters | ||
+ | linuxdistribution: | ||
+ | description: Distribution of choice | ||
+ | type: string | ||
+ | default: F17 | ||
+ | constraints: | ||
+ | - in | ||
+ | values: | ||
+ | - F18 | ||
+ | - F17 | ||
+ | - U10 | ||
+ | - RHEL-6.1 | ||
+ | - RHEL-6.2 | ||
+ | - RHEL-6.3 | ||
+ | </pre> | ||
+ | In the example above, the component defined in the <code>blog</code> service would match against any component known to the system that could satisfy: | ||
+ | <pre> | ||
+ | application: | ||
+ | name: wordpress | ||
+ | role: web | ||
+ | </pre> | ||
+ | given that no other requirements were placed on the component. This requirement translates as "I need a component implementation that can provide an application named <code>wordpress</code> and that can satisfy the <code>web</code> role" | ||
− | + | * '''relations''': Specify connections to other services and/or components. Examples: - wordpress app to mysql database - load balancer to webhead. | |
− | + | * '''options''' - This section of the template identifies the parameters that can be supplied upon provisioning. It follows syntax like component options, except for constraints. See the HOT Options section for more information. | |
− | . | ||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | + | See options section for syntax for expressing more complex logic, like greater than, less than, etc.... | |
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
* '''resources''' - static resources to be created at planning time and shared across the blueprint. For example, users and keys: | * '''resources''' - static resources to be created at planning time and shared across the blueprint. For example, users and keys: | ||
Line 366: | Line 451: | ||
blueprint-type: "Application" | blueprint-type: "Application" | ||
flavor: "Single Server" | flavor: "Single Server" | ||
− | + | ||
− | |||
− | |||
− | |||
* '''help''' - Detailed help on this option (how to use it) | * '''help''' - Detailed help on this option (how to use it) | ||
* '''sample''' - An example of what the data will look like. This could be shown as the background of a text control. | * '''sample''' - An example of what the data will look like. This could be shown as the background of a text control. | ||
Line 527: | Line 609: | ||
==== Heat Blueprint Examples ==== | ==== Heat Blueprint Examples ==== | ||
− | This is | + | Here is the smallest possible <code>Heat Blueprint</code> for launching a single server: |
+ | blueprint: | ||
+ | services: | ||
+ | "server": | ||
+ | component: | ||
+ | resource-type: compute | ||
+ | environment: | ||
+ | providers: | ||
+ | - id: nova: | ||
+ | vendor: openstack | ||
+ | |||
+ | A Hello World <code>Heat Blueprint</code>: | ||
+ | blueprint: | ||
+ | name: "Hello World" | ||
+ | services: | ||
+ | "my_server": | ||
+ | component: | ||
+ | resource-type: compute | ||
+ | interface: linux | ||
+ | options: #Options are used by the user interface, but they are not required for a truly minimal blueprint | ||
+ | memory: | ||
+ | label: Server Size | ||
+ | default: 512 | ||
+ | type: integer | ||
+ | constrains: | ||
+ | - resource_type: compute | ||
+ | setting: memory | ||
+ | os: | ||
+ | label: Operating System | ||
+ | default: Ubuntu 12.04 | ||
+ | type: string | ||
+ | constrains: | ||
+ | - resource_type: compute | ||
+ | setting: os | ||
+ | region: | ||
+ | type: string | ||
+ | environment: | ||
+ | id: rackspace-open-cloud | ||
+ | name: Rackspace Open Cloud | ||
+ | providers: | ||
+ | - id: nova: | ||
+ | vendor: openstack | ||
+ | |||
+ | This is a fragment of a <code>Heat Blueprint</code> using <code>options</code>, blueprint <code>meta-data</code> and schema versioning. | ||
blueprint: | blueprint: | ||
id: 0255a076c7cf4fd38c69b6727f0b37ea | id: 0255a076c7cf4fd38c69b6727f0b37ea | ||
− | name: | + | name: WordPress w/ MySQL on VMs |
meta-data: | meta-data: | ||
schema-version: 0.7 | schema-version: 0.7 | ||
Line 537: | Line 662: | ||
lb: | lb: | ||
component: | component: | ||
− | interface: | + | interface: vip |
type: load-balancer | type: load-balancer | ||
− | |||
− | |||
relations: | relations: | ||
− | web: http | + | web: |
− | + | service: web | |
− | + | interface: http | |
+ | attributes: | ||
+ | inbound: https/443 | ||
+ | algorithm: least_connections | ||
options: | options: | ||
url: | url: | ||
Line 574: | Line 700: | ||
service: master | service: master | ||
attribute: private_key | attribute: private_key | ||
+ | ... | ||
− | + | Fragment of a <code>Heat Blueprint</code> for deploying an HA installation of WordPress: | |
# A WordPress architecture template | # A WordPress architecture template | ||
blueprint: &wp | blueprint: &wp | ||
Line 584: | Line 711: | ||
component: *loadbalancer | component: *loadbalancer | ||
relations: | relations: | ||
− | web: http | + | web: #arbitrary name for relationship |
− | + | service: web | |
− | + | interface: http | |
+ | attributes: | ||
+ | inbound: https/443 | ||
+ | algorithm: least_connections | ||
web: | web: | ||
component: *wordpress_reference_id # wordpress component above | component: *wordpress_reference_id # wordpress component above | ||
relations: {backend: mysql} | relations: {backend: mysql} | ||
backend: | backend: | ||
− | + | component: *mysql | |
options: | options: | ||
instance_count: | instance_count: | ||
Line 602: | Line 732: | ||
constraints: | constraints: | ||
- greater-than: 1 # this is an HA config | - greater-than: 1 # this is an HA config | ||
+ | ... | ||
=== Deployments === | === Deployments === | ||
Line 679: | Line 810: | ||
provider: nova | provider: nova | ||
status: up | status: up | ||
− | + | service: web | |
− | |||
instance: | instance: | ||
id: 2098383 | id: 2098383 | ||
+ | flavor: 1 | ||
+ | image: 119 | ||
private_ip: 10.10.1.1 | private_ip: 10.10.1.1 | ||
public_ip: 2.2.2.18 | public_ip: 2.2.2.18 | ||
Line 692: | Line 824: | ||
type: server | type: server | ||
status: up | status: up | ||
+ | service: web | ||
provider: nova | provider: nova | ||
− | |||
− | |||
instance: | instance: | ||
id: 2098387 | id: 2098387 | ||
+ | flavor: 1 | ||
+ | image: 119 | ||
private_ip: 10.10.1.8 | private_ip: 10.10.1.8 | ||
public_ip: 2.2.2.22 | public_ip: 2.2.2.22 | ||
Line 705: | Line 838: | ||
'2': | '2': | ||
type: load-balancer | type: load-balancer | ||
+ | service: lb | ||
dns-name: CMDEP32ea304-lb1.rackcloudtech.com | dns-name: CMDEP32ea304-lb1.rackcloudtech.com | ||
instance: | instance: | ||
Line 713: | Line 847: | ||
'3': | '3': | ||
type: database | type: database | ||
+ | service: backend | ||
provider: databases | provider: databases | ||
dns-name: CMDEP32ea304-db1.rackcloudtech.com | dns-name: CMDEP32ea304-db1.rackcloudtech.com | ||
− | |||
− | |||
instance: | instance: | ||
id: 99958744 | id: 99958744 | ||
+ | flavor: 1 | ||
+ | disk: 2 | ||
+ | |||
+ | == Comparison with CFN Terminology == | ||
+ | |||
+ | Moved to [[Heat/Vocabulary]] | ||
== API == | == API == | ||
See also: the [[Heat/Open_API]] to support this Open DSL. | See also: the [[Heat/Open_API]] to support this Open DSL. |
Latest revision as of 13:04, 30 April 2013
Contents
- 1 Proposal Only
- 2 Overview
- 3 Roles
- 4 DSL Design Goals
- 5 Terminology
- 6 Options and Inputs
- 7 DSL Foundational Concepts
- 8 Comparison with CFN Terminology
- 9 API
Proposal Only
Please recognize that this is only a proposed DSL to be considered for development. This specification is NOT YET IMPLEMENTED.
An evolution trying to simplify this DSL further is at Heat/DSL2, and further evolution with real templates at gist-5408410, and open questions in an etherpad.
See also:
Overview
This is a DSL to be used with HEAT to allow a simple REST API to allow a more rich set of capabilities to be exposed. It is designed to be declarative in nature, meaning that it will identify what to deploy and orchestrate rather than explicitly how to deploy or orchestrate it. The syntax is intended to be simple, and human readable.
- Anyone, but generally a subject-matter expert (SME), writes a
Heat Blueprint
for how an app can be deployed. - The Heat Blueprint contains
Components
,Relationships
between these components, andOptions
andConstraints
on how the app works. - A User defines
Environments
where they want to deploy apps (ex. a laptop, an OpenStack Cloud, a Rackspace US Cloud account) - The User picks a blueprint (ex. a Scalable WordPress blueprint) and deploys it to an environment of their choice. That's a
Deployment
and results in a fully built and running, multi-component app. - Heat knows how to add/remove servers (scaling) and can verify the app is running and perform troubleshooting (configuration management)
Here is a visual overview of the parts of a Blueprint.
Notes:
- Not all possible relations are shown.
- This diagram is meant as a guide, not a normative reference.
Roles
- Cloud Service Provider - A service entity offering hosted cloud services on OpenStack or another cloud technology. Also known as a Vendor.
- SME - The person responsible for creating Heat Blueprints. He/she encodes the logic and constraints of a system or application based on their own deep knowledge and expertise with the Components of that system. For example a WordPress expert would act as the SME to create the "Best Practice WordPress by Rackspace" Heat Blueprint.
- Tenant - A single customer of a Cloud Service Provider
- User - Deployment End-User - The person using Heat to initiate orchestrations
DSL Design Goals
- Simplicity for the end user should be valued first ahead of all other design goals. Make it easy to read, troubleshoot, debug, etc.
- Flexibility should allow the DSL to be used for a reasonable variety of use cases, but should not aim to support *all* possible use cases.
Terminology
DSL
DSL
is an acronym for Domain Specific Language. This is the syntax used to express a configuration or instructions to exercise the features of the system through its API, while keeping the API itself minimalistic.
Heat Orchestration Template (HOT)
A Heat Orchestration Template
is an orchestration document that details everything that is needed to carry out an orchestration. Think of this as the functional equivalent of an AWS CloudFormation Template. It is expressed as YAML. Note: JSON is a subset of YAML. Any YAML parser will also accept JSON. HOTs may be shared in common among multiple Tenants of different Cloud Service Providers. They can contain all of the vendor independent specifications for launching a particular service or application. For example, Rackspace may publish a "Best Practice WordPress by Rackspace" template that can be used by any Tenant of any Cloud Service Provider. The template is created by a subject-matter expert. See: Heat Blueprints
Environment
An Environment
is a logical target for a Deployment. It is unique to a given user, but independent from Region, Cell, Cloud, etc. A user may have one Environment for Production, another for Staging, and perhaps another for Dev/Test. An arbitrary number of Environments may be specified. A given Environment may mix Components from different Cloud Service Providers, perhaps Compute services come from the local OpenStack cloud, but backups happen on the Rackspace Cloud. See: Environments
Deployment
A Deployment
is comprised of a Heat Blueprint, an Environment and any Inputs. A Deployment may be bundled in a yaml file containing either explicit definitions of or references to these artifacts. Typically a user would refer to existing Environments and Heat Blueprints that they reference. See: Deployments
Component
A Component
is an abstract representation for capabilities or services offered by a Cloud Service. A universal catalog of components identifiers will be created in public repositories in a similar way as Juju Charms and OpsCode cookbooks. See: Components
Resource
Resources represent artifacts of a running Deployment and come in two types:
- A
Static Resource
is an orchestration artifact that can be used to customize a component, such as a security key-pair or user credentials to be used for compute resource access. See: Static Resources - A
Deployment Resource
is an orchestration artifact that represents the instantiation of a component as part of a runningDeployment
such as a compute node, a software artifact, or load balancer. See: Providers and Deployments
Options and Inputs
Options can be exposed by Heat Blueprints and Components. An option is the definition of a user-selectable value that can supplied for a Heat Blueprint or Component.
When launching a Deployment, the values selected for options are stored as an input to the deployment under the 'inputs' key. Inputs can be applied in various scopes in the deployment hierarchy as follows:
- Global inputs (apply to everything):
- inputs: domain: mydomain.com
- Heat Blueprint inputs (apply to a setting on the Heat Blueprint):
- inputs: blueprint: domain: mydomain.com
- See also: Heat Blueprint Options
- Service inputs (apply to a particular service in the blueprint):
- inputs: services: "backend": use_encryption: true
- Provider inputs (apply to a provider and any resourcers that provider provides):
- inputs: providers: 'legacy': region: dallas
- Resource type inputs. These can be applied under services or providers as follows:
- inputs: services: "backend": 'database': 'memory': 512 Mb providers: 'nova': 'compute': 'operating-system': Ubuntu 12.04 LTS
Options can be associated with one or more options using constraints. Example:
blueprint: options: "my_setting": default: 1 constrains: [{service: web, resource_type: compute, setting: foo}]
The above setting would apply to (constrains) any setting called 'foo' under a 'compute' resource in the 'web' service of the blueprint. More precisely scoped options will override broader options. For example, a service or provider option will override a global option.
DSL Foundational Concepts
Components
A component is the equivalent of a Chef recipe or a Juju charm. They are the primitive building blocks of an application deployment. These can be supplied as part of a deployment or looked up from the server.
Example Component
# Definitions of components used (similar to Juju charm syntax) components: - &wordpress_reference_id id: wordpress revision: 3 summary: "A popular blog engine" provides: url: interface: http requires: db: interface: mysql server: relation: host interface: linux options: url: type: String default: wp.test.local description: the url to use to host your blog on - &mysql.1 id: mysql revision: 1 summary: "A popular database. Note, this is a cloud database and therefore does not need a host" provides: db: mysql
- name - A short descriptive string that identifies the component for humans. Example: mssql (not Microsoft SQL Server).
- role - Allows one component to act in several roles(ex. master, slave)
- version - A string representing the version of this component. Example: 1.0.0
- description - A short description of the component. Example: Microsoft SQL Server
- is - Describes the type of resource based on a closed list of OpenStack primitives (compute, database, load-balancer, application, ...)
- requires What this resource needs in order to function. The need can be specific or general. General would be anything with a mysql interface (for WordPress). Specific would be "I need a host compute resource that is an Ubuntu 12.04 or later machine".
- Two options for "requires" Syntax (short form, and long form):
- Short Form
- Syntax: type: interface
- Example:
requires: database: mysql
- Long Form
- Syntax: requirement_key: value
- Example for "I need a host compute resource that is an Ubuntu 12.04 or later machine":
requires: host: # this is a user-supplied identifier type: compute interface: linux distro: debian relation: host
- Note that requirement_keys can be arbitrary to label the requirement. Example requirements from above:
- type: the type of required resource
- interface: the general interface requirement of the resource (should be a flavor of linux)
- distro: an arbitrary key that states that qualifying components must provide distro:debian to satisfy this requirement
- relation: host (this says I am hosted on this resource. I cannot exist without it. I am down when it is down). "host" is a keyword.
- constraint: - 'os': ['debian', 'redhat'] The name+key/value syntax allows for requiring more than one of the same resource (ex. log and data mysql databases) as well as adding additional constraints, etc....
- options - For listing the optional settings I can set on this component.
They take the general form:
<option-name>: default: <value> required: < optional | required | auto-generated> (default: optional) type: < string | integer | boolean | url> (default: string)
Example:
username: default: root required: required type: string
- provides - An array or hash of resource_type:interface entries
- Note: The array can be represented in YAML with entries preceded with dashes.
- For example, let's say we get a Hosting API that provides a "website" resource. Website could provide:
Example:
components: - &website_reference_id id: website provides: - database: mysql - application: name: php version: [5.0, 5.5, 6.3-BETA]
Static Resources and Deployment Resources
A Static Resource
is an artifact that can be defined up-front by the blueprint author and used to customize one or more aspects of the blueprint. A good example is defining a shared security key-pair that will be created and shared by all resources in the blueprint. They can be expressed as arbitrary key:value pairs and are defined in the Heat Blueprint. Static Resources are processed by the system and generate shared Deployment Resources
during provisioning.
Example:
resources: sync-keys: type: key-pair constrains: - setting: private_sync_key resource_type: application service: drupal attribute: private_key - setting: public_sync_key service: drupal resource_type: application attribute: public_key_ssh
Providers
A Provider is a plug-in for Heat that understands how to instantiate and manage a Deployment Resource
via one or more Component
implementations. For Example, the Rackspace Compute Provider knows how to interact with the Rackspace Open Cloud to create compute resources. The OpsCode Chef Provider knows how to operate Chef and apply Cookbooks. Note: Provider != Cloud Service Provider.
Fields:
- vendor - A short string identifying the provenance of the provider. Example: openstack
- provides - A list of what is provided in a list of key:value pairs
- constraints - A list of constraints about how this provider can be offered. Expressed as a list of key:value pairs
Example:
providers: - id: nova vendor: openstack provides: - compute: linux - compute: windows constraints: - region: ORD
Example in the context of an Environment
:
environment: providers: - id: chef-solo vendor: opscode
Action Items:
- A Provider may have algorithms or user-created mapping files to map items from the Heat namespace to the provider-specific namespace. Ex. wordpress/database/db_user for OpsCode Chef. This could be used to dynamically generate component definitions and we need to be able to map things back after they've flowed through Heat.
Providers API
GET /providers GET /tenant_id/environments/environment_id/providers/provider_id/catalog/
Returns a list of components, grouped by resource type
See also: Heat/Open_API
Environments
Environments provide a 'context' and optional constraints as well as a list of providers. For example, if I have an environment with a Cloud Databases provider 'constrained' to region LON where Cloud Databases don't have 32GB instances, then the catalog won't have 32GB instances.
- id - A unique identifier provided by client (or user).
- name - String for convenience to identify the environment
- providers - Predefined keys are names (capabilities) available from the providers. The provider is identified in Heat based on the id and vendor field.
prod_compute:
id: nova: vendor: rackspace
The above results in Heat dynamically loading heat.providers.rackspace.nova (last two are vendor.id)
chef: id: chef-solo vendor: opscode database: id: database vendor: rackspace constraints: - region: DFW catalog: ...
About constraints: optional constraints may be applied at the provider level such as the example below to restrict the region to "DFW". The syntax follows normal constraint syntax, but can be shortened to key:value shorthand. For the normal syntax, a 'value' defines what the constraint evaluates to.
About catalog: - a way to inject a catalog into the provider (two uses for this are #1 testing, #2 only want to show 1GB instances). So if a catalog is provided, the provider will use that. Otherwise, it could log on and query the underlying service (list images, list flavors, get cookbooks, etc...). You'll see this in app.yaml.
The Environment will define what this looks like based on environment providers and constraints. A blueprint may be incompatible with an environment if the environment cannot provide the required resources OR meet the stated or inherited constraints.
Note that defining an Environment in a Deployment is optional and the system will default to considering all providers and their catalogs when attempting to satisfy the components requested in a Blueprint. Conflict resolution is an area for further discussion and design and should influence the request and response cycle of deployment api endpoints.
Example Environment
# Environment environment: &environment_1000_stag name: Rackspace Cloud US - staging providers: nova: id: nova vendor: rackspace provides: - compute: linux - compute: windows constraints: - region: ORD lb: id: load-balancer vendor: rackspace provides: - loadbalancer: http database: id: database vendor: rackspace provides: - database: mysql chef: id: chef-server vendor: opscode provides: - application: http # see catalog for list of apps like wordpress, drupal, etc... - database: mysql # this is mysql installed on a host
Heat Orchestration Templates (HOT)
These define the architecture for an application. The template describes the resources needed to make an application run and which components to connect. HOTs can have options that determine the final deployment topology and the values that go into the individual component options. The SME determines what options to expose and with what constraints to apply on the options available to the end user.
An application is a collection of components that provide a useful capability. An application is distinct from infrastructure. A HOT describes a design that provides an application with certain 'ilities (scalability, availability, etc...). It defines the components, their relationships to each other, and the various constraints on the components and relationships that must be met in order for the application to work as specified.
When expressing services, Heat Orchestration Templates contain:
- One or more services (all under one 'services' entry)
- An arbitrary name for service ID (each one has its own ID)
- One or more components per service with the expectation that we'll be able to pull in nested templates (via inclusion or reference) as Components in the future.
- Relations between services
Attributes:
- id - unique identifier provided by the user/client
- name - a short string naming the service
- description: a more detailed description of the service
- services - like tiers, but not restricted to the concept of tiers. Currently, there is one component defined per service.
Example HOT:
name: Simple Wordpress services: blog: requires: - compute: type: linux constraints: os: {option: linuxdistribution} flavor: {option: instancetype} components: wordpress: requires: - application: name: wordpress role: web relations: - blog: db db: requires: - database: type: mysql constraints: name: { option: dbname } username: { option: dbusername } password: { option: dbpassword } rootpw: { option: dbrootpassword } options: instancetype: description: Webserver instance type type: string default: m1.large constraints: - in: values: - m1.large - m1.xlarge - m2.large - m2.xlarge message: "Must be one of m1.large, m1.xlarge, m2.large, m2.xlarge" dbname: description: The wordpress database name type: string default: wordpress constraints: - len: min: 1 max: 64 message: Must be between 1 and 64 characters - matches: expr: [a-zA-Z][a-zA-Z0-9]* message: must begin with a letter and contain only alphanumeric characters dbusername: description: The wordpress database admin account username type: string default: admin constraints: - len: min: 1 max: 16 message: must be between 1 and 16 characters - matches: expr: [a-zA-Z0-9]* message: must contain only alphanumeric characters dbpassword: description: The WordPress database admin account password type: password default: admin constraints: - len: min: 1 max: 41 message: must be between 1 and 41 characters - matches: expr: [a-zA-Z0-9]* message: must contain only alphanumeric characters dbrootpassword: description: Root password for MySQL type: password default: admin constraints: - len: min: 1 max: 41 message: must be between 1 and 41 characters - matches: expr: [a-zA-Z0-9]* message: must contain only alphanumeric characters linuxdistribution: description: Distribution of choice type: string default: F17 constraints: - in values: - F18 - F17 - U10 - RHEL-6.1 - RHEL-6.2 - RHEL-6.3
In the example above, the component defined in the blog
service would match against any component known to the system that could satisfy:
application: name: wordpress role: web
given that no other requirements were placed on the component. This requirement translates as "I need a component implementation that can provide an application named wordpress
and that can satisfy the web
role"
- relations: Specify connections to other services and/or components. Examples: - wordpress app to mysql database - load balancer to webhead.
- options - This section of the template identifies the parameters that can be supplied upon provisioning. It follows syntax like component options, except for constraints. See the HOT Options section for more information.
See options section for syntax for expressing more complex logic, like greater than, less than, etc....
- resources - static resources to be created at planning time and shared across the blueprint. For example, users and keys:
my_key: type: key-pair
Note: private/public key pair will be created before deploying the workflow constrains:
service: my_database_thang setting: key attribute: private_key
The above example will take the private_key value from the generated keys and apply it as the value for 'key' in the my_database_thang component.
Heat Blueprint Options
You can specify options and inputs on Heat Blueprints. The supported option fields are detailed here.
Fields
- label - The short label to use when displaying an option to the user. description: A full description of this option (what it is)
- meta-data - A place where you can hang additional information, such as the version of the schema used, or other arbitrary data needed by your subsystems.
- schema-version - A string that identifies the schema version used in this Heat Blueprint. Example: 0.7
Example:
meta-data: schema-version: 1.0 application-name: "Drupal" blueprint-type: "Application" flavor: "Single Server"
- help - Detailed help on this option (how to use it)
- sample - An example of what the data will look like. This could be shown as the background of a text control.
- display-hints - a mapping (key, value pairs) of hints for how to order and display this option in relation to other options ....
- group - a group name used to group options together. Examples to hint Horizon:
- deployment: this is a deployment option and shows right under the deployment name
- application: this is an application option and shows on the first screen of options
- compute: this is an option that should be shown under the compute server options section
- load-balancer: this is an option that should be shown under the load balancer options section
- database: this is an option that should be shown under the database options section
- dns: this is an option that should be shown under the dns options section
- order - Indicate the relative order of this option within its group (as an integer)
- list-type - What is the type of the entries in the list. Used to identify if it should be a specific resource type and list or attribute. The format is resource-type.list where resource-type is a known Heat resource type (compute, database, etc...) and the list is one of the lists from the provider [TODO: define these lists more precisely in the DSL or schema. Right now, they exist only in the provider catalogs]. Examples (and what we will initially support):
- compute.memory - list of available compute image sizes
- _compute.os - list of available compute operating systems
- load-balancer.algorithm - list of available load balancer algorithms
- encrypted-protocols - The subset of protocols that are encrypted so that we know when to show ssl cert controls. Ex. [https, pop3s]
- always-accept-certificates - if a blueprint always accepts and handles the certificates (especially if the url is entered in free-form supporting any protocol)
- default - The default value to use for the option.
NOTE: YAML will assume numbers are ints, so enclose strings in "quotation marks" as a best practice. Special values for this are =generate_password() which will generate a random password on the server. TODO: Consider adding parameters to generate_password() so the heat Blueprint author can make the password generated match their (or their application's) requirements. This would be used by the blueprint author in tandem with constraints (below) for validation on the client side.
- type - the data type of this option. Valid types are: string, integer, boolean, password, url, region, and text (multi-line string). See later for a description of the url type which has some special attributes.
- choice - a list of items to select from (used to display a drop-down). The entries are either plain strings or a mapping with value and name entries where value is what is passed to Heat and name is what is displayed to the user. Note: does not apply validation. If you want validation, use in a constraint. This is used for display only. Example:
choice: - name: Ubuntu 12.04 value: q340958723409587230459872345 - name: Ubuntu 12.10 value: 2384729387w0tw9879t87ywt3y42
- constrains - A list of mappings used as a way to set or limit aspects of the blueprint with the value (or parts of) the option.
- required - Boolean. Set to true if this option must be supplied by the user.
- constraints - An array of mappings (key/value pairs) in constraints syntax. Supported constraints are:
- greater-than
- less-than
- greater-than-or-equal-to
- less-than-or-equal-to
- min-length - for strings
- max-length - for strings
- allowed-chars - Example: "ABCDEFGabcdefg01234565789!&@" - TODO: see if regex can be used here
- required-chars - Example: "ABCDEFG" - TODO: see if regex can be used here
- in - a list of acceptable values (these could also be used by clients to display drop-downs)
- protocols - Used for URL types. This lists allowed protocols in the URL.
- regex - Do not use look-forward/behind. Keep these simple so they are supported in javascript (client) and python (server). While many of the above can also be written as regex rules, both are available to Heat Blueprint authors to use the one that suits them best.
- constraints: message - You can add a message key/value pair to any constraint. We recommend adding a message to regex constraints so it is easy to understand what they do when reviewed and so clients/servers can generate useful error messages and people reading the blueprint don't have to decipher the regexs. Ex. "must have 8-16 characters"
Browser clients may parse constraints and apply validation rules. A good practice is to have multiple simple regex constraints to allow browser clients to provide clear and useful feedback to the user for each rule they may break. For example, list lowercase, uppercase, and numeric requirements for a password as three constraints with a message unique to each.
Example Heat Blueprint with Options:
blueprint: options: database_name: label: Database Name sample: db1 display-hints: order: 1 group: database default: wp_db description: "This is the name of the database that will be created to host your application's data" type: string constraints: - regex: ^(?=.*).{2,15}$ message: must be between 2 and 15 characters long - regex: ^[A-Za-z0-9]*$ message: can only contain alphanumeric characters database_password: label: Database Password display-hints: order: 2 group: database description: "This is the password to use to access the database that will host your application's data" type: password constraints: - regex: ^(?=.*).{8,15}$ message: must be between 8 and 15 characters long - regex: ^(?=.*\d) message: must contain a digit - regex: ^(?=.*[a-z]) message: must contain a lower case letter - regex: ^(?=.*[A-Z]) message: must contain an upper case letter
Common Types
The URL Type
Options of type url provide some advanced handling of common url use cases. The option can be used simply as a string that accepts a url. In this case, the only benefit of setting the type to url is that a client application can perform certain validation to make sure the provided value is a valid URL (according to RFC 3986).
Example:
option: my_web_site: type: url
It is useful, however, to be able to handle different parts of a URL (i.e the scheme or protocol, domain, path, port, username, password, etc...) separately. They may be validated independently (e.g. make sure the protocol is http or https only). The parts may be wired up to different parts of the blueprint using constraints (e.g. use the domain part for a dns setting). The way that is supported is that the url type has attributes that can be accessed in the Heat Blueprint or other parts of Heat. These attributes are:
* scheme - this is the first part of the URL * protocol - this is the first part of the URL as well (an alias to scheme) * netloc - the dns name or address part * port - this is the port if specified (e.g. the port in http://localhost:8080 is 8080) * path - the path of the resource or file * private_key - the private_key of a certificate to use if the protocol is an encrypted one * public_key - the public_key of a certificate to use if the protocol is an encrypted one * intermediate_key - the intermediate key chain of a certificate to use if the protocol is an encrypted one
These attributes can be specified in constraints:
options: my_url: label: Site Address type: url constraints: - protocols: [http, https] constrains: - type: load-balancer service: lb attribute: protocol # This picks out the 'http' or 'https' part of the URL setting: protocol - type: compute service: web attribute: "private_key" # This picks out the cert setting: ssl_certificate - type: compute service: web attribute: "intermediate_key" # This picks up the intermediate cert setting: ssl_intermediate_certificate
You can constrain a list of protocols using the protocols
constraint.
options: my_url: label: Site Address type: url constraints: - protocols: [http, https]
And there are special display-hints used to aid a client in rendering and validating the url. These are encrypted-protocols
and always-accept-certificates
which are documented in constraints.
When supplying the value for a url as an input, it can be supplied as a string or as a mapping with attributes.
As a string it would be my_site_address: https://mydomain.com/blog
.
As a mapping, it would look be:
inputs: my_url: url: https://domain.com/path # 'url' is a special shortcut - see note below private_key: | ----- BEGIN ... intermediate_key: | ----- BEGIN ... public_key: | ----- BEGIN ...
Note: A common use case is to supply the url and keys. A shortcut is available that accepts a key called url
that can be used to supply the url without having to provide all the components of the url.
Heat Blueprint Examples
Here is the smallest possible Heat Blueprint
for launching a single server:
blueprint: services: "server": component: resource-type: compute environment: providers: - id: nova: vendor: openstack
A Hello World Heat Blueprint
:
blueprint: name: "Hello World" services: "my_server": component: resource-type: compute interface: linux options: #Options are used by the user interface, but they are not required for a truly minimal blueprint memory: label: Server Size default: 512 type: integer constrains: - resource_type: compute setting: memory os: label: Operating System default: Ubuntu 12.04 type: string constrains: - resource_type: compute setting: os region: type: string environment: id: rackspace-open-cloud name: Rackspace Open Cloud providers: - id: nova: vendor: openstack
This is a fragment of a Heat Blueprint
using options
, blueprint meta-data
and schema versioning.
blueprint: id: 0255a076c7cf4fd38c69b6727f0b37ea name: WordPress w/ MySQL on VMs meta-data: schema-version: 0.7 services: lb: component: interface: vip type: load-balancer relations: web: service: web interface: http attributes: inbound: https/443 algorithm: least_connections options: url: label: Site Address description: 'The domain you wish to host your blog on. (ex: example.com)' type: url required: true default: http://example.com/ display-hints: group: application order: 1 encrypted-protocols: [https] sample: http://example.com/ constraints: - protocols: [http, https] - regex: '^([a-zA-Z]{2,}?\:\/\/[a-zA-Z0-9\-]+(?:\.[a-zA-Z0-9\-]+)*\.[a-zA-Z]{2,6}(?:\/?|(?:\/[\w\-]+)*)(?:\/?|\/\w+\.[a-zA-Z]{2,4}(?:\?[\w]+\=[\w\-]+)?)?(?:\&[\w]+\=[\w\-]+)*)$' message: must be a valid web address constrains: - setting: allow_insecure service: lb resource_type: load-balancer value: true # turn on HTTP if protocol is HTTPS (provider handles it) resources: sync-keys: type: key-pair constrains: - setting: private_sync_key resource_type: application service: master attribute: private_key ...
Fragment of a Heat Blueprint
for deploying an HA installation of WordPress:
# A WordPress architecture template blueprint: &wp id: "6fcc7f31-08f8-4664-90e3-58fffc71f773" name: Multi-server Wordpress services: lb: component: *loadbalancer relations: web: #arbitrary name for relationship service: web interface: http attributes: inbound: https/443 algorithm: least_connections web: component: *wordpress_reference_id # wordpress component above relations: {backend: mysql} backend: component: *mysql options: instance_count: type: number label: Number of Instances description: The number of instances for the specified task. default: 2 constrains: - {service: web, resource_type: compute, setting: count} constraints: - greater-than: 1 # this is an HA config ...
Deployments
A deployment defines and points to a running application and the infrastructure it is running on. It basically says "I took blueprint X and deployed it to environment Y using the following options". It combines a blueprint, an environment to deploy the resources to, and any additional inputs specific to this deployment.
General form:
- id - a User defined string id
- name: A short string identifying the Deployment
- blueprint: can be a reference (YAML), but for now, it's a full copy of a Heat Blueprint. We'll support references using URI later (git, local references, etc...).
- environment: same as blueprint - right now we make a full copy
- inputs: - these are for setting levers and dials... there are different scopes: global, blueprint, service, and provider scopes. So you can set the memory size at the provider level (i.e. all servers should be 1GB). Or, all servers in service X should be 1GB and all servers in service Y should be 2GB.
option_foo: bar
Example Heat Blueprint using an input
blueprint: blueprint_input_foo: bar services: my_db_thang: service_input_foo: baz compute: (filter by resource type!) service_resource_input_foo: boo providers: 'nova': compute: os: ubuntu
Note: More specific inputs override more general ones (see get_setting code in deployment)
Action Items
- Determine the need for global inputs. Do we really need them? Or at least put them in a "globals" category
- Define how we reference blueprints and environments in deployments without including full copies. Reference UUID? URL to file in git or local reference? etc.
- Improve syntax for indicating a generated value, or remove it and let a generated value be the default. Or create system for code to be included in blueprint to be used to generate, validate values. See artifacts prototype in app.yaml.
Examples of Deployments:
deployment: blueprint: name: Simple Wordpress wordpress: config: *wordpress # points to wordpress component relations: {db: mysql} environment: name: rackcloudtech-test providers: - compute: vendor: rackspace constraints: - region: ORD - loadbalancer: &rax-lbaas - database: &rax-dbaas - common: vendor: rackspace credentials: - rackspace: ...
Example Deployment with provides
and requires
in components and providers as keys:
deployment: environment: providers: nova: provides: - compute vendor: rackspace
Example Deployment
# Actual running app and the parameters supplied when deploying it deployment: blueprint: *wp environment: *environment_1000_stag inputs: instance_count: 4 resources: '0': type: server provider: nova status: up service: web instance: id: 2098383 flavor: 1 image: 119 private_ip: 10.10.1.1 public_ip: 2.2.2.18 dns-name: srv1.stabletransit.com relations: web-backend: state: up '1': type: server status: up service: web provider: nova instance: id: 2098387 flavor: 1 image: 119 private_ip: 10.10.1.8 public_ip: 2.2.2.22 dns-name: srv2.stabletransit.com relations: web-backend: state: up '2': type: load-balancer service: lb dns-name: CMDEP32ea304-lb1.rackcloudtech.com instance: id: 8668444 relations: lb-web: state: up '3': type: database service: backend provider: databases dns-name: CMDEP32ea304-db1.rackcloudtech.com instance: id: 99958744 flavor: 1 disk: 2
Comparison with CFN Terminology
Moved to Heat/Vocabulary
API
See also: the Heat/Open_API to support this Open DSL.