Difference between revisions of "Heat/DSL"
Ziad Sawalha (talk | contribs) (→Roles) |
Ziad Sawalha (talk | contribs) m (→Component) |
||
Line 36: | Line 36: | ||
=== Component === | === Component === | ||
− | A Component is an abstract representation for capabilities or services offered by a Cloud Service | + | A Component is an abstract representation for capabilities or services offered by a Cloud Service. A universal catalog of components identifiers will be created in public repositories in a similar way as [http://jujucharms.com Juju Charms] and OpsCode [https://github.com/opscode-cookbooks cookbooks]. |
+ | |||
=== Options and Inputs === | === Options and Inputs === | ||
Options can be exposed by Heat Blueprints and Components. An option is the definition of a user-selectable value that can supplied for a Heat Blueprint or Component. | Options can be exposed by Heat Blueprints and Components. An option is the definition of a user-selectable value that can supplied for a Heat Blueprint or Component. |
Revision as of 15:02, 11 April 2013
Contents
DRAFT ONLY - WORK IN PROGRESS
Please recognize that this is only a proposed DSL to be considered for development. This is NOT YET IMPLEMENTED.
Overview
This is a DSL to be used with HEAT to allow a simple REST API to allow a more rich set of capabilities to be exposed. It is designed to be declarative in nature, meaning that it will identify what to deploy and orchestrate rather than explicitly how to deploy or orchestrate it. The syntax is intended to be simple, and human readable.
- Anyone, but generally a subject-matter expert (SME), writes a
Heat Blueprint
for how an app can be deployed. - The Heat Blueprint contains
Components
,Relationships
between these components, andOptions
andConstraints
on how the app works. - A User defines
Environments
where they want to deploy apps (ex. a laptop, an OpenStack Cloud, a Rackspace US Cloud account) - The User picks a blueprint (ex. a Scalable WordPress blueprint) and deploys it to an environment of their choice. That's a
Deployment
and results in a fully built and running, multi-component app. - Heat knows how to add/remove servers (scaling) and can verify the app is running and perform troubleshooting (configuration management)
Roles
- SME - The person responsible for creating Heat Blueprints. He/she encodes the logic and constraints of a system or application based on their own deep knowledge and expertise with the Components of that system. For example a WordPress expert would act as the Architect to create the "Best Practice WordPress by Rackspace" Heat Blueprint.
- Tenant - A single customer of a Cloud Service Provider
- User - Deployment End-User - The person using Heat to initiate orchestrations
- Cloud Service Provider - A service entity offering hosted cloud services on OpenStack or another cloud technology. Also known as a Vendor.
DSL Design Goals
- Simplicity for the end user should be valued first ahead of all other design goals.
- Human Readability. Artifacts presented to the orchestration API should be human readable for simplified troubleshooting and development. See Simplicity.
- Flexibility should allow the DSL to be used for a reasonable variety of use cases, but should not aim to support *all* possible use cases.
Terminology
DSL
DSL = Domain Specific Language. This is the syntax used to express a configuration or instructions to exercise the features of the system through it's API, without while keeping the API itself minimalistic.
Blueprint
A Heat Blueprint (not to be confused with Launchpad Blueprint) is an orchestration document that details everything that is needed to carry out an orchestration. Think of this as the functional equivalent of an AWS CloudFormation Template. It is expressed as YAML. Note: JSON is a subset of YAML. Any YAML parser will also accept JSON. YAML offers easier readability for humans. Heat Blueprints may be shared in common among multiple Tenants of different Cloud Service Providers. They can contain all of the vendor independent specifications for launching a particular service or application. For example, Rackspace may publish a "Best Practice WordPress by Rackspace" Heat Blueprint that can be used by any Tenant of any Cloud Service Provider. The Heat Blueprint is created by an Architect
Environment
A logical target for a Deployment. It is unique to a given user, but independent from Region, Cell, Cloud, etc. A user may have one Environment for production, another for Staging, and perhaps another for dev/test. An arbitrary number of environments may be specified. A given Environment may mix Components from different Cloud Service Providers, perhaps Compute services come from the local OpenStack cloud, but backups happen on the Rackspace Cloud.
Deployment
A Deployment is comprised of a Heat Blueprint and an Environment. A Deployment may be bundled in an app.yaml file that contains the Deployment, Heat Blueprint, and Environment all together for convenience. Typically a user would refer to existing Environments and Heat Blueprints that they reference.
Component
A Component is an abstract representation for capabilities or services offered by a Cloud Service. A universal catalog of components identifiers will be created in public repositories in a similar way as Juju Charms and OpsCode cookbooks.
Options and Inputs
Options can be exposed by Heat Blueprints and Components. An option is the definition of a user-selectable value that can supplied for a Heat Blueprint or Component.
When launching a Deployment, the values selected for options are stored as an input to the deployment under the 'inputs' key. Inputs can be applied at multiple levels in the deployment hierarchy as follows:
- Global inputs (apply to everything):
- inputs: domain: mydomain.com
- Heat Blueprint inputs (apply to a setting on the Heat Blueprint):
- inputs: blueprint: domain: mydomain.com
- Service inputs (apply to a particular service in the blueprint):
- inputs: services: "backend": use_encryption: true
- Provider inputs (apply to a provider and any resourcers that provider provides):
- inputs: providers: 'legacy': region: dallas
- Resource type inputs. These can be applied under services or providers as follows:
- inputs: services: "backend": 'database': 'memory': 512 Mb providers: 'nova': 'compute': 'operating-system': Ubuntu 12.04 LTS
Options can be associated with one or more options using constraints. Example:
blueprint: options: "my_setting": default: 1 constrains: [{service: web, resource_type: compute, setting: foo}]
The above setting would apply to (constrains) any setting called 'foo' under a 'compute' resource in the 'web' service of the blueprint. More precisely scoped options will override broader options. For example, a service or provider option will override a global option.
DSL Resources
Component
A component is the equivalent of a Chef recipe or a Juju charm. They are the primitive building blocks of an application deployment. These can be supplied as part of a deployment or looked up from the server.
Example Component
# Definitions of components used (similar to Juju charm syntax) components: - &wordpress_reference_id id: wordpress revision: 3 summary: "A pretty popular blog engine" provides: url: interface: http requires: db: interface: mysql server: relation: host interface: linux options: url: type: String default: wp.test.local description: the url to use to host your blog on - &mysql.1 id: mysql revision: 1 summary: "A pretty popular database. Note, this is a cloud database and therefore does not need a host" provides: db: mysql
Provider
A Provider is a plug-in for Heat that understands how to operate a service or system. For Example, the Rackspace Compute Provider knows how to interact with the Rackspace Open Cloud to create compute resources. The OpsCode Chef Provider knows how to operate Chef and apply Cookbooks. Note: Provider != Cloud Service Provider.
- name - A short descriptive string that identifies the component for humans. Example: mssql (not Microsoft SQL Server)
- role - Allows the
- version - A string representing the version of this component. Example: 1.0.0
- description - A short description of the component. Example: Microsoft SQL Server
- is - Describes the type of resource based on a closed list of OpenStack primitives (compute, database, load-balancer, application, ...)
- requires What this resource needs in order to function. The need can be specific or general. General would be anything with a mysql interface (for WordPress). Specific would be "I need a host compute resource that is an Ubuntu 12.04 or later machine".
- Two options for "requires" Syntax (short form, and long form):
- Short Form
- Syntax: type: interface
- Example:
database: mysql
- Long Form
- Syntax: requirement_key: value
- Example for "I need a host compute resource that is an Ubuntu 12.04 or later machine":
server: compute interface: debian relation: host
- Note that requirement_keys can be arbitrary to label the requirement. Example requirements from above:
- server: (this is the arbitrary key - the label of this requirement) of type type: compute
- interface: debian (this says I expect to be able to do apt-get, vs yum)
- relation: host (this says I am hosted on this resource. I cannot exist without it. I am down when it is down). "host" is a keyword.
- constraint: - 'os': ['debian', 'redhat'] The name+key/value syntax allows for requiring more than one of the same resource (ex. log and data mysql databases) as well as adding additional constraints, etc....
- provides - An array of resource_type:interface entries
- Note: The array can be represented in YAML with entries preceded with dashes.
- For example, let's say we get a Hosting API that provides a "website" resource. Website could provide:
database: mysql application: php
- options - For listing the optional settings I can set on this component. Example:
username: default: root required: optional | required | auto-generated (default: optional) type: string (currently we support string, int, boolean) (default: string)
Action Items:
- How can we tie this to a "list" from the provider label: User Name (used for display friendliness) description: ... source_field_name: the name as it is known by the underlying provider. Ex. wordpress/database/db_user for OpsCode Chef. We need this currently since we dynamically generate component definitions and we need to be able to map things back after they've flowed through Heat.
- Should be able to get rid of this in the future with clean, reversible mapping logic in providers. type: string regex: for validation sample: for display, to show what this looks like. help: help text. password:
Providers API
GET /providers GET /tenant_id/environments/environment_id/providers/provider_id/catalog/
Returns a list of components, grouped by resource type
Environments
Environments provide a 'context' and optional constraints as well as a list of providers. For example, if I have an environment with a Cloud Databases provider 'constrained' to region LON where Cloud Databases don't have 32GB instances, then the catalog won't have 32GB instances.
- id - A unique identifier provided by client (or user).
- name - String for convenience to identify the environment
- providers - Predefined keys are names (capabilities) available from the providers. The provider is identified in Heat based on the key and vendor field.
nova: vendor: rackspace
The above results in Heat dynamically loading heat.providers.rackspace.nova (last two are vendor.key)
chef-solo: vendor: opscode database: vendor: rackspace constraints: region=DFW catalog: ...
About constraints: optional constraints may be applied at the provider level such as the example below to restrict the region to "DFW". The syntax follows normal constraint syntax, but can be shortened to key:value shorthand. For the normal syntax, a 'value' defines what the constraint evaluates to.
About catalog: - a way to inject a catalog into the provider (two uses for this are #1 testing, #2 only want to show 1GB instances). So if a catalog is provided, the provider will use that. Otherwise, it could log on and query the underlying service (list images, list flavors, get cookbooks, etc...). You'll see this in app.yaml.
The Environment will define what this looks like based on environment providers and constraints. A blueprint may be incompatible with an environment if the environment cannot provide the required resources OR meet the stated or inherited constraints.
Example Environment
# Environment environment: &environment_1000_stag name: Rackspace Cloud US - staging providers: nova: vendor: rackspace provides: - compute: linux - compute: windows constraints: - region: ORD load-balancer: vendor: rackspace provides: - loadbalancer: http database: vendor: rackspace provides: - database: mysql chef-solo: vendor: opscode provides: - application: http # see catalog for list of apps like wordpress, drupal, etc... - database: mysql # this is mysql installed on a host
Heat Blueprints
These define the architecture for an application. The blueprint describes the resources needed to make an application run, how to connect, and how scale them. Heat Blueprints can have options that determine the final deployment topology and the values that go into the individual component options. The Architect determines what options to expose and with what constraints to apply on the options available to the end user.
An application is a collection of components that provide a useful capability. An application is distinct from infrastructure. A Heat Blueprint describes a design that provides an application with certain 'ilities (scalability, availability, etc...). It defines the components, their relationships to each other, and the various constraints on the components and relationships that must be met in order for the application to work as specified.
When expressing services, Heat Blueprints contain:
- One or more services (all under one 'services' entry)
- An arbitrary name for service ID (each one has its own ID)
- One component per service with the expectation that we'll be able to pull in Heat Blueprints as Components in the future.
- Relations between services
Attributes:
- id - unique identifier provided by the user/client
- name - a short string naming the service
- description: a more detailed description of the service
- services - like tiers, but not restricted to the concept of tiers. Currently, there is one component defined per service.
Example Heat Blueprint:
id: wordpress name: my_wp_blog description: My WordPress Blog
The id or key/value pairs are used find a suitable component. The example above would match against a component like this:
component: wordpress ...
When using a key/value pair:
component: type: application interface: http constraints: - count: 2
Note: We have not yet defined any schema to be able to say something like "greater than one"
- relations: Specify connections to other services. Examples: - wordpress app to mysql database - load balancer to webhead Two syntaxes for relations; short and long:
Short syntax: service_name:interface. Example:
my_db_thang: mysql
Long syntax: arbitrary name, key values. Example:
db: interface: mysql service: my_db_thang
- options - This is critical part of a Heat Blueprint that identifies the parameters that can be supplied upon instantiation. It follows syntax like component options, except for constraints. See the Heat Blueprint Options section for more information.
- database_bigness - Option for the size of the database default: 2GB
Action Item: Need to define syntax/schema for expressing more complex logic, like greater than, less than, etc....
- constrains - Specifies what values within components that this option constrains. Key/value to select the options, and then the option name.
Example:
service: my_database_thang setting: memory service: my_database_log_thang setting: memory choice: - value: 1024 name: 1gig - value: 2048 name: 2GB
- resources - static resources to be created at planning time and shared across the blueprint. For example, users and keys:
my_key: type: key-pair
Note: private/public key pair will be created before deploying the workflow constrains:
service: my_database_thang setting: key attribute: private_key
The above example will take the private_key value from the generated keys and apply it as the value for 'key' in the my_database_thang component.
Heat Blueprint Options
Fields
- label - the short label to use when displaying the option to the user. description: A full description of this option (what it is)
- help - Detailed help on this option (how to use it)
- sample - An example of what the data will look like. This could be shown as the background of a text control.
- display-hints - a mapping (key, value pairs) of hints for how to order and display this option in relation to other options ....
- group - a group name used to group options together. Examples to hint Horizon:
- deployment: this is a deployment option and shows right under the deployment name
- application: this is an application option and shows on the first screen of options
- compute: this is an option that should be shown under the compute server options section
- load-balancer: this is an option that should be shown under the load balancer options section
- database: this is an option that should be shown under the database options section
- dns: this is an option that should be shown under the dns options section
- order - Indicate the relative order of this option within its group (as an integer)
- list-type - What is the type of the entries in the list. Used to identify if it should be a specific resource type and list or attribute. The format is resource-type.list where resource-type is a known Heat resource type (compute, database, etc...) and the list is one of the lists from the provider [TODO: define these lists more precisely in the DSL or schema. Right now, they exist only in the provider catalogs]. Examples (and what we will initially support):
- compute.memory - list of available compute image sizes
- _compute.os - list of available compute operating systems
- load-balancer.algorithm - list of available load balancer algorithms
- encrypted-protocols - The subset of protocols that are encrypted so that we know when to show ssl cert controls. Ex. [https, pop3s]
- always-accept-certificates - if a blueprint always accepts and handles the certificates (especially if the url is entered in free-form supporting any protocol)
- default - The default value to use for the option.
NOTE: YAML will assume numbers are ints, so enclose strings in "quotation marks" as a best practice. Special values for this are =generate_password() which will generate a random password on the server. TODO: Consider adding parameters to generate_password() so the heat Blueprint author can make the password generated match their (or their application's) requirements. This would be used by the blueprint author in tandem with constraints (below) for validation on the client side.
- type - the data type of this option. Valid types are: string, integer, boolean, password, url, region, and text (multi-line string). See later for a description of the url type which has some special attributes.
- choice - a list of items to select from (used to display a drop-down). The entries are either plain strings or a mapping with value and name entries where value is what is passed to Heat and name is what is displayed to the user. Note: does not apply validation. If you want validation, use in a constraint. This is used for display only. Example:
choice: - name: Ubuntu 12.04 value: q340958723409587230459872345 - name: Ubuntu 12.10 value: 2384729387w0tw9879t87ywt3y42
- constrains - A list of mappings used as a way to set or limit aspects of the blueprint with the value (or parts of) the option.
- required - Boolean. Set to true if this option must be supplied by the user.
- constraints - An array of mappings (key/value pairs) in constraints syntax. Supported constraints are:
- greater-than
- less-than
- greater-than-or-equal-to
- less-than-or-equal-to
- min-length - for strings
- max-length - for strings
- allowed-chars - Example: "ABCDEFGabcdefg01234565789!&@" - TODO: see if regex can be used here
- required-chars - Example: "ABCDEFG" - TODO: see if regex can be used here
- in - a list of acceptable values (these could also be used by clients to display drop-downs)
- protocols - Used for URL types. This lists allowed protocols in the URL.
- regex - Do not use look-forward/behind. Keep these simple so they are supported in javascript (client) and python (server). While many of the above can also be written as regex rules, both are available to Heat Blueprint authors to use the one that suits them best.
- constraints: message - You can add a message key/value pair to any constraint. We recommend adding a message to regex constraints so it is easy to understand what they do when reviewed and so clients/servers can generate useful error messages and people reading the blueprint don't have to decipher the regexs. Ex. "must have 8-16 characters"
Browser clients may parse constraints and apply validation rules. A good practice is to have multiple simple regex constraints to allow browser clients to provide clear and useful feedback to the user for each rule they may break. For example, list lowercase, uppercase, and numeric requirements for a password as three constraints with a message unique to each.
Example Heat Blueprint with Options:
blueprint: options: database_name: label: Database Name sample: db1 display-hints: order: 1 group: database default: wp_db description: "This is the name of the database that will be created to host your application's data" type: string constraints: - regex: ^(?=.*).{2,15}$ message: must be between 2 and 15 characters long - regex: ^[A-Za-z0-9]*$ message: can only contain alphanumeric characters database_password: label: Database Password display-hints: order: 2 group: database description: "This is the password to use to access the database that will host your application's data" type: password constraints: - regex: ^(?=.*).{8,15}$ message: must be between 8 and 15 characters long - regex: ^(?=.*\d) message: must contain a digit - regex: ^(?=.*[a-z]) message: must contain a lower case letter - regex: ^(?=.*[A-Z]) message: must contain an upper case letter
Common Types
The URL Type
Options of type url provide some advanced handling of common url use cases. The option can be used simply as a string that accepts a url. In this case, the only benefit of setting the type to url is that a client application can perform certain validation to make sure the provided value is a valid URL (according to RFC 3986).
Example:
option: my_web_site: type: url
It is useful, however, to be able to handle different parts of a URL (i.e the scheme or protocol, domain, path, port, username, password, etc...) separately. They may be validated independently (e.g. make sure the protocol is http or https only). The parts may be wired up to different parts of the blueprint using constraints (e.g. use the domain part for a dns setting). The way that is supported is that the url type has attributes that can be accessed in the Heat Blueprint or other parts of Heat. These attributes are:
* scheme - this is the first part of the URL * protocol - this is the first part of the URL as well (an alias to scheme) * netloc - the dns name or address part * port - this is the port if specified (e.g. the port in http://localhost:8080 is 8080) * path - the path of the resource or file * private_key - the private_key of a certificate to use if the protocol is an encrypted one * public_key - the public_key of a certificate to use if the protocol is an encrypted one * intermediate_key - the intermediate key chain of a certificate to use if the protocol is an encrypted one
These attributes can be specified in constraints:
options: my_url: label: Site Address type: url constraints: - protocols: [http, https] constrains: - type: load-balancer service: lb attribute: protocol # This picks out the 'http' or 'https' part of the URL setting: protocol - type: compute service: web attribute: "private_key" # This picks out the cert setting: ssl_certificate - type: compute service: web attribute: "intermediate_key" # This picks up the intermediate cert setting: ssl_intermediate_certificate
You can constrain a list of protocols using the protocols
constraint.
options: my_url: label: Site Address type: url constraints: - protocols: [http, https]
And there are special display-hints used to aid a client in rendering and validating the url. These are encrypted-protocols
and always-accept-certificates
which are documented in constraints.
When supplying the value for a url as an input, it can be supplied as a string or as a mapping with attributes.
As a string it would be my_site_address: https://mydomain.com/blog
.
As a mapping, it would look be:
inputs: my_url: url: https://domain.com/path # 'url' is a special shortcut - see note below private_key: | ----- BEGIN ... intermediate_key: | ----- BEGIN ... public_key: | ----- BEGIN ...
Note: A common use case is to supply the url and keys. A shortcut is available that accepts a key called url
that can be used to supply the url without having to provide all the components of the url.
Heat Blueprint Examples
This is an example of a Heat Blueprint using options
, blueprint meta-data and schema versioning.
blueprint: id: 0255a076c7cf4fd38c69b6727f0b37ea name: Managed Cloud WordPress w/ MySQL on VMs meta-data: schema-version: 0.7 services: lb: component: interface: http type: load-balancer constraints: - algorithm: ROUND_ROBIN relations: web: http master: http ... options: url: label: Site Address description: 'The domain you wish to host your blog on. (ex: example.com)' type: url required: true default: http://example.com/ display-hints: group: application order: 1 encrypted-protocols: [https] sample: http://example.com/ constraints: - protocols: [http, https] - regex: '^([a-zA-Z]{2,}?\:\/\/[a-zA-Z0-9\-]+(?:\.[a-zA-Z0-9\-]+)*\.[a-zA-Z]{2,6}(?:\/?|(?:\/[\w\-]+)*)(?:\/?|\/\w+\.[a-zA-Z]{2,4}(?:\?[\w]+\=[\w\-]+)?)?(?:\&[\w]+\=[\w\-]+)*)$' message: must be a valid web address constrains: - setting: allow_insecure service: lb resource_type: load-balancer value: true # turn on HTTP if protocol is HTTPS (provider handles it) resources: sync-keys: type: key-pair constrains: - setting: private_sync_key resource_type: application service: master attribute: private_key
Example of a Heat Blueprint for deploying an HA installation of WordPress
# A WordPress architecture template blueprint: &wp id: "6fcc7f31-08f8-4664-90e3-58fffc71f773" name: Multi-server Wordpress services: lb: component: *loadbalancer relations: web: http exposed: true open-ports: [80/tcp] web: component: *wordpress_reference_id # wordpress component above relations: {backend: mysql} backend: components: *mysql options: instance_count: type: number label: Number of Instances description: The number of instances for the specified task. default: 2 constrains: - {service: web, resource_type: compute, setting: count} constraints: - greater-than: 1 # this is an HA config
Deployments
A deployment defines and points to a running application and the infrastructure it is running on. It basically says "I took blueprint X and deployed it to environment Y using the following options". It combines a blueprint, an environment to deploy the resources to, and any additional inputs specific to this deployment.
General form:
- id - a User defined string id
- name: A short string identifying the Deployment
- blueprint: can be a reference (YAML), but for now, it's a full copy of a Heat Blueprint. We'll support references using URI later (git, local references, etc...).
- environment: same as blueprint - right now we make a full copy
- inputs: - these are for setting levers and dials... there are different scopes: global, blueprint, service, and provider scopes. So you can set the memory size at the provider level (i.e. all servers should be 1GB). Or, all servers in service X should be 1GB and all servers in service Y should be 2GB.
option_foo: bar blueprint: blueprint_input_foo: bar services: my_db_thang: service_input_foo: baz compute: (filter by resource type!) service_resource_input_foo: boo providers: 'nova': compute: os: ubuntu
Note: More specific inputs override more general ones (see get_setting code in deployment)
Action Items
- Determine the need for global inputs. Do we really need them? Or at least put them in a "globals" category
- Define how we reference blueprints and environments in deployments without including full copies. Reference UUID? URL to file in git or local reference? etc.
- Improve syntax for indicating a generated value, or remove it and let a generated value be the default. Or create system for code to be included in blueprint to be used to generate, validate values. See artifacts prototype in app.yaml.
Examples of Deployments:
deployment: blueprint: name: Simple Wordpress wordpress: config: *wordpress # points to wordpress component relations: {db: mysql} environment: name: rackcloudtech-test providers: - compute: vendor: rackspace constraints: - region: ORD - loadbalancer: &rax-lbaas - database: &rax-dbaas - common: vendor: rackspace credentials: - rackspace: ...
Example Deployment with provides
and requires
in components and providers as keys:
deployment: environment: providers: nova: provides: - compute vendor: rackspace
Example Deployment
# Actual running app and the parameters supplied when deploying it deployment: blueprint: *wp environment: *environment_1000_stag inputs: instance_count: 4 resources: '0': type: server provider: nova status: up flavor: 1 image: 119 instance: id: 2098383 private_ip: 10.10.1.1 public_ip: 2.2.2.18 dns-name: srv1.stabletransit.com relations: web-backend: state: up '1': type: server status: up provider: nova flavor: 1 image: 119 instance: id: 2098387 private_ip: 10.10.1.8 public_ip: 2.2.2.22 dns-name: srv2.stabletransit.com relations: web-backend: state: up '2': type: load-balancer dns-name: CMDEP32ea304-lb1.rackcloudtech.com instance: id: 8668444 relations: lb-web: state: up '3': type: database provider: databases dns-name: CMDEP32ea304-db1.rackcloudtech.com flavor: 1 disk: 2 instance: id: 99958744
API
See also: the Heat/Open_API to support this Open DSL.