Jump to: navigation, search

Heat/DSL

< Heat
Revision as of 22:56, 10 April 2013 by Adrian Otto (talk | contribs) (Environments)

DRAFT ONLY - WORK IN PROGRESS

Please recognize that this is only a proposed DSL to be considered for development.

Overview

This is a DSL to be used with HEAT to allow a simple REST API to allow a more rich set of capabilities to be exposed. It is designed to be declarative in nature, meaning that it will identify what to deploy and orchestrate rather than explicitly how to deploy or orchestrate it. The syntax is intended to be simple, and human readable.

Roles

  • Architect - The person responsible for creating Heat Blueprints. He/she encodes the logic and constraints of a system or application based on their own deep knowledge and expertise with the Components of that system. For example a WordPress expert would act as the Architect to create the "Best Practice WordPress by Rackspace" Heat Blueprint.
  • Tenant - A single customer of a Cloud Service Provider
  • User - Deployment End-User - The person using Heat to initiate orchestrations
  • Cloud Service Provider - A service entity offering hosted cloud services on OpenStack or another cloud technology. Also known as a Vendor.

DSL Design Goals

  • Simplicity for the end user should be valued first ahead of all other design goals.
  • Human Readability. Artifacts presented to the orchestration API should be human readable for simplified troubleshooting and development. See Simplicity.
  • Flexibility should allow the DSL to be used for a reasonable variety of use cases, but should not aim to support *all* possible use cases.

Terminology

DSL

DSL = Domain Specific Language. This is the syntax used to express a configuration or instructions to exercise the features of the system through it's API, without while keeping the API itself minimalistic.

Blueprint

A Heat Blueprint (not to be confused with Launchpad Blueprint) is an orchestration document that details everything that is needed to carry out an orchestration. Think of this as the functional equivalent of an AWS CloudFormation Template. It is expressed as YAML. Note: JSON is a subset of YAML. Any YAML parser will also accept JSON. YAML offers easier readability for humans. Heat Blueprints may be shared in common among multiple Tenants of different Cloud Service Providers. They can contain all of the vendor independent specifications for launching a particular service or application. For example, Rackspace may publish a "Best Practice WordPress by Rackspace" Heat Blueprint that can be used by any Tenant of any Cloud Service Provider. The Heat Blueprint is created by an Architect

Environment

A logical target for a Deployment. It is unique to a given user, but independent from Region, Cell, Cloud, etc. A user may have one Environment for production, another for Staging, and perhaps another for dev/test. An arbitrary number of environments may be specified. A given Environment may mix Components from different Cloud Service Providers, perhaps Compute services come from the local OpenStack cloud, but backups happen on the Rackspace Cloud.

Deployment

A Deployment is comprised of a Heat Blueprint and an Environment. A Deployment may be bundled in an app.yaml file that contains the Deployment, Heat Blueprint, and Environment all together for convenience. Typically a user would refer to existing Environments and Heat Blueprints that they reference.

Component

A Component is an abstract representation for capabilities or services offered by a Cloud Service Provider. The ids for Components are universal, and will be recorded in a central registry.

DSL Resources

Provider

A Provider is a plug-in for Heat that understands how to operate a service or system. For Example, the Rackspace Compute Provider knows how to interact with the Rackspace Open Cloud to create compute resources. The OpsCode Chef Provider knows how to operate Chef and apply Cookbooks. Note: Provider != Cloud Service Provider.

  • name - A short descriptive string that identifies the component for humans. Example: mssql (not Microsoft SQL Server)
  • role - Allows the
  • version - A string representing the version of this component. Example: 1.0.0
  • description - A short description of the component. Example: Microsoft SQL Server
  • is - Describes the type of resource based on a closed list of OpenStack primitives (compute, database, load-balancer, application, ...)
  • requires What this resource needs in order to function. The need can be specific or general. General would be anything with a mysql interface (for WordPress). Specific would be "I need a host compute resource that is an Ubuntu 12.04 or later machine".
Two options for "requires" Syntax (short form, and long form):
Short Form
Syntax: type: interface
Example:
database: mysql
Long Form
Syntax: requirement_key: value
Example for "I need a host compute resource that is an Ubuntu 12.04 or later machine":
server: compute
interface: debian
relation: host
Note that requirement_keys can be arbitrary to label the requirement. Example requirements from above:
server: (this is the arbitrary key - the label of this requirement) of type type: compute
interface: debian (this says I expect to be able to do apt-get, vs yum)
relation: host (this says I am hosted on this resource. I cannot exist without it. I am down when it is down). "host" is a keyword.
constraint: - 'os': ['debian', 'redhat'] The name+key/value syntax allows for requiring more than one of the same resource (ex. log and data mysql databases) as well as adding additional constraints, etc....
  • provides - An array of resource_type:interface entries
Note: The array can be represented in YAML with entries preceded with dashes.
For example, let's say we get a Hosting API that provides a "website" resource. Website could provide:
database:mysql
application: php
  • options - For listing the optional settings I can set on this component. Example:
username: 
    default: root 
    required: optional | required | auto-generated (default: optional) 
    type: string (currently we support string, int, boolean) (default: string) 

Action Items:

  • How can we tie this to a "list" from the provider label: User Name (used for display friendliness) description: ... source_field_name: the name as it is known by the underlying provider. Ex. wordpress/database/db_user for OpsCode Chef. We need this currently since we dynamically generate component definitions and we need to be able to map things back after they've flowed through Heat.
  • Should be able to get rid of this in the future with clean, reversible mapping logic in providers. type: string regex: for validation sample: for display, to show what this looks like. help: help text. password:

Providers API

GET /providers
GET /tenant_id/environments/environment_id/providers/provider_id/catalog/

Returns a list of components, grouped by resource type

Environments

Environments provide a 'context' and optional constraints as well as a list of providers. For example, if I have an environment with a Cloud Databases provider 'constrained' to region LON where Cloud Databases don't have 32GB instances, then the catalog won't have 32GB instances.

  • id - A unique identifier provided by client (or user).
  • name - String for convenience to identify the environment
  • providers - Predefined keys are names (capabilities) available from the providers. The provider is identified in Heat based on the key and vendor field.
nova: 
    vendor: rackspace

The above results in Heat dynamically loading heat.providers.rackspace.nova (last two are vendor.key)

chef-solo: 
    vendor: opscode 
    database: 
        vendor: rackspace 
        constraints: region=DFW
        catalog: ...

About constraints: optional constraints may be applied at the provider level such as the example below to restrict the region to "DFW". The syntax follows normal constraint syntax, but can be shortened to key:value shorthand. For the normal syntax, a 'value' defines what the constraint evaluates to.

About catalog: - a way to inject a catalog into the provider (two uses for this are #1 testing, #2 only want to show 1GB instances). So if a catalog is provided, the provider will use that. Otherwise, it could log on and query the underlying service (list images, list flavors, get cookbooks, etc...). You'll see this in app.yaml.

The Environment will define what this looks like based on environment providers and constraints. A blueprint may be incompatible with an environment if the environment cannot provide the required resources OR meet the stated or inherited constraints.

Heat Blueprints

An application is a collection of components that provide a useful capability. An application is distinct from infrastructure. A Heat Blueprint describes a design that provides an application with certain 'ilities (scalability, availability, etc...). It defines the components, their relationships to each other, and the various constraints on the components and relationships that must be met in order for the application to work as specified.

When expressing services, Heat Blueprints contain:

  • One or more services (all under one 'services' entry)
  • An arbitrary name for service ID (each one has its own ID)
  • One component per service with the expectation that we'll be able to pull in Heat Blueprints as Components in the future.
  • Relations between services

Attributes:

  • id - unique identifier provided by the user/client
  • name - a short string naming the service
  • description: a more detailed description of the service
  • services - like tiers, but not restricted to the concept of tiers. Currently, there is one component defined per service.

Example Heat Blueprint:

id: wordpress
name: my_wp_blog
description: My WordPress Blog

The id or key/value pairs are used find a suitable component. The example above would match against a component like this:

component: wordpress
...

When using a key/value pair:

component: 
    type: application 
    interface: http 
    constraints: 
    - count: 2

Note: We have not yet defined any schema to be able to say something like "greater than one"

  • relations: Specify connections to other services. Examples: - wordpress app to mysql database - load balancer to webhead Two syntaxes for relations; short and long:

Short syntax: service_name:interface. Example:

my_db_thang: mysql

Long syntax: arbitrary name, key values. Example:

db: 
    interface: mysql 
    service: my_db_thang 
  • options - This is critical part of a Heat Blueprint that identifies the parameters that can be supplied upon instantiation. It follows syntax like component options, except for constraints. See the Heat Blueprint Options section for more information.
  • database_bigness - Option for the size of the database default: 2GB

Action Item: Need to define syntax/schema for expressing more complex logic, like greater than, less than, etc....

  • constrains - Specifies what values within components that this option constrains. Key/value to select the options, and then the option name.

Example:

service: my_database_thang 
    setting: memory
service: my_database_log_thang 
    setting: memory choice: 
        - value: 1024 
          name: 1gig 
        - value: 2048 
          name: 2GB 
  • resources - static resources to be created at planning time and shared across the blueprint. For example, users and keys:
my_key: 
    type: key-pair 

Note: private/public key pair will be created before deploying the workflow constrains:

service: my_database_thang 
    setting: key 
    attribute: private_key

The above example will take the private_key value from the generated keys and apply it as the value for 'key' in the my_database_thang component.

Heat Blueprint Options

Fields
  • label - the short label to use when displaying the option to the user. description: A full description of this option (what it is)
  • help - Detailed help on this option (how to use it)
  • sample - An example of what the data will look like. This could be shown as the background of a text control.
  • display-hints - a mapping (key, value pairs) of hints for how to order and display this option in relation to other options ....
  • group - a group name used to group options together. Examples to hint Horizon:
    • deployment: this is a deployment option and shows right under the deployment name
    • application: this is an application option and shows on the first screen of options
    • compute: this is an option that should be shown under the compute server options section
    • load-balancer: this is an option that should be shown under the load balancer options section
    • database: this is an option that should be shown under the database options section
    • dns: this is an option that should be shown under the dns options section
  • order - Indicate the relative order of this option within its group (as an integer)
  • list-type - What is the type of the entries in the list. Used to identify if it should be a specific resource type and list or attribute. The format is resource-type.list where resource-type is a known Checkmate resource type (compute, database, etc...) and the list is one of the lists from the provider [TODO: define these lists more precisely in the DSL or schema. Right now, they exist only in the provider catalogs]. Examples (and what we will initially support):
    • compute.memory - list of available compute image sizes
    • _compute.os - list of available compute operating systems
    • load-balancer.algorithm - list of available load balancer algorithms
      • encrypted-protocols - The subset of protocols that are encrypted so that we know when to show ssl cert controls. Ex. [https, pop3s]
      • always-accept-certificates - if a blueprint always accepts and handles the certificates (especially if the url is entered in free-form supporting any protocol)
  • default - The default value to use for the option.


NOTE: YAML will assume numbers are ints, so enclose strings in "quotation marks" as a best practice. Special values for this are =generate_password() which will generate a random password on the server. TODO: Consider adding parameters to generate_password() so the heat Blueprint author can make the password generated match their (or their application's) requirements. This would be used by the blueprint author in tandem with constraints (below) for validation on the client side.

  • type - the data type of this option. Valid types are: string, integer, boolean, password, url, region, and text (multi-line string). See later for a description of the url type which has some special attributes.
  • choice - a list of items to select from (used to display a drop-down). The entries are either plain strings or a mapping with value and name entries where value is what is passed to Checkmate and name is what is displayed to the user. Note: does not apply validation. If you want validation, use in a constraint. This is used for display only. Example:
choice:
- name: Ubuntu 12.04
  value: q340958723409587230459872345
- name: Ubuntu 12.10
  value: 2384729387w0tw9879t87ywt3y42
  • constrains - A list of mappings used as a way to set or limit aspects of the blueprint with the value (or parts of) the option.
  • required - Boolean. Set to true if this option must be supplied by the user.
  • constraints - An array of mappings (key/value pairs) in constraints syntax. Supported constraints are:
    • greater-than
    • less-than
    • greater-than-or-equal-to
    • less-than-or-equal-to
    • min-length - for strings
    • max-length - for strings
    • allowed-chars - Example: "ABCDEFGabcdefg01234565789!&@" - TODO: see if regex can be used here
    • required-chars - Example: "ABCDEFG" - TODO: see if regex can be used here
    • in - a list of acceptable values (these could also be used by clients to display drop-downs)
    • protocols - Used for URL types. This lists allowed protocols in the URL.
    • regex - Do not use look-forward/behind. Keep these simple so they are supported in javascript (client) and python (server). While many of the above can also be written as regex rules, both are available to Heat Blueprint authors to use the one that suits them best.
  • constraints: message - You can add a message key/value pair to any constraint. We recommend adding a message to regex constraints so it is easy to understand what they do when reviewed and so clients/servers can generate useful error messages and people reading the blueprint don't have to decipher the regexs. Ex. "must have 8-16 characters"

Browser clients may parse constraints and apply validation rules. A good practice is to have multiple simple regex constraints to allow browser clients to provide clear and useful feedback to the user for each rule they may break. For example, list lowercase, uppercase, and numeric requirements for a password as three constraints with a message unique to each.

Example Heat Blueprint with Options:

blueprint:
  options:
    database_name:
      label:  Database Name
      sample: db1
      display-hints:
        order: 1
        group: database
     default: wp_db
     description: "This is the name of the database that will be created to host your application's data"
     type: string
     constraints:
     - regex: ^(?=.*).{2,15}$
       message: must be between 2 and 15 characters long
     - regex: ^[A-Za-z0-9]*$
       message: can only contain alphanumeric characters
    database_password:
      label:  Database Password
      display-hints:
        order: 2
        group: database
     description: "This is the password to use to access the database that will host your application's data"
     type: password
     constraints:
     - regex: ^(?=.*).{8,15}$
       message: must be between 8 and 15 characters long
     - regex: ^(?=.*\d)
       message: must contain a digit
     - regex: ^(?=.*[a-z])
       message: must contain a lower case letter
     - regex: ^(?=.*[A-Z])
       message: must contain an upper case letter
Common Types
The URL Type

Options of type url provide some advanced handling of common url use cases. The option can be used simply as a string that accepts a url. In this case, the only benefit of setting the type to url is that a client application can perform certain validation to make sure the provided value is a valid URL (according to RFC 3986).

Example:

option:
  my_web_site:
    type: url

It is useful, however, to be able to handle different parts of a URL (i.e the scheme or protocol, domain, path, port, username, password, etc...) separately. They may be validated independently (e.g. make sure the protocol is http or https only). The parts may be wired up to different parts of the blueprint using constraints (e.g. use the domain part for a dns setting). The way that is supported is that the url type has attributes that can be accessed in the blueprint or other parts of Checkmate. These attributes are:

* scheme - this is the first part of the URL
* protocol - this is the first part of the URL as well (an alias to scheme)
* netloc - the dns name or address part
* port - this is the port if specified (e.g. the port in http://localhost:8080 is 8080)
* path - the path of the resource or file
* private_key - the private_key of a certificate to use if the protocol is an encrypted one
* public_key - the public_key of a certificate to use if the protocol is an encrypted one
* intermediate_key - the intermediate key chain of a certificate to use if the protocol is an encrypted one

These attributes can be specified in constraints:

options:
  my_url:
    label: Site Address
    type: url
    constraints:
    - protocols: [http, https]
    constrains:
    - type: load-balancer
      service: lb
      attribute: protocol  # This picks out the 'http' or 'https' part of the URL
      setting: protocol
    - type: compute
      service: web
      attribute: "private_key"  # This picks out the cert
      setting: ssl_certificate
    - type: compute
      service: web
      attribute: "intermediate_key"  # This picks up the intermediate cert
      setting: ssl_intermediate_certificate

You can constrain a list of protocols using the protocols constraint.

options:
 my_url:
    label: Site Address
    type: url
    constraints:
    - protocols: [http, https]

And there are special display-hints used to aid a client in rendering and validating the url. These are encrypted-protocols and always-accept-certificates which are documented in constraints.

When supplying the value for a url as an input, it can be supplied as a string or as a mapping with attributes.

As a string it would be my_site_address: https://mydomain.com/blog.

As a mapping, it would look be:

inputs:
  my_url:
    url: https://domain.com/path  # 'url' is a special shortcut - see note below
    private_key: |
      -----  BEGIN ...
    intermediate_key: |
      -----  BEGIN ...
    public_key: |
      -----  BEGIN ...

Note: A common use case is to supply the url and keys. A shortcut is available that accepts a key called url that can be used to supply the url without having to provide all the components of the url.

Heat Blueprint Example

This is an example of a Heat Blueprint using options, blueprint meta-data and schema versioning.

blueprint:
  id: 0255a076c7cf4fd38c69b6727f0b37ea
  name: Managed Cloud WordPress w/ MySQL on VMs
  meta-data:
    schema-version: 0.7
  services:
    lb:
      component:
        interface: http
        type: load-balancer
        constraints:
        - algorithm: ROUND_ROBIN
      relations:
        web: http
        master: http
    ...
  options:
    url:
      label: Site Address
      description: 'The domain you wish to host your blog on. (ex: example.com)'
      type: url
      required: true
      default: http://example.com/
      display-hints:
        group: application
        order: 1
        encrypted-protocols: [https]
        sample: http://example.com/
      constraints:
      - protocols: [http, https]
      - regex: '^([a-zA-Z]{2,}?\:\/\/[a-zA-Z0-9\-]+(?:\.[a-zA-Z0-9\-]+)*\.[a-zA-Z]{2,6}(?:\/?|(?:\/[\w\-]+)*)(?:\/?|\/\w+\.[a-zA-Z]{2,4}(?:\?[\w]+\=[\w\-]+)?)?(?:\&[\w]+\=[\w\-]+)*)$'
        message: must be a valid web address
      constrains:
      - setting: allow_insecure
        service: lb
        resource_type: load-balancer
        value: true  # turn on HTTP if protocol is HTTPS (provider handles it)
  resources:
    sync-keys:
      type: key-pair
      constrains:
      - setting: private_sync_key
        resource_type: application
        service: master
        attribute: private_key

Deployments

id: ... name: ... blueprint: can be a reference (YAML), but for now, it's a full copy of a Heat Blueprint. We'll support references using URI later (git, local references, etc...). environment: same as blueprint - right now we make a full copy inputs: - these are for setting levers and dials... there are different scopes: global, blueprint, service, and provider scopes. So you can set the memory size at the provider level (i.e. all servers should be 1GB). Or, all servers in service X should be 1GB and all servers in service Y should be 2GB.

option_foo: bar 
blueprint: 
    blueprint_input_foo: bar 
services: 
    my_db_thang: 
        service_input_foo: baz 
        compute: (filter by resource type!)
        service_resource_input_foo: boo 
   providers: 
       'nova': 
           compute: 
               os: ubuntu 

Note: more specific inputs override more general ones (see get_setting code in deployment)

Action Items

  • Determine the need for global inputs. Do we really need them? Or at least put them in a "globals" category
  • Define how we reference blueprints and environments in deployments without including full copies. Reference UUID? URL to file in git or local reference? etc.
  • Improve syntax for indicating a generated value, or remove it and let a generated value be the default. Or create system for code to be included in blueprint to be used to generate, validate values. See artifacts prototype in app.yaml.

Examples of Deployments:

 deployment:
   blueprint:
       name: Simple Wordpress
       wordpress:
         config: *wordpress  # points to wordpress component
         relations: {db: mysql}
   environment:
         name: rackcloudtech-test
         providers:
         - compute:
           vendor: rackspace
           constraints:
           - region: ORD
         - loadbalancer: &rax-lbaas
         - database: &rax-dbaas
         - common:
           vendor: rackspace
           credentials:
           - rackspace:
             ...

Example Deployment with provides and requires in components and providers as keys:

 deployment:
   environment:
     providers:
       nova:
         provides:
         - compute
         vendor: rackspace