Jump to: navigation, search

Heat/DSL2

This is an idea which tries to evolve the first Heat/DSL proposal incorporating some related work/concepts from elsewhere, including TOSCA and CAMP. These specs are both evolving and there is scope for them to converge to something simple, intuitive, and powerful: this is an aim towards that convergence, and is believed to map nicely on to all three (Heat, TOSCA, and CAMP), and make it easy to represent other popular tools and techniques (e.g. Juju, Puppet, Chef, OpenShift).

This is WIP, supplied to guide discussion not stop it. Comments and further evolution is wanted. Also see Heat/Vocabulary table.

Overview

This variant distills the first proposal (Heat/DSL) down to two essential concepts:

  • Components are the pieces in the resulting model, either OpenStack primitives such as server or load balancer or subnet, or higher level concepts such as database or service/tier/autoscaling-group, or artifacts supplied by the user to be used in the application
  • Requirements tell the engine which components and environments are suitable, how they are configured, and how they are wired together (for example by looking at capabilities)


Then a blueprint is a set of components and requirements that define an application template. An environment is a place where a blueprint/components can be deployed, and a deployment is the living instantiation of a blueprint to an environment (aka a stack).

The heavy lifting is done by defined types of components and requirements. Some of these would be standardised within OpenStack Heat (e.g. server), whereas others could be based on standard models elsewhere (e.g. Juju, Chef, Puppet, Salt, Rerun, RPM's) or provided by vendors (e.g. PaaSes or cloud services) with the expectation that portable requirement types evolve in other communities (e.g. TOSCA, Java Community Process, PaaSes, etc).

A very rough-draft python interpreter for this is at [1]. There is no engine and it's not integrated to Heat but is a start, and could be evolved as this spec evolves, figuring out if/how to connect it to Heat and cherry pick code Rackspace are going to donate for Heat/DSL. Key differences to that initial proposal are: (1) removing service as top-level concept, as it can be modelled as a component type (see complex examples below); (2) removing providers, interfaces, and relation as top-level concepts, as these can be modelled by requirements. (I like the idea of including options and parameters from the first DSL, but I have not (yet) done that here.)

In this proposal (and the original proposal as I understand it) the abundant metadata (about looking up values, mapping, etc) is externalised and handled by requirements.

Simple Examples

A basic template of a server with an install script, using this proposed DSL:

name: "server running a custom install script" 
# blueprint defining components, using map/dictionary syntax
components:
  my_server:
    type: server
    run: my_script
    requires:
      minRam: 4gb
  my_script:
    type: com.example.script.ShellScript
    content: http://example.com/my_install.sh
    sudo: true

For readability, the 'id' is implicit for components, and the 'type' implicit for requirements. Here's the same thing in JSON (also valid YAML), using "list" syntax where those keys are explicit.

{
  "name": "server running a custom install script" 
  "components": [
    { "id": "my_server",
      "type": "server",
      "requires": [
        { "minRam": "4gb" },
        { "run": "my_script" } ] 
    }, {
      "id": "my_script",
      "type": "com.example.script.ShellScript",
      "content": "http://example.com/my_install.sh",
      "sudo": true
    } ]
}

And here's a template using higher level components which might be defined by a Heat extension (e.g. TOSCA, OpenShift, etc -- which in turn map on to OpenStack primitives like server). This mixes list and map syntax, and shows how global requirements might be used to enforce both jurisdiction and trusted components.

name: "war file deployed to appserver container (vm or cluster or paas)" 
# blueprint defining components, with no 'id' or 'type', and using list syntax
components:
- content: hello.war
  requires:
  - type: com.example.java:WarDeploymentRequirement
requires:
# force use of jetty server and openstack servers, in USA
  providers: { allow: [ com.example.*, org.jetty.*, org.openstack.* ] }
  jurisdiction: { iso3166: "us" }

Schema Definition

The basic shape of a blueprint is:

name: <optional string name>
components: <mandatory list of components or map of components by id>
requires: <optional list of global requirements or map of global requirements by type, or value of default requirement>

A component consists of the following attributes:

id: <optional string user-supplied identifier>
type: <optional type imposing requirements>
requires: <optional list of requirements or map of requirements by type>
content: <optional reference to a piece of content>
<other attributes as understood by the type>

The type of the component can be explicit or inferred based on requirements declared on it or pointing to it (using the 'fulfillment' attribute in the requirements, below). Abstract types and sub-types are envisaged, where e.g. server might be an abstract type understanding some requirements (e.g. minRam), with concrete sub-types such as AWS::EC2::Instance or OpenStack::Nova::Server.

And a requirement consists of the following attributes:

type: <mandatory string known requirement type>
fulfillment: <optional component(s) who fulfill this requirement>
<other attributes as understood by the requirement type>

There would be a library of requirements defined by Heat, some global such as providers and jurisdiction (in war example above), others specific to types of components and understood by those types, such as minRam and run on server (in the script example above). For readability, in cases where the requirement type defines a default attribute (such as minRam and run in the script example above), an abbreviated syntax may be used which simply supplies a value for that requirement attribute, if it is unambiguous; and similarly if the component type defines a default requirement type the type of the requirement could be omitted.

Fulfillment references to other components use the notation "id:other_id" (or a list thereof), where other_id is an identifier on a component, or simply "other_id" where that is unambiguous, or a Heat-specific URL to a pre-existing component, to indicate that the component(s) must supply a capability matching the requirement. Often it can be avoided, but it is one relatively simple way to deal with certain complex real-world situations, such as pointing two different webapps at the same database instance, or specifying additional requirements on a component used to fulfill one requirement. Some illustrations are included below.

Complex Examples

Here is a cluster running PHP installed by RPM (as an example poison -- Juju, Chef, Puppet, etc done similarly), a tarball unpacked, and a load balancer and DNS set up (please tweak as appropriate to map more nicely on to existing heat concepts).

name: "elastic php app"
# example using 'tier' type with 'nodespec' attribute to define the node,
# 3 nodes, with "loadbalancer" and "dns" reqs which quantum fulfils
components:
  my_app_tier:
    type: tier         # defined type in heat, understanding attributes and requirements below
    initialSize: 3
    nodespec:
      type: server    # (this might be a default for tier's 'nodespec')
      requires:       # set up each of the server instances
        os: rhel
        # these requirements point to "in-line" components
        run: { type: rpm, rpms: [ php ], repos: [ http://rpms.example.com/repo/ ] }
        unpack: { type: tgz, content: http://example.com/my_app.tgz, target: /var/html }
    requires:
      loadbalancer:         # known requirement type, on the _tier_ (treated as the pool)
        publicPort: 80
        membersPort: 80
        fulfillment: id:my_lb     # point to the LB, which has an addl requirement (dns)
  my_lb:        # type is implied by the 'loadbalancer' requirement above
    requires:
      dns:      # another defined requirement type
        hostname: myapp.example.com

This is an example taken from a CAMP proposal draft, using higher level components which could be supported by extensions to Heat backend (or implemented in TOSCA, etc).

name: "war file with database using CDI"
# complex 3-tier example using map syntax and 'fulfillment' to ensure
# WAR file's container(s) are injected with DB which has run SQL init
components:
  hello_war:      # no type 
    content: hello.war
    requires:
      com.example.java:WarDeploymentRequirement:
        fulfillment: frontend
  hello_sql: 
    content: hello.sql
    type: com.example.database:Schema   # here, type of component defined
    requires: backend        # assume Schema defines a default req type "DB"
  frontend:                # "platform component" implied by WarDeplReq above
    requires:
      database:            # frontend type must recognise a named "database" req
        mode: CDI          # assume that req supports various injection modes
        fulfillment: backend   # ensure this is the same DB which ran our hello.sql
      com.example.lb:LoadBalanced:     # longhand req form: this is the 'type'
        protocol: https                # assume that type recognises these attributes
        algorithm: round-robin
        sticky-sessions: true
        fulfillment: lb
  lb: { tags: [ "load-balancer" ] }    # tag it so we can monitor it afterwards

The API

The REST API should support POSTing this YAML, and should support POSTing a ZIP containing this YAML file in a well-known place (e.g. `heat.yaml`), where file references are resolved local to that ZIP -- so, for instance, artifacts such as scripts, WARs, and keys could be included. Also this would make it very easy to be compatible with TOSCA's "CSAR" format and CAMP's "PDP" format.

The REST API could be as described at Heat/Open_API, or it could be cut-down version of CAMP (which is very close to this, with a few more soft distinctions which might be useful at runtime -- application components (artifacts) vs platform components (things supplied by the platform, e.g. a server, or a database); and components (instances in a deployment) v templates (types available); etc).