Heat/DSL2

This is an idea which tries to evolve the first Heat/DSL proposal incorporating some related work/concepts from elsewhere, including TOSCA and CAMP. These specs are both evolving and there is scope for them to converge to something simple, intuitive, and powerful: this is an aim towards that convergence, and is believed to map nicely on to all three (Heat, TOSCA, and CAMP), and make it easy to represent other popular tools and techniques (e.g. Juju, Puppet, Chef, OpenShift).

''This is WIP, supplied to guide discussion not stop it. Comments and further evolution is wanted. Also see Heat/Vocabulary table.''

Overview
This variant distills the first proposal (Heat/DSL) down to two essential concepts:
 * are the pieces in the resulting model, either OpenStack primitives such as server or load balancer or subnet, or higher level concepts such as database or service/tier/autoscaling-group, or artifacts supplied by the user to be used in the application
 * tell the engine which components and environments are suitable, how they are configured, and how they are wired together (for example by looking at capabilities)

Then a  is a set of components and requirements that define an application template. An  is a place where a blueprint/components can be deployed, and a  is the living instantiation of a blueprint to an environment (aka a stack).

The heavy lifting is done by defined types of components and requirements. Some of these would be standardised within OpenStack Heat (e.g. server), whereas others could be based on standard models elsewhere (e.g. Juju, Chef, Puppet, Salt, Rerun, RPM's) or provided by vendors (e.g. PaaSes or cloud services) with the expectation that portable requirement types evolve in other communities (e.g. TOSCA, Java Community Process, PaaSes, etc).

A very rough-draft python interpreter for this is at. There is no engine and it's not integrated to Heat but is a start, and could be evolved as this spec evolves, figuring out if/how to connect it to Heat and cherry pick code Rackspace are going to donate for Heat/DSL. Key differences to that initial proposal are: (1) removing service as top-level concept, as it can be modelled as a component type (see complex examples below); (2) removing providers, interfaces, and relation as top-level concepts, as these can be modelled by requirements. (I like the idea of including options and parameters from the first DSL, but I have not (yet) done that here.)

In this proposal (and the original proposal as I understand it) the abundant metadata (about looking up values, mapping, etc) is externalised and handled by requirements.

Simple Examples
A basic template of a server with an install script, using this proposed DSL:

name: "server running a custom install script" components: my_server: type: server run: my_script requires: minRam: 4gb my_script: type: com.example.script.ShellScript content: http://example.com/my_install.sh    sudo: true
 * 1) blueprint defining components, using map/dictionary syntax

For readability, the 'id' is implicit for components, and the 'type' implicit for requirements. Here's the same thing in JSON (also valid YAML), using "list" syntax where those keys are explicit.

{  "name": "server running a custom install script" "components": [ { "id": "my_server", "type": "server", "requires": [ { "minRam": "4gb" }, { "run": "my_script" } ] }, {      "id": "my_script", "type": "com.example.script.ShellScript", "content": "http://example.com/my_install.sh", "sudo": true } ] }

And here's a template using higher level components which might be defined by a Heat extension (e.g. TOSCA, OpenShift, etc -- which in turn map on to OpenStack primitives like server). This mixes list and map syntax, and shows how global requirements might be used to enforce both jurisdiction and trusted components.

name: "war file deployed to appserver container (vm or cluster or paas)" components: - content: hello.war requires: - type: com.example.java:WarDeploymentRequirement requires: providers: { allow: [ com.example.*, org.jetty.*, org.openstack.* ] } jurisdiction: { iso3166: "us" }
 * 1) blueprint defining components, with no 'id' or 'type', and using list syntax
 * 1) force use of jetty server and openstack servers, in USA

Schema Definition
The basic shape of a  is:

name: components:  requires: 

A  consists of the following attributes:

id:  type:  requires:  content:  

The type of the component can be explicit or inferred based on requirements declared on it or pointing to it (using the 'fulfillment' attribute in the requirements, below). Abstract types and sub-types are envisaged, where e.g. server might be an abstract type understanding some requirements (e.g. minRam), with concrete sub-types such as AWS::EC2::Instance or OpenStack::Nova::Server.

And a  consists of the following attributes:

type:  fulfillment:  

There would be a library of requirements defined by Heat, some global such as providers and jurisdiction (in war example above), others specific to types of components and understood by those types, such as minRam and run on server (in the script example above). For readability, in cases where the requirement type defines a default attribute (such as minRam and run in the script example above), an abbreviated syntax may be used which simply supplies a value for that requirement attribute, if it is unambiguous; and similarly if the component type defines a default requirement type the type of the requirement could be omitted.

Fulfillment references to other components use the notation "id:other_id" (or a list thereof), where other_id is an identifier on a component, or simply "other_id" where that is unambiguous, or a Heat-specific URL to a pre-existing component, to indicate that the component(s) must supply a capability matching the requirement. Often it can be avoided, but it is one relatively simple way to deal with certain complex real-world situations, such as pointing two different webapps at the same database instance, or specifying additional requirements on a component used to fulfill one requirement. Some illustrations are included below.

Complex Examples
Here is a cluster running PHP installed by RPM (as an example poison -- Juju, Chef, Puppet, etc done similarly), a tarball unpacked, and a load balancer and DNS set up (please tweak as appropriate to map more nicely on to existing heat concepts).

name: "elastic php app" components: my_app_tier: type: tier        # defined type in heat, understanding attributes and requirements below initialSize: 3 nodespec: type: server   # (this might be a default for tier's 'nodespec') requires:      # set up each of the server instances os: rhel # these requirements point to "in-line" components run: { type: rpm, rpms: [ php ], repos: [ http://rpms.example.com/repo/ ] } unpack: { type: tgz, content: http://example.com/my_app.tgz, target: /var/html } requires: loadbalancer:        # known requirement type, on the _tier_ (treated as the pool) publicPort: 80 membersPort: 80 fulfillment: id:my_lb    # point to the LB, which has an addl requirement (dns) my_lb:       # type is implied by the 'loadbalancer' requirement above requires: dns:     # another defined requirement type hostname: myapp.example.com
 * 1) example using 'tier' type with 'nodespec' attribute to define the node,
 * 2) 3 nodes, with "loadbalancer" and "dns" reqs which quantum fulfils

This is an example taken from a CAMP proposal draft, using higher level components which could be supported by extensions to Heat backend (or implemented in TOSCA, etc).

name: "war file with database using CDI" components: hello_war:     # no type content: hello.war requires: com.example.java:WarDeploymentRequirement: fulfillment: frontend hello_sql: content: hello.sql type: com.example.database:Schema  # here, type of component defined requires: backend       # assume Schema defines a default req type "DB" frontend:               # "platform component" implied by WarDeplReq above requires: database:           # frontend type must recognise a named "database" req mode: CDI         # assume that req supports various injection modes fulfillment: backend  # ensure this is the same DB which ran our hello.sql com.example.lb:LoadBalanced:    # longhand req form: this is the 'type' protocol: https               # assume that type recognises these attributes algorithm: round-robin sticky-sessions: true fulfillment: lb  lb: { tags: [ "load-balancer" ] }    # tag it so we can monitor it afterwards
 * 1) complex 3-tier example using map syntax and 'fulfillment' to ensure
 * 2) WAR file's container(s) are injected with DB which has run SQL init

The API
The REST API should support POSTing this YAML, and should support POSTing a ZIP containing this YAML file in a well-known place (e.g. `heat.yaml`), where file references are resolved local to that ZIP -- so, for instance, artifacts such as scripts, WARs, and keys could be included. Also this would make it very easy to be compatible with TOSCA's "CSAR" format and CAMP's "PDP" format.

The REST API could be as described at Heat/Open_API, or it could be cut-down version of CAMP (which is very close to this, with a few more soft distinctions which might be useful at runtime -- application components (artifacts) vs platform components (things supplied by the platform, e.g. a server, or a database); and components (instances in a deployment) v templates (types available); etc).