Heat/Blueprints/hot-software-config

= HOT Software Config = --stevebaker (talk) 21:10, 15 October 2013 (UTC)

NOTE: This specification is still under discussion and has not been endorsed by the heat project.

Background
The blueprint software-configuration-provider and its specification are the most complete attempt so far to specify the HOT solution to Heat software config.

The blueprints hot-software-config and hot-software-config-deps appear to be closely related to software-configuration-provider

This specification is attempting to solve the same problems as but takes an alternative approach in some areas. It is hoped that a consensus is reached among heat-core developers so that implementation can begin.

The blueprint native-tools-bootstrap-config and specification is related to this specification, and will evolve together as changes are made.

Also see Angus's fantastic (giggle) approach at this 

Components vs Resources: Representing software config
A heat resource represents the configuration and life-cycle of a cloud resource which is managed via its API. A small number of heat resources do not map to a cloud resource, and instead represent internal heat state. This has often been the cause of issues and serves as a warning to using the resource abstraction for purposes it was not designed for. Other orchestration/configuration management tools have chosen single abstractions to represent both orchestration and configuration management. Depending on the abstraction chosen, generally one or other of these areas end up with a poor representation in these tools.

Component definition
Software configuration for Heat involves defining units of work (components) that need to be executed on compute resources whenever that resource is created or updated.

Each component needs to be able to specify the following:
 * The type of the configuration component
 * Configuration data which represents the unit of work the component will perform

A HOT template will have a new top-level section called. Just like,   will contain named blocks where each block represents a single component that can be referenced elsewhere by its name. Each component will also have the following attributes:
 * specifies which component implementation to use, which indicates the which configuration management tool syntax will be used in the  section.
 * data which is passed to the configuration management tool to perform the component's desired configuration. There is no processing of HOT intrinsic functions in  blocks; they are intended to be self-contained units of configuration. Ideally it should be possible to invoke the content of   sections directly with the configuration management tool without any modification or involvement from Heat.  Depending on the format required by the configuration management tool, the   attribute could either contain a string containing a script, or a YAML structure.
 * as an alternative to  specifies a URL or relative file path to load configuration from. Paths that resolve to local files will need to be resolved by the heat client.

An example of a  section: components: install_mysql: type: Heat::SoftwareConfig config: packages: yum: mysql: [] mysql-server: [] services: systemd: mysqld: {enabled: 'true', ensureRunning: 'true'} install_wordpress: type: Heat::SoftwareConfig config: packages: yum: httpd: [] wordpress: [] services: systemd: httpd: {enabled: 'true', ensureRunning: 'true'} foo_file: type: Heat::SoftwareConfig config: files: /tmp/foo/bar: content: 'foo_bar_contents'

Component invocation
Components can only be invoked on compute resources. The properties for AWS::EC2::Instance must remain compatible with cfn, however OS::Nova::Server has no such limitation. OS::Nova::Server will gain a  property which specifies a list of components to apply to the server in the desired order. Each entry in the  list will contain the following properties:
 * the name of the component to invoke, as specified in the  section
 * values from stack resources, parameters, attributes or static values get mapped to configuration management variables in each component. How these params get treated will depend on the component type, and the configuration management tool used by that type.

An example of a OS::Nova::Server with a  property: resources: the_server: type: OS::Nova::Server properties: components: - name: install_things - name: write_foo_bar params: foo_bar_contents: get_param: the_file_contents - name: repo_epel_5_testing - name: some_user_data

Why not relationships hosted_on, depends_on, connects_to?
A previous proposal for HOT components specified that each component definition can specify a  attribute which allows the following relations to be defined:
 * the defining component is hosted on the referenced component or resource
 * the defining component has a general dependency on the referenced component or resource
 * the defining component needs to connect to the referenced component or resource. In contrast to depends_on, a connects_to relation generally involves execution of some code to establish the connection

This proposal deliberately does not have a  attribute at all. Each relationship type is not need for the following reasons:
 * is represented in the  property of the OS::Nova::Server.   reverses the convention that Heat already has in establishing a relationship between two entities. If the component/resource relationship is defined by the resource then a layering can be established whereby the component layer never makes reference to the stack/resource layer.
 * is represented by the order of the components specified in the  property of the OS::Nova::Server.  Practically speaking, one component does not "depend on" another at all. Instead one component might have some precondition which could be satisfied by another specific component, or by a different component which performs that precondition in a different way, or that precondition might already be satisfied by the server image. Modelling the satisfaction of these preconditions is a core purpose of many configuration management tools. Doing this in HOT components would be reinventing something which is more completely implemented by the very tools that some components will be using.  Therefore this proposal takes the position that component dependencies should not be modeled at all, and template authors should ensure themselves that preconditions of a component are satisfied either through specifying component order, or by building custom images.
 * the proposal of this relationship type does not yet have enough detail to evaluate it, but if it is necessary there should be a way of modelling it in another way.

Composability
The stated intent of HOT components is to separate application configuration from stack architecture, and to allow application configuration components to be reused in different templates.

Composability is maximized by using the config-file attribute so that configuration scripts exist as their own re-usable files. Consider the following pseudo example: components: install_db: type: Heat::Puppet config-file: install_db.pp  configure_db_for_app: type: Heat::Puppet config-file: configure_db_for_app.pp  configure_app: type: Heat::Puppet config-file: configure_app.pp resources: db_and_app: type: OS::Nova::Server properties: components: - name: install_db - name: configure_db_for_app - name: configure_app params: db_server: localhost

A template which specifies a simple 2-tier architecture could specify an identical components section and specify the following for the resources: resources: db_server: type: OS::Nova::Server properties: components: - name: install_db - name: configure_db_for_app app_server: type: OS::Nova::Server properties: components: - name: configure_app params: db_server: get_attr: [db_server, first_address]

Another template could specify load balanced HA of the app_server while still using the same configure_app.pp component configuration.

Concurrency and dependencies
Consider the following resources: resources: db_server: type: OS::Nova::Server app_server: type: OS::Nova::Server properties: components: - name: do_first_thing - name: configure_app params: db_server: get_attr: [db_server, first_address] - name: do_last_thing

Heat's current behavior would be to not start creating the resource  until   creation is completed. This dependency is determined by the call to  in the   component.

However there is potential for increased concurrency in this scenario. could be created at the same time as, and the   component can perform its configuration. Further configuration progress can then block until  is created, then   and   can perform their configuration.

Consider the following alternative to : app_server: type: OS::Nova::Server properties: component_execution: async components: - name: do_first_thing - name: configure_app params: db_server: get_attr: [db_server, first_address] - name: do_last_thing

Specifying  would do the following:
 * Prevent resource dependencies from being established for any get_attr or get_resource calls inside the  property (in this case,   will no longer depend on  )
 * Prevent metadata for  and   from being built at all until all values are able to be resolved
 * Rely on os-collect-config polling to trigger  and   when   creation is complete.

Wait conditions
Currently in Heat, wait conditions are the accepted way to communicate data or events from servers back to the orchestration engine. Currently writing templates with wait conditions can be cumbersome since it requires:
 * a wait condition resource
 * a wait handle resource
 * server configuration to invoke cfn-signal (or curl) with the wait handle URL
 * a  call on the wait condition to establish a resource dependency on the wait condition result.

A completely different mechanism may be implemented in the future, but in the meantime this blueprint should enable a more succinct representation of waitconditions to simplify their use.

The actual approach will be specified by blueprint hot-software-config-deps so this will not be explored in detail here, but one possible solution is presented just to confirm that components have the potential of allowing this simplification: components: install_db: type: Heat::Puppet config-file: install_db.pp  configure_db_for_app: type: Heat::Puppet config-file: configure_db_for_app.pp  return_db_url: type: Heat::WaitCondition config: | #!/bin/sh # shell script which builds and returns a db URL configure_app: type: Heat::Puppet config-file: configure_app.pp resources: db_server: type: OS::Nova::Server properties: components: - name: install_db - name: configure_db_for_app - name: return_db_url params: wait_condition_resource: db_server_url app_server: type: OS::Nova::Server properties: components: - name: configure_app params: db_server: get_attr [db_server_url, first_data]

Here, the  component implements whatever is required to call cfn-signal with some arbitrary data, so that the only job of the config script is to build the data to be signaled.

Specifying  will implicitly create the wait handle resource and the wait condition resource that can be referenced by other resources.