Difference between revisions of "Heat/ConvergenceDesign"

Revision as of 06:57, 31 October 2014

Etherpad link for discussions : https://etherpad.openstack.org/p/convergence

Convergence Design

This page proposes the design for the convergence blueprints.
Note: Currently this page does not address design for 3rd blueprint "Convergence continuous observer"

The convergence blueprint has to be divided into many implement-able subsidiary blue-prints. The plan is to proceed in following order, in terms for blue-print implementations:

Persist dependency graph and version the resources. This will remove the need for back-up stack; there would be only one stack in DB at any point of time. The dependency tasks are computed from database instead of from in-memory graph. This forms the basis for everything needed to be done to distribute resource tasks and make Heat engine resilient. The spec is available at https://review.openstack.org/#/c/123749

Described in detail here: https://wiki.openstack.org/w/images/f/fc/Persist-graph-stack-update2.pdf

Concurrent stack update. As of now, the stack is locked for the entire duration of update. To allow concurrent updates, the lock duration should not last till update is complete. The stack should be locked only when the stack dependency graph is being updated/read. The graph create/update has to be decoupled from request processing. When a request arrives, the stack is locked, the dependency graph is updated and lock is released. The threads which pick-up the tasks, will lock the stack and take the next set of ready nodes from the graph. Concurrent stack update is handled by continuous forward merge of dependency graph, and saving the templates in the order in which the requests arrive.
Decouple resource deletion tasks and move it to garbage collector. The deletion of stale resources (older versions) can be done by another thread after the stack has been updated. An update on stack updates the changed resources and only marks the removed resources as deleted. GC will actually delete the resources when it comes up. With this, the update cycles are made shorter to allow more realistic and quicker stack updates. For deleting a stack, the stack is marked as deleted and GC reclaims the resources in back-ground, in a distributed manner.
Convergence observer: The above three tasks form the fundamental for convergence. With Observer, we create the DB entries for desired and observed properties and delegate the task of observing the resources to observer instead of doing it in the create/update worker threads. It enables worker to pick-up other tasks. When a resource is observed to be ready, it sends a message which is processed by heat engine and next set of convergence tasks are computed from DB.
Convergence worker/Engine: Convergence jobs (resource create/update/delete etc.) for resources are produced on a queue, which are taken up by the worker/convergence engine. There will be a new topic on which Heat engines listen for convergence jobs and will distribute the job among its workers. If no worker is free, another Heat engine takes up the job.
Continuous convergence observer: <TODO: More clarity is needed>

Flow Diagrams

Flowchart

Sequence

Data Model Change

Data model for persisting the graph

Add a new table 'resource_graph'

Column name	Type	Constraint
resource_name	varchar(255)	NOT NULL ForeignKey resource.name
needed_by	varchar(255)	NULL ForeignKey resource.name
stack_id	varchar(36)	NOT NULL ForeignKey stack.id
retry_count	Integer	Default (0)
observed_ready	Boolean	Default False

Data model for persisting the resources' observed properties

Add a new table 'resource_observed_properties'

Column name	Type	Constraint
resource_observed_id	varchar(36)	NOT NULL ForeignKey resource_observed.id
resource_name	varchar(255)	NULL
stack_id	varchar(36)	NOT NULL ForeignKey stack.id
prop_name	String
prop_value	String

@@ Line 7: / Line 7: @@
 The convergence blueprint has to be divided into many implement-able subsidiary blue-prints. The plan is to proceed in following order, in terms for blue-print implementations:
 # Persist dependency graph and version the resources. This will remove the need for back-up stack; there would be only one stack in DB at any point of time. The dependency tasks are computed from database instead of from in-memory graph. This forms the basis for everything needed to be done to distribute resource tasks and make Heat engine resilient. The spec is available at https://review.openstack.org/#/c/123749
+Described in detail here: https://wiki.openstack.org/w/images/f/fc/Persist-graph-stack-update2.pdf
 # Concurrent stack update. As of now, the stack is locked for the entire duration of update. To allow concurrent updates, the lock duration should not last till update is complete. The stack should be locked only when the stack dependency graph is being updated/read. The graph create/update has to be decoupled from request processing. When a request arrives, the stack is locked, the dependency graph is updated and lock is released. The threads which pick-up the tasks, will lock the stack and take the next set of ready nodes from the graph. Concurrent stack update is handled by continuous forward merge of dependency graph, and saving the templates in the order in which the requests arrive.
 # Decouple resource deletion tasks and move it to garbage collector. The deletion of stale resources (older versions) can be done by another thread after the stack has been updated. An update on stack updates the changed resources and only marks the removed resources as deleted. GC will actually delete the resources when it comes up. With this, the update cycles are made shorter to allow more realistic and quicker stack updates. For deleting a stack, the stack is marked as deleted and GC reclaims the resources in back-ground, in a distributed manner.