Etherpad link for discussions : https://etherpad.openstack.org/p/convergence
The convergence blueprint has to be divided into many implement-able subsidiary blue-prints. The plan is to proceed in following order, in terms for blue-print implementations:
- Persist dependency graph and version the resources. This will remove the need for back-up stack; there would be only one stack in DB at any point of time. The dependency tasks are computed from database instead of from in-memory graph. This forms the basis for everything needed to be done to distribute resource tasks and make Heat engine resilient. The spec is available at https://review.openstack.org/#/c/123749. Described in detail at: https://wiki.openstack.org/w/images/f/fc/Persist-graph-stack-update2.pdf
- Concurrent stack update. As of now, the stack is locked for the entire duration of update. To allow concurrent updates, the lock duration should not last till update is complete. The stack should be locked only when the stack dependency graph is being updated/read. The graph create/update has to be decoupled from request processing. When a request arrives, the stack is locked, the dependency graph is updated and lock is released. The threads which pick-up the tasks, will lock the stack and take the next set of ready nodes from the graph. Concurrent stack update is handled by continuous forward merge of dependency graph, and saving the templates in the order in which the requests arrive.
- Decouple resource deletion tasks and move it to garbage collector. The deletion of stale resources (older versions) can be done by another thread after the stack has been updated. An update on stack updates the changed resources and only marks the removed resources as deleted. GC will actually delete the resources when it comes up. With this, the update cycles are made shorter to allow more realistic and quicker stack updates. For deleting a stack, the stack is marked as deleted and GC reclaims the resources in back-ground, in a distributed manner.
- Convergence observer: The above three tasks form the fundamental for convergence. With Observer, we create the DB entries for desired and observed properties and delegate the task of observing the resources to observer instead of doing it in the create/update worker threads. It enables worker to pick-up other tasks. When a resource is observed to be ready, it sends a message which is processed by heat engine and next set of convergence tasks are computed from DB.
- Convergence worker/Engine: Convergence jobs (resource create/update/delete etc.) for resources are produced on a queue, which are taken up by the worker/convergence engine. There will be a new topic on which Heat engines listen for convergence jobs and will distribute the job among its workers. If no worker is free, another Heat engine takes up the job.
- Continuous convergence observer: <TODO: More clarity is needed>
- Rollback Logic: File:RollbackLogic.pdf