StructuredStateManagement
Contents
Summary
Move away from ad-hoc states and state transitions to a more concrete organized structured state management in nova.
What problems does this solve
- Increases the [stability, extendability, reliability] of nova.
- Makes it easier to [debug, test, understand, verify, review] nova code.
- Removes hard to discover state-transition dependencies and interactions with clearly defined state-transition dependencies and interactions.
- Ensures state transitions are done reliably and correctly by isolating those transitions to a single place.
- Removes the need for periodic tasks to cleanup garbage left by nova's ad-hoc states.
- Fixes a variety of problems that previously had piecemeal like patches applied.
- Eliminates the inherent fragility of a ad-hoc workflow.
- Creates the path for smart resource scheduling.
- Makes it possible to do [resizing, live migration] in a more secure and manageable manner.
- Makes it possible to audit & track the state transitions performed on a given resource.
- Moves toward the path of multi-stage booting where an instances dependent resources are first reserved, the resources configured, the instance configured, and then finally the instance is powered-on.
Issues that would likely not have happened with a better state management system
- https://blueprints.launchpad.net/nova/+spec/compute-instance-cleanup-service
- https://bugs.launchpad.net/nova/+bug/1050979
- https://bugs.launchpad.net/nova/+bug/1061024
- https://bugs.launchpad.net/nova/+bug/1082414
- ...
Requirements
https://etherpad.openstack.org/task-system
Discussions
https://etherpad.openstack.org/the-future-of-orch
Plan of record
- Create prototype.
- Get feedback from summit session.
- Get more feedback from email list & heat folks about common library.
- Adjust prototype as needed from feedback.
- Split prototype into small chunks.
- Adjust tests for each small chunks.
- Start to submit prototype chunks into http://review.openstack.org (disabling whole/pieces component until ready to turn on?).
Design
Design details
In order to implement of this new orchestration layer the following key concepts must be built into the design from the start.
- A set of atomic tasks that can be organized into a workflow.
- Task resumption.
- Task rollback.
- Task tracking.
- Resource locking.
- Workflow sharding/ownership.
- Simplicity (allowing for extension and verifiability).
- Tolerant to upgrades.