StructuredStateManagement

Summary

Move away from ad-hoc states and state transitions to a more concrete organized structured state management in the various openstack projects (initial interests are in nova/heat).

What problems does this solve in general

• Increases the [stability, extendability, reliability] of nova/heat.
• Makes it easier to [debug, test, understand, verify, review] nova/heat code.
• Removes hard to discover state-transition dependencies and interactions with clearly defined state-transition dependencies and interactions.
• Ensures state transitions are done reliably and correctly by isolating those transitions to a single place.
• Fixes a variety of problems that previously had piecemeal like patches applied.
• Eliminates the inherent fragility of a ad-hoc workflow.
  • They are hard to debug, hard to verify, hard to adjust, hard to understand (just hard in general)...
• Makes it possible to audit & track the state transitions performed on a given resource.

What problems does this solve in nova

• Removes the need for periodic tasks to cleanup garbage (orphaned instances, orphaned resources...) left by nova's ad-hoc states.
• Creates the path for smart resource scheduling.
• Makes it possible to do [resizing, live migration] in a more secure and manageable manner.
  • Discussion about how this can be done correctly require a intermediary to orchestrate this ownership transfer.
• Makes it possible for nova to have multi-stage booting where an instances and its dependent resources are first reserved, the resources configured, the instance configured, and then finally the instance is powered-on (thus completing the instance provisioning process).
• Addresses the underlying key point of http://www.slideshare.net/harlowja/nova-states-summit/9 where states will now be fully recovered from on cutting.

Issues that would likely not have happened with a better state management system

• In nova
  • https://blueprints.launchpad.net/nova/+spec/compute-instance-cleanup-service
  • https://bugs.launchpad.net/nova/+bug/1050979
  • https://bugs.launchpad.net/nova/+bug/1061024
  • https://bugs.launchpad.net/nova/+bug/1082414
  • ...

Connected blueprints

• https://blueprints.launchpad.net/nova/+spec/structured-state-management
  • https://blueprints.launchpad.net/nova/+spec/structured-state-management-core-library
  • https://blueprints.launchpad.net/nova/+spec/structured-state-management-run-instance-path

Connected wikis

• https://wiki.openstack.org/wiki/Convection

Requirements

https://etherpad.openstack.org/task-system

Discussions

https://etherpad.openstack.org/the-future-of-orch

Plan of record

1. Create prototype
   1. Create library and use said library in nova for run_instance api in nova.
2. Get feedback others
   1. Get feedback from summit session on said discussion and prototype.
   2. Get more feedback from email list & heat folks about common library.
3. Adjust prototype as needed from feedback.
4. Split prototype+library into small chunks.
5. Adjust tests for each small chunks.
6. Start to submit chunks into http://review.openstack.org (disabling whole/pieces component until ready to turn on?).

Prototype

https://github.com/Yahoo/NovaOrc

Design

Design details

In order to implement of this new orchestration layer the following key concepts must be built into the design from the start.

1. A set of atomic tasks that can be organized into a workflow.
2. Task resumption.
3. Task rollback.
4. Task tracking.
5. Resource locking.
6. Workflow sharding/ownership.
7. Simplicity (allowing for extension and verifiability).
8. Tolerant to upgrades.