Jump to: navigation, search

Difference between revisions of "StructuredStateManagement"

(→‎How it will be addressed)
 
(85 intermediate revisions by 2 users not shown)
Line 1: Line 1:
−
== Summary ==
+
Drafter:  [[Harlowja]]
  
−
Move away from ad-hoc states and state transitions for resource acquisition and modification to a more concrete organized structured state management system in nova.
+
Revised on: {{REVISIONMONTH1}}/{{REVISIONDAY}}/{{REVISIONYEAR}} by {{REVISIONUSER}}
  
−
=== What problems does this solve in general ===
+
=== Rationale ===
  
−
* Increases the [stability, extendability, reliability] of the various openstack projects.
+
Move away from ad-hoc states and state transitions for resource acquisition and modification to a more well-defined organized state management system. This new state management system will have advanced new & shiny features such as greater stability, automatic recovery mechanisms and greater scalability than what currently exists  (and more!).
−
* Makes it easier to [debug, test, understand, verify, review] the projects which have a workflow-like concept.
 
−
* Removes hard to discover state-transition dependencies and interactions with clearly defined state-transition dependencies and interactions.
 
−
* Ensures state transitions are done reliably and correctly by isolating those transitions to a single place/entity.
 
−
* Fixes a variety of problems that previously had piecemeal like patches applied to attempt to solve them (avoiding fixing the larger problem).
 
−
* Eliminates the inherent  fragility of the current ad-hoc workflows that exist in the openstack projects.
 
−
** They are by there ad-hoc nature hard to debug, hard to verify, hard to adjust, hard to understand (just hard in general)...
 
−
* Makes it possible to audit & track the state transitions performed on a given resource.
 
−
** This kind of functionality has started to appear in nova, but the ad-hoc nature was preserved :-(
 
−
*** See: https://github.com/openstack/nova/blob/stable/grizzly/nova/compute/utils.py#L305
 
−
* Addresses the underlying key point of http://www.slideshare.net/harlowja/nova-states-summit/9 where states will now be fully recovered from on ''cutting''.
 
  
−
==== What problems does this solve in nova (+ the general ones) ====
+
=== Definitions ===
  
−
* Removes the need for periodic tasks to cleanup ''garbage'' (orphaned instances, orphaned resources...) left by nova's ad-hoc states.
+
;State
−
* Creates the path for ''smart'' resource scheduling.
+
: The particular condition that someone or something is in at a specific time: "the state of the instance request".
−
* Makes it possible to do [resizing, live migration] in a more secure and manageable manner.
+
;State transition
−
** Discussion about how this can be done correctly require a intermediary to orchestrate this ownership transfer.
+
: Altering a state by applying a function on-top of that state (of which said function may take inputs and provide outputs) resulting in a new state.
−
* Makes it possible for nova to have multi-stage booting where an instances and its dependent resources are first reserved, the resources configured, the instance configured, and then finally the instance is ''powered-on'' (thus completing the instance provisioning process).
+
;Task
 +
: The application of a state transition on a given state.
 +
;Workflow
 +
: The sequence of administrative/other processes through which a piece of work passes from initiation to completion.
 +
;State management engine
 +
: An individual or entity that arranges or control the elements of, as to achieve a desired overall effect.
  
−
==== Issues that would likely not have happened with a better state management system ====
+
=== What problems is this attempting to solve ===
 +
 
 +
* Increasing the ['extendability', 'recoverability', 'reliability', 'stability'] of states and state transitions.
 +
* Making it easier to [ 'add', 'debug', 'review', 'test', 'understand', 'verify'] existing & new states and state transitions.
 +
* Removing hard to discover state & transition dependencies and interactions.
 +
* Ensuring state transitions are done ['correctly', 'reliably'] by isolating those transitions to a entity whose exclusive responsibility is to ['correctly', 'reliably'] perform said transitions.
 +
* Fixing a variety of problems that previously had piecemeal like patches applied to solve them.
 +
* Eliminating the inherent ''fragility'' of the current ad-hoc workflows.
 +
* Allows for upgrading a cloud/software with inflight actions without needing later manual cleanup of said  actions.
 +
* Makes it possible to audit & track the state transitions performed on a given resource in a unified manner.
 +
** '''Note:''' that there currently exists notifications, logging, event reporting as different mechanisms (which are not used in a uniform manner).
 +
* Removes the need for certain types of periodic tasks which start to consume more and more resources as  you grow your cluster larger.
 +
* Moves toward the path where actions will automatically recover on ''cutting'' events (node failure, resource failure, network failure..).
 +
* Removes the need for periodic tasks to cleanup ''garbage'' (orphaned instances/resources/tasks...) left behind.
 +
* Creates the foundation for a more reliable and automatic recovery process when errors do occur.
 +
* Encourages the ['altering', 'extension'] of default workflows with a more ['complex', 'custom', 'experimental'] workflows.
 +
 
 +
==== Fixing a known need ====
  
 
* https://blueprints.launchpad.net/nova/+spec/compute-instance-cleanup-service
 
* https://blueprints.launchpad.net/nova/+spec/compute-instance-cleanup-service
 +
* https://blueprints.launchpad.net/cinder/+spec/cinder-state-machine
 +
* https://bugs.launchpad.net/nova/+bug/1173408
 +
* https://bugs.launchpad.net/nova/+bug/1173413
 +
* https://bugs.launchpad.net/nova/+bug/1173417
 +
* https://bugs.launchpad.net/nova/+bug/1173420
 
* https://bugs.launchpad.net/nova/+bug/1050979
 
* https://bugs.launchpad.net/nova/+bug/1050979
 
* https://bugs.launchpad.net/nova/+bug/1061024
 
* https://bugs.launchpad.net/nova/+bug/1061024
 
* https://bugs.launchpad.net/nova/+bug/1082414
 
* https://bugs.launchpad.net/nova/+bug/1082414
−
* ...
+
* https://bugs.launchpad.net/nova/+bug/1173429
 +
* https://bugs.launchpad.net/nova/+bug/1173430
 +
* (and more)
  
−
=== Connected blueprints ===
+
=== Blueprints ===
  
 
* https://blueprints.launchpad.net/nova/+spec/structured-state-management
 
* https://blueprints.launchpad.net/nova/+spec/structured-state-management
−
** https://blueprints.launchpad.net/nova/+spec/structured-state-management-core-library
+
* https://blueprints.launchpad.net/cinder/+spec/cinder-state-machine
−
** https://blueprints.launchpad.net/nova/+spec/structured-state-management-run-instance-path
+
 
 +
=== Related papers ===
  
−
=== Connected wikis ===
+
* http://www.netdb.cis.upenn.edu/papers/tropic_tr.pdf
 +
* http://research.microsoft.com/pubs/64604/osr2007.pdf
  
−
https://wiki.openstack.org/wiki/Convection
+
=== Related wikis ===
  
−
=== Requirements ===
+
* https://wiki.openstack.org/wiki/Convection
 +
* https://wiki.openstack.org/wiki/NovaOrchestration/WorkflowEngines ('''old''')
 +
 
 +
=== Potential Requirements ===
  
 
https://etherpad.openstack.org/task-system
 
https://etherpad.openstack.org/task-system
  
−
=== Discussions ===
+
=== Summit Discussions ===
  
−
https://etherpad.openstack.org/the-future-of-orch
+
'''Havana summit:'''
 +
* https://etherpad.openstack.org/the-future-of-orch
 +
* https://etherpad.openstack.org/Summit-Havana-Cinder-Safe-Shutdown
  
 
=== Plan of record ===
 
=== Plan of record ===
Line 55: Line 81:
 
==== Step 1 ====
 
==== Step 1 ====
  
−
# Create prototype
+
'''Create prototype'''
−
## Create library and use said library in nova for ''run_instance'' action in nova.
+
 
 +
# Create core workflow/task library and prototype using said library in nova for ''run_instance'' action.
 +
# Split this action (refactored) into small atomic task chunks (don't aim for perfection just yet, since its a prototype).
 +
# Organize chunks into a workflow and test workflow.
 +
# Show working prototype at summit session (with associated docs...)
  
 
==== Step 2 ====
 
==== Step 2 ====
  
−
# Get feedback on prototype.
+
# Get feedback on prototype from people involved in making it + other interested parties.
−
## Get feedback from summit session.
+
# Get feedback from summit session.
−
## Get more feedback from email list + other interested parties.
+
# Get more feedback from email list + other interested parties.
 +
# Form group of interested folks that would like to help move forward the prototypes design principles & usage in other core projects.
  
 
==== Step 3 ====
 
==== Step 3 ====
  
−
# Adjust nova prototype as needed from feedback.
+
# Select single first target project to use new ''taskflow'' library.
−
# Split nova prototype into small chunks.
+
# Incubate ''taskflow'' library using parts from prototype inside said project.
−
# Adjust tests for each small chunks (depending on what it changes).
+
## Keep other active projects involved from the start (so that said library can be easily used there).
 +
# Prove library in said single first target project by using it to do a key workflow.
 +
## Adjust workflow in chunks in said target project to use said library.
 +
# Adjust tests for each small chunks (depending on what it changes) and add new ones for new functionality.
 
# Submit chunks into http://review.openstack.org (disabling whole/pieces component until ready to turn on?).
 
# Submit chunks into http://review.openstack.org (disabling whole/pieces component until ready to turn on?).
  
 
==== Step 4 ====
 
==== Step 4 ====
  
−
# Pick another nova action and refactor it to use design from prototype
+
# Move ''taskflow'' library (and associated tests) into oslo.
−
# Split this other action (refactored) into small chunks.
+
 
−
# Adjust tests for each small chunks (depending on what it changes).
+
==== Step 5 ====
 +
 
 +
# Pick another flow and refactor it in said first project ''and/or'' pick another interested project for said flow.
 +
# Split this other flow into small chunks using said library.
 +
# Adjust unit tests for each small chunks (depending on what it changes) and add new ones for new functionality.
 
# Submit chunks into http://review.openstack.org (disabling whole/pieces component until ready to turn on?).
 
# Submit chunks into http://review.openstack.org (disabling whole/pieces component until ready to turn on?).
−
# Repeat
+
# Rinse & repeat.
  
 +
== Prototype ==
  
−
=== Prototype ===
+
https://github.com/Yahoo/NovaOrc
  
−
https://github.com/Yahoo/NovaOrc
+
=== Design ===
  
−
== Design ==
 
 
[[File:New-arch.png|thumbnail|center]]
 
[[File:New-arch.png|thumbnail|center]]
  
 +
===  Workflow ===
 +
[[File:Run_workflow.png|thumbnail|center]]
  
 +
== More  Details! ==
  
 +
See: [[StructuredStateManagementDetails]]
  
−
=== Design details ===
+
See: [[StructuredWorkflowPrimitives]]
−
 
 
−
In order to implement of this new orchestration layer the following key concepts must be built into the design from the start.
 
−
 
 
−
# A set of '''atomic''' tasks that can be organized into a workflow.
 
−
# Task resumption.
 
−
# Task rollback.
 
−
# Task tracking.
 
−
# Resource locking.
 
−
# Workflow sharding/ownership.
 
−
# Simplicity (allowing for extension and verifiability).
 
−
# Tolerant to upgrades.
 
−
 
 
−
==== Atomic tasks ====
 
−
 
 
−
===== Why it matters =====
 
−
 
 
−
Tasks that are created (either via code or other operation) must be atomic so that the task as a unit can be said to have completed or the task as a unit can be said to have failed. This allows for said task to be rolled back as a unit. It is also useful to be able to be able to accurately track exactly what tasks have been applied to a given workflow, which is inherently useful for correct status tracking (and is directly tied to how resumption is done).
 
−
 
 
−
===== How it will be addressed  =====
 
−
 
 
−
Tasks which previously were very unorganized in the ''run_instance'' path of nova will need to be refactored into clearly defined tasks (likely with an ''apply()'' method and a ''rollback()'' method).  These tasks will be split up so that each task performs a clear single piece of work in an atomic manner (aka not one big task that does many different things) where possible. This will also help make testing of said task easier (since it will have a clear set of inputs and a clear set of expected outputs/side-effects, of which the ''rollback()'' method should undo).
 
−
 
 
−
'''For example''' this could be a task/state baseclass:
 
−
 
 
−
<nowiki>
 
−
class State(base.Base):
 
−
    __metaclass__ = abc.ABCMeta
 
−
 
 
−
    def __init__(self):
 
−
        super(State, self).__init__()
 
−
 
 
−
    def __str__(self):
 
−
        return "State: %s" % (self.__class__.__name__)
 
−
 
 
−
    @abc.abstractmethod
 
−
    def apply(self, context, *args, **kwargs):
 
−
        raise NotImplementedError()
 
−
 
 
−
    def revert(self, context, result, chain, excp, cause):
 
−
        pass</nowiki>
 
−
 
 
−
A set of these objects can be ''ran'' with the following:
 
−
 
 
−
<nowiki>
 
−
class StateChain(object):
 
−
    def __init__(self, name, tolerant=False, parents=None):
 
−
        self.reversions = []
 
−
        self.name = name
 
−
        self.tolerant = tolerant
 
−
        self.states = OrderedDict()
 
−
        self.results = OrderedDict()
 
−
        self.parents = parents
 
−
        self.result_fetcher = None
 
−
        self.change_tracker = None
 
−
        self.listeners = []
 
−
 
 
−
    def __setitem__(self, name, performer):
 
−
        self.states[name] = performer
 
−
 
 
−
    def __getitem__(self, name):
 
−
        return self.results[name]
 
−
 
 
−
    def run(self, context, *args, **kwargs):
 
−
        for (name, performer) in self.states.items():
 
−
            try:
 
−
                self._on_state_start(context, performer, name)
 
−
                # See if we have already ran this... (resumption!)
 
−
                result = None
 
−
                if self.result_fetcher:
 
−
                    result = self.result_fetcher(context, name, self)
 
−
                if result is None:
 
−
                    result = performer.apply(context, *args, **kwargs)
 
−
                # Keep a pristine copy of the result in the results table
 
−
                # so that if said result is altered by other further states
 
−
                # the one here will not be.
 
−
                #
 
−
                # Note: python is by reference objects, so someone else could screw with this,
 
−
                # which would be bad if we need to rollback and a result we created was modified by someone else...
 
−
                self.results[name] = copy.deepcopy(result)
 
−
                self._on_state_finish(context, performer, name, result)
 
−
            except Exception as ex:
 
−
                with excutils.save_and_reraise_exception():
 
−
                    try:
 
−
                        self._on_state_error(context, name, ex)
 
−
                    except:
 
−
                        pass
 
−
                    cause = (name, performer, (args, kwargs))
 
−
                    self.rollback(context, name, self, ex, cause)
 
−
        return self
 
−
 
 
−
    def _on_state_error(self, context, name, ex):
 
−
        if self.change_tracker:
 
−
            self.change_tracker(context, ERRORED, name, self)
 
−
        for i in self.listeners:
 
−
            i.notify(context, ERRORED, name, self, error=ex)
 
−
 
 
−
    def _on_state_start(self, context, performer, name):
 
−
        if self.change_tracker:
 
−
            self.change_tracker(context, STARTING, name, self)
 
−
        for i in self.listeners:
 
−
            i.notify(context, STARTING, name, self)
 
−
 
 
−
    def _on_state_finish(self, context, performer, name, result):
 
−
        # If a future state fails we need to ensure that we
 
−
        # revert the one we just finished.
 
−
        self.reversions.append((name, performer))
 
−
        if self.change_tracker:
 
−
            self.change_tracker(context, COMPLETED, name, self,
 
−
                                result=result.to_dict())
 
−
        for i in self.listeners:
 
−
            i.notify(context, COMPLETED, name, self, result=result)
 
−
 
 
−
    def rollback(self, context, name, chain=None, ex=None, cause=None):
 
−
        if chain is None:
 
−
            chain = self
 
−
        for (i, (name, performer)) in enumerate(reversed(self.reversions)):
 
−
            try:
 
−
                performer.revert(context, self.results[name], chain, ex, cause)
 
−
            except excp.NovaException:
 
−
                # Ex: WARN: Failed rolling back stage 1 (validate_request) of
 
−
                # chain validation due to nova exception
 
−
                # WARN: Failed rolling back stage 2 (create_db_entry) of
 
−
                # chain init_db_entry due to nova exception
 
−
                msg = _("Failed rolling back stage %s (%s)"
 
−
                        " of chain %s due to nova exception.")
 
−
                LOG.warn(msg, (i + 1), performer.name, self.name)
 
−
                if not self.tolerant:
 
−
                    # This will log a msg AND re-raise the Nova exception if
 
−
                    # the chain does not tolerate exceptions
 
−
                    raise
 
−
            except Exception:
 
−
                # Ex: WARN: Failed rolling back stage 1 (validate_request) of
 
−
                # chain validation due to unknown exception
 
−
                # WARN: Failed rolling back stage 2 (create_db_entry) of
 
−
                # chain init_db_entry due to unknown exception
 
−
                msg = _("Failed rolling back stage %s (%s)"
 
−
                        " of chain %s, due to unknown exception.")
 
−
                LOG.warn(msg, (i + 1), performer.name, self.name)
 
−
                if not self.tolerant:
 
−
                    # Log a msg AND re-raise the generic Exception if the
 
−
                    # Chain does not tolerate exceptions
 
−
                    raise
 
−
        if self.parents:
 
−
            # Rollback any parents chains
 
−
            for p in self.parents:
 
−
                p.rollback(context, name, chain, ex, cause)</nowiki>
 
−
 
 
−
==== Task resumption ====
 
−
 
 
−
===== Why it matters =====
 
−
 
 
−
===== How it will be addressed =====
 
−
 
 
−
==== Task rollback ====
 
−
 
 
−
===== Why it matters =====
 
−
 
 
−
===== How it will be addressed =====
 
−
 
 
−
==== Task tracking ====
 
−
 
 
−
===== Why it matters =====
 
−
 
 
−
===== How it will be addressed =====
 
−
 
 
−
==== Resource locking ====
 
−
 
 
−
===== Why it matters =====
 
−
 
 
−
===== How it will be addressed =====
 
−
 
 
−
==== Workflow sharding/ownership ====
 
−
 
 
−
===== Why it matters =====
 
−
 
 
−
===== How it will be addressed =====
 
−
 
 
−
==== Simplicity ====
 
−
 
 
−
===== Why it matters =====
 
−
 
 
−
===== How it will be addressed =====
 
−
 
 
−
==== Tolerant to upgrades ====
 
−
 
 
−
===== Why it matters =====
 
  
−
===== How it will be addressed =====
+
Get involved: [[StructuredWorkflows]]

Latest revision as of 20:16, 25 May 2013

Drafter: Harlowja

Revised on: 5/25/2013 by Harlowja

Rationale

Move away from ad-hoc states and state transitions for resource acquisition and modification to a more well-defined organized state management system. This new state management system will have advanced new & shiny features such as greater stability, automatic recovery mechanisms and greater scalability than what currently exists (and more!).

Definitions

State
The particular condition that someone or something is in at a specific time: "the state of the instance request".
State transition
Altering a state by applying a function on-top of that state (of which said function may take inputs and provide outputs) resulting in a new state.
Task
The application of a state transition on a given state.
Workflow
The sequence of administrative/other processes through which a piece of work passes from initiation to completion.
State management engine
An individual or entity that arranges or control the elements of, as to achieve a desired overall effect.

What problems is this attempting to solve

  • Increasing the ['extendability', 'recoverability', 'reliability', 'stability'] of states and state transitions.
  • Making it easier to [ 'add', 'debug', 'review', 'test', 'understand', 'verify'] existing & new states and state transitions.
  • Removing hard to discover state & transition dependencies and interactions.
  • Ensuring state transitions are done ['correctly', 'reliably'] by isolating those transitions to a entity whose exclusive responsibility is to ['correctly', 'reliably'] perform said transitions.
  • Fixing a variety of problems that previously had piecemeal like patches applied to solve them.
  • Eliminating the inherent fragility of the current ad-hoc workflows.
  • Allows for upgrading a cloud/software with inflight actions without needing later manual cleanup of said actions.
  • Makes it possible to audit & track the state transitions performed on a given resource in a unified manner.
    • Note: that there currently exists notifications, logging, event reporting as different mechanisms (which are not used in a uniform manner).
  • Removes the need for certain types of periodic tasks which start to consume more and more resources as you grow your cluster larger.
  • Moves toward the path where actions will automatically recover on cutting events (node failure, resource failure, network failure..).
  • Removes the need for periodic tasks to cleanup garbage (orphaned instances/resources/tasks...) left behind.
  • Creates the foundation for a more reliable and automatic recovery process when errors do occur.
  • Encourages the ['altering', 'extension'] of default workflows with a more ['complex', 'custom', 'experimental'] workflows.

Fixing a known need

Blueprints

Related papers

Related wikis

Potential Requirements

https://etherpad.openstack.org/task-system

Summit Discussions

Havana summit:

Plan of record

Step 1

Create prototype

  1. Create core workflow/task library and prototype using said library in nova for run_instance action.
  2. Split this action (refactored) into small atomic task chunks (don't aim for perfection just yet, since its a prototype).
  3. Organize chunks into a workflow and test workflow.
  4. Show working prototype at summit session (with associated docs...)

Step 2

  1. Get feedback on prototype from people involved in making it + other interested parties.
  2. Get feedback from summit session.
  3. Get more feedback from email list + other interested parties.
  4. Form group of interested folks that would like to help move forward the prototypes design principles & usage in other core projects.

Step 3

  1. Select single first target project to use new taskflow library.
  2. Incubate taskflow library using parts from prototype inside said project.
    1. Keep other active projects involved from the start (so that said library can be easily used there).
  3. Prove library in said single first target project by using it to do a key workflow.
    1. Adjust workflow in chunks in said target project to use said library.
  4. Adjust tests for each small chunks (depending on what it changes) and add new ones for new functionality.
  5. Submit chunks into http://review.openstack.org (disabling whole/pieces component until ready to turn on?).

Step 4

  1. Move taskflow library (and associated tests) into oslo.

Step 5

  1. Pick another flow and refactor it in said first project and/or pick another interested project for said flow.
  2. Split this other flow into small chunks using said library.
  3. Adjust unit tests for each small chunks (depending on what it changes) and add new ones for new functionality.
  4. Submit chunks into http://review.openstack.org (disabling whole/pieces component until ready to turn on?).
  5. Rinse & repeat.

Prototype

https://github.com/Yahoo/NovaOrc

Design

New-arch.png

Workflow

Run workflow.png

More Details!

See: StructuredStateManagementDetails

See: StructuredWorkflowPrimitives

Get involved: StructuredWorkflows

Retrieved from "https://wiki.openstack.org/w/index.php?title=StructuredStateManagement&oldid=23471"