Difference between revisions of "TaskFlow"
Adrian Otto (talk | contribs) (→Executive Summary) |
|||
| Line 5: | Line 5: | ||
[[File:Flo.jpg|thumbnail]] | [[File:Flo.jpg|thumbnail]] | ||
| − | Taskflow is | + | Taskflow is a Python library for OpenStack that helps make task execution easy, consistent, and reliable. It allows the creation of lightweight <code>task</code> objects and/or functions that are combined together into <code>flows</code> (aka: workflows). It includes components for running these <code>flows</code> in a manner that can be stopped, resumed, and safely reverted. Projects implementing the Taskflow library enjoy added state resiliency, and fault tolerance. It simplifies crash recovery. Think of it as a way to protect an action, similar to the way transactions protect operations in a RDBMS. If a manager process is terminated while an action was in progress, there is a risk that unprotected code would leave the system in a degraded or inconsistent state. With Taskflow, interrupted actions may be resumed or rolled back automatically when a manager process is resumed. |
| + | |||
| + | Using Taskflow to organize actions into lightweight <code>task</code> objects makes atomic code sequences easily testable. Lightweight <code>tasks</code> are arranged into <code>flows</code> (aka: workflows). A <code>flow</code> facilitates the execution of a defined sequence of ordered <code>tasks</code>. A <code>flow</code> is a ''structure'' (a set of tasks linked together), so it allows the calling code and the workflow to be disconnected so <code>flows</code> can be reused. Taskflow provides a few mechanisms for running <code>flows</code> and lets the developer pick and choose which one will work for their needs. | ||
| + | |||
| + | ===Conceptual Example=== | ||
| + | This pseudo code illustrates what how a taskflow would work for those who are familiar with SQL transactions. | ||
| + | |||
| + | START TRANSACTION | ||
| + | task1: call nova API to launch a server | ||
| + | task2: when task1 finished, call cinder API to attach block storage to the server | ||
| + | ...perform other tasks... | ||
| + | COMMIT | ||
| + | |||
| + | The above <code>flow</code> could be used by Heat as part of an orchestration to add a server with block storage attached. It may launch several of these in parallel to prepare a number of identical servers. | ||
== Background == | == Background == | ||
Revision as of 21:45, 21 June 2013
Revised on: 6/21/2013 by Adrian Otto
Contents
Executive Summary
Taskflow is a Python library for OpenStack that helps make task execution easy, consistent, and reliable. It allows the creation of lightweight task objects and/or functions that are combined together into flows (aka: workflows). It includes components for running these flows in a manner that can be stopped, resumed, and safely reverted. Projects implementing the Taskflow library enjoy added state resiliency, and fault tolerance. It simplifies crash recovery. Think of it as a way to protect an action, similar to the way transactions protect operations in a RDBMS. If a manager process is terminated while an action was in progress, there is a risk that unprotected code would leave the system in a degraded or inconsistent state. With Taskflow, interrupted actions may be resumed or rolled back automatically when a manager process is resumed.
Using Taskflow to organize actions into lightweight task objects makes atomic code sequences easily testable. Lightweight tasks are arranged into flows (aka: workflows). A flow facilitates the execution of a defined sequence of ordered tasks. A flow is a structure (a set of tasks linked together), so it allows the calling code and the workflow to be disconnected so flows can be reused. Taskflow provides a few mechanisms for running flows and lets the developer pick and choose which one will work for their needs.
Conceptual Example
This pseudo code illustrates what how a taskflow would work for those who are familiar with SQL transactions.
START TRANSACTION task1: call nova API to launch a server task2: when task1 finished, call cinder API to attach block storage to the server ...perform other tasks... COMMIT
The above flow could be used by Heat as part of an orchestration to add a server with block storage attached. It may launch several of these in parallel to prepare a number of identical servers.
Background
Taskflow started as a prototype with the NTTdata corporation along with Yahoo! for nova and has moved into a more general solution/library that can form the underlying structure of multiple OpenStack projects at once.
Wiki with requirements and more background:
Why
OpenStack code is highly unstructured (likely due to organic growth of code and architecture) and even though the unstructured code does work there are many failure scenarios skipped and/or recovery scenarios which are not possible in unstructured code. With the aid of taskflow and the structuring/organization it brings it allows for these scenarios to be a non-problem and allows for new & very desirable functionality to be introduced (resumption, reversion).
Design
Key primitives: https://wiki.openstack.org/wiki/StructuredWorkflowPrimitives
Tasks
Flows
Activation
Distributed:
Traditional:
