Jump to: navigation, search

Convection

Revision as of 00:56, 17 April 2013 by Kebray (talk | contribs) (What is a Workflow?)

PROPOSAL ONLY: Workflow-as-a-Service (Convection)

Please note that this is a PROPOSAL ONLY. This is not yet implemented.

What is Convection

Convection is a proposal for a new Workflow-as-a-Service project for OpenStack clouds. Convection could be a public facing API service that provides task and state management capabilities, enabling OpenStack API consumers to build complex multi-step applications running on an OpenStack cloud which could be a public cloud, private cloud, or a hybrid cloud. Convection could also be a service that other OpenStack projects could leverage to perform work. e.x. One possible method for Heat to perform orchestration of standing up cloud stacks could be to leverage a Workflow service for task oriented steps of spinning up and connecting cloud resources. Conversely, customers wanting to run meta-workflows could leverage Heat as one task in their meta workflow where orchestration of a stack is one task in a larger meta workflow.

What is a Workflow?

Definition There are static workflows and dynamic workflows.

Isn't Workflow an overloaded term?

YES! There are arguments about what the term Workflow actually means. For Convection conversation purposes, let's define the following terminology:

Workflow Terms

  1. Deterministic (Static) Workflow: In an academic context, a workflow is sometimes described as a collection of ordered tasks that occur with a defined start, order, and end. Some tasks may be able to execute in parallel, but a pre-determined tree of workflow steps (and parellel branches) is known before runtime, and the flow of the tree is followed upon every execution of the workflow.
  2. Event Based (Dynamic) Workflow: A collection of tasks, some of which may or may not have a required order of execution, where task execution is coordinated through communication of events by individual task start/stop/status notifications. In an event based workflow system, there could be a central task execution coordinator that handles listening for events of task completion and sending events for new tasks to start. Or, code that executes an individual task can encode its own logic to know when to execute based off events directly sent from other tasks.

I make no attempt to specify an idealistic implementation here in this PROPOSAL. I simply want to document some workflow concepts, and leverage the community for collaborative design of a useful Workflow system for OpenStack clouds.

Use Cases for Workflow-as-a-Service

Possible use cases for a Workflow Service could include:

  • Long running processes
  • Batch Processes (e.g. encoding/decoding)
  • Task execution management for spinning up resources via an orchestration system (such as Heat)
  • Scheduled workflows via Nova Scheduler
  • etc

Why the name Convection?

Convection was a name proposed by Tim Simpson (Reddwarf developer). The idea is that (1) Convection "conveys," implying order; (2) Convection is often thought in terms of ovens which produce heat, and the OpenStack project Heat could be one possible consumer of Workflow where task flows could be analogous to air flow in a convection oven.

Potential Workflow-as-a-Service Capabilities

The following is a list of PROPOSED capabilities for Convection. These are not necessarily required for a minimum viable service and are just ideas of what a workflow service might provide:

  • Workflow Engine: A workflow engine could provide generic task and state management capabilities. A workflow engine could act as a central state coordinator, enabling workflow client applications to be distributed across public cloud and on-premise deployments. Workflow clients offload state management to the Workflow service thereby allowing the workflow clients to be stateless, scalable, and tolerant of process and client failures. The workflow engine could support configurable constraints at both the workflow and task level, e.g. timeouts, retry count, retry intervals, etc.
  • Workflow Repository: A workflow repository could expose a set of pre-determined common task flows (e.g. spin up a server and add it to a load balancer).
  • Command Line Tool / Dashboard: Since OpenStack is a cloud operating system, some operating system tools like top to see a list of running jobs in the cloud could be very useful. Tools could provide a drill down of existing workflows, currently running workflows, workflows in states of various execution: running, completed, failed, ready-to-run... and provide the ability to resubmit/retry failed workflow jobs. Workflow tools could also provide analytics -- metrics which could help identify performance bottlenecks or common areas of failures in a workflow that is repeated over and over. Some possible metrics could be: average execution time for a workflow, average execution time for individual workflow tasks, task/workflow failure rates, etc.