TaskFlow/Best practices

Mind-Altering Substances

Using taskflow requires a slight shift in mind-set and changes a little bit of how your normal code workflow would run. The taskflow team has tried to keep the amount of mind-altering required to use taskflow to as much of a minimum as possible (since mind-altering means learning new concepts, or suppressing existing ones) to make it easy to adopt taskflow into your service/application/library.

Exceptions

Exceptions that occur in a task, and which are not caught by the internals of a task will be default currently trigger reversion of the workflow that task was in to start occurring. If multiple tasks in a workflow raise exceptions (say they are executing at the same time via a parallel engine or a distributed engine) then the individual paths that lead to that task will be reverted (if an ancestor task is shared by multiple failing tasks, it will be reverted only once). In the future reversion strategies should be able to make this more customizable (allowing more customizable ways to handle or alter the reversion process).

Execution flow

When a set of tasks and associated structure that contains those tasks (aka the flows that create that structure) are given to an engine, along with a possible (but not needed) backend where the engine can store intermediate results (which is needed if the workflow should be able to resume on failure) the engine becomes the execution unit that is responsible for reliably executing the tasks that are contained in the flows that you provide it. That task will ensure the structure that is provided is retained when executing. For example a linear ordering of tasks by using a linear_flow structure will always be ran in linear order. A set of tasks that are structured in dependency ordering will always be ran in that dependency order. These constraints the engine must adhere to; note other constraints may be enforced by the engine type that is being activated (ie a single threaded engine will only run in a single thread, a distributed or worker based engine will run remotely). So when selecting an engine to use, make sure to carefully select the desired feature set that will work for your application.

Control flow

This is a variation in how a programmer normally programs, in order to be able to track the execution of your workflow, the workflow must be split up into small pieces (in a way similar to functions) that taskflow can then run in a ahead of time defined structure (aka a flow). Taskflow engines using this information then can run your structure in a well defined and resumable manner. This does though currently have a few side-effects in that certain traditional operations (if-then-else, do-while, fork-join, map-reduce) become more complex in that those types of control flows do not easily map to a representation that can be easily resumed. To keep taskflow relatively minimal we are trying to use the minimal run workflow operation to completion with customized reverting strategies to accomplish most of the use-cases in openstack. If these control flows become valueable then we will revisit if and how we should make these accessible to users of taskflow.

NOTE: Inside of a task the execute() method of that task may use whichever existing control flow it desires (any supported by python), but outside of the execute() the set of control flow operators are more minimal (due to above reasoning).