Difference between revisions of "TaskFlow/Best practices"

Revision as of 10:16, 12 December 2013

Revised on: 12/12/2013 by Ivan Melnikov

Why

Since taskflow creates a path toward stable, resumable, and trackable workflows it is helpful to have a small set of recommended best practices to make sure you as a user of taskflow (the library) maximize the benefit you receive from using taskflow. Certain common patterns and best practices will be listed below to show the what, why and how so that you can maximize your taskflow experience and minimize the pain associated with understanding some of the taskflow key primitives.

Note: this list will likely continue growing as new usages of taskflow emerge (and new features are introduced into taskflow), please feel free to add any of your own below.

Tips & Tricks

Create reuseable tasks that do one thing well; if that task has side-effects undo those side-effects in the revert method.
Clearly define what a task provides so that future (potentially not yet created) tasks can depend on those outputs.
Clearly define what a task requires so that engine/s can deduce the correct order to run your tasks in.
If your tasks have requirements with the same name, but you want to accept a different input, use the remapping/rebinding feature.
Using shared inputs by declaring what a task requires is desirable, but be careful though about sharing the output of a task (especially if the output may not be thread-safe) with more than one receiving task (prefer immutability if this is the case).
- Be careful since it is very easy to switch an engine from serial to parallel (this is a feature not a bug).
- If this is a still concern use the eventlet based executor to avoid synchronization issues yet still run in a parallel manner (but not using threads).
Link tasks that do one thing well together with the supplied patterns to create flows that do many things well together.
For hierarchical workflows, prefer pattern composition (a flow with a subflow...) over excessive manual linking.
Prefer automatic ordering deduction over manual ordering deduction as the former is more resilient to future alternations while the latter is not.
Be very careful about RPC boundaries and be meticulous about how these boundaries affect your flows & tasks & your application.
Always associate a (major, minor) version with your tasks so that on software upgrades the previously (potentially partially) completed tasks can be migrated & resumed/reverted.
Return serializable objects from tasks not resources (or other non-serializable objects); expect that all returned objects may be persisted indefinitely.
Clean up after yourself, logbooks should be eventually expired and the underlying data deleted; do this periodically.
- TODO: the following blueprint should make this programmable & less manual.
Clearly name your tasks with relevant names; names are how restarted flows are re-associated with there previously (potentially partially) completed tasks for resumption or reversion so choose carefully.
Raise meaningful exceptions to trigger task and flow reversion; the exception which triggered the flow reversion may be persistently stored (and can be referred to later), make sure it is as useful & meaningful as possible.
Be careful with conditions, currently all tasks in a flow will run (unconditional); understand and design ahead of time for this.
- TODO: there will be a blueprint to help with this soon.

Mind-Altering Substances

Using taskflow requires a slight shift in mind-set and changes a little bit of how your normal code would run and how you typically structure your code while programming. The taskflow team has tried to keep the amount of mind-altering required to use taskflow to as much of a minimum as possible (since mind-altering means learning new concepts, or suppressing existing ones) to make it easy to adopt taskflow into your service/application/library. The below are common kinds of mind-blown experiences that may occur when starting to get used to taskflow. The effects may stay with your throughout your life (you have been warned).

Exceptions

Mind-blown: has my exception logic changed, what does it mean if a task throws an exception, who catches it, what happens, wtfbbq???

Exceptions that occur in a task, and which are not caught by the internals of a task will by default currently trigger reversion of the entire workflow that task was in (the engine is responsible for handling this reversion process, as it is also responsible for handling the happy path as well). If multiple tasks in a workflow raise exceptions (say they are executing at the same time via a parallel engine via processes/threads or a distributed engine) then the individual paths that lead to that task will be reverted (if an ancestor task is shared by multiple failing tasks, it will be reverted only once).

NOTE: in the future reversion strategies should be able to make this more customizable (allowing more customizable ways to handle or alter the reversion process so that you can better decide what to do with unhandled exceptions).

Execution flow

Mind-blown: all my tasks belong to engine???

When a set of tasks and associated structure that contains those tasks (aka the flows that create that structure) are given to an engine, along with a possible (but not needed) backend where the engine can store intermediate results (which is needed if the workflow should be able to resume on failure) the engine becomes the execution unit that is responsible for reliably executing the tasks that are contained in the flows that you provide it. That engine will ensure the structure that is provided is retained when executing. For example a linear ordering of tasks by using a linear_flow structure will always be ran in linear order. A set of tasks that are structured in dependency ordering will always be ran in that dependency order. These constraints the engine must adhere to; note other constraints may be enforced by the engine type that is being activated (ie a single threaded engine will only run in a single thread, a distributed or worker based engine may run remotely). So when selecting an engine to use, make sure to carefully select the desired feature set that will work for your application.

Nesting

Mind-blown: without functions (which are now task) how do we model and execute actions that require composition/nesting (no infinite recursion please)???

First, let me describe a little bit why this is hard since it may not be very obvious. In a traditional structure & execution style (without a structured workflow) a function Y may call another function Z and treat what Z does as a blackbox. This type of structure and execution style does not inherently lead to a structure that can be executed by another party (the engine in taskflows case), it also does not easily (without language level features/additions) allow for anyway to resume from the function Z if the program crashes while calling the function Z (and so-on, if Z calls another function this same problem occurs...). This is not to say that carefully designed software can not do this, it just means that they will likely build something like taskflow to solve this problem anyway. To avoid this problem, and enable the features that taskflow creates (resuming, execution control) we need to flip this kind of model on its head (or at least turn it 90 degrees).

The mindshift that taskflow introduces to get around the blackbox problem (Y calling Z, Z calling more functions and so-on) is to change the normal Y->Z structure into a set of dependencies & task inputs and outputs with results being passed between those tasks (in a way similar to message passing). This simple model then allows taskflow (and its engine concept) to be able to restart from a given point by resuming from the last task that has completed. Note that this still makes it difficult to nest tasks. To address this limitation taskflow provides a way to nest tasks and flows. For example, a linear_flow Y' can contain tasks [A, B, C] and then another linear_flow Z' can contain [D, E, Y', F]. This means that the F task listed can depend on all things that Y' (and D, E) have produced before F will start executing (Y' becomes like a blackbox that produces some output, similar in nature to the function Z from above). This kind of composition does not restrict taskflow from resuming, as taskflow internally knows what composes subflows (like Y') and can resume from that nested flow's task if it needs to.

NOTE: coroutines in future versions of python pep-3156 have a similar task like model (not identical, but similar). The issue with coroutines is that they still do not provide the capability to resume, revert or structure your code in a way that maps closely to your actual workflow to be executed. They do though create a base architecture that can be built on to help make this seem easier to accomplish. It is expected that taskflows abstractions should relatively easily map onto python 3.4 which is expected to have a version of pep-3156 once python 3.4 matures.

Control flow

Mind-blown: where did my complex control flow go???

This one is a slight variation in how a programmer normally programs execution control flow. In order to be able to track the execution of your workflow, the desired workflow must be split up into small pieces (in a way similar to functions) ahead of time without a large way to change that execution order at run-time. Taskflow engines using this relatively static structure then can run your structure in a well defined and resumable manner (this relatively static set has been shown to be good enough, by papers & research such as tropic).

This does though currently have a few side-effects in that certain traditional operations (if-then-else, do-while, fork-join, switch...) become more complex in that those types of control flows do not easily map to a representation that can be easily resumed or ran in a potentially distributed manner (since they alter control flow while executing, or create complex and hard to model dependencies between tasks). To keep taskflow relatively minimal (and simple) we have tried to reduce the allowed set to a more manageable and currently smaller set (do the simple things well and add in complexity later, if and when it's needed). If these control flows become valueable then we will revisit if and how we should make these accessible to users of taskflow.

NOTE: inside of a task the execute() method of that task may use whichever existing control flow it desires (any supported by python), but outside of the execute() the set of control flow operators are more minimal (due to the above reasoning/limitations).