TaskFlow/StateFlow Pattern

bp: https://blueprints.launchpad.net/taskflow/+spec/stateflow-pattern

StateFlow pattern is yet another idea how to implement conditional execution in TaskFlow.

Operational description
As any pattern, StateFlow combines several tasks and/or other patterns into new flow. Each subflow (be it other pattern or task) of StateFlow is considered to be a state (like in state machine). Only one "state" is active (has running tasks) at any moment when StateFlow is executed. When subflow is finished, a desision what to do next is made based on its results: should we switch to another "state" (execute another subflow), consider this StateFlow executed or revert everything.

More precisely, when "state" finishes, it emits an event. Author of the flow can bind any result subflow provides to be the event. The result should be a string. There also can be a default catch-all transition.

Simple example
flow = StateFlow('get user') welcome = WelcomeTask lookup = FindExistingUserTask create = make_create_user_flow meta = AddUserMetadataTask` save = SaveUserTask

flow.add(welcome, lookup, create, save) flow.initial_state(welcome)

flow.on(welcome, 'SUCCESS', lookup) flow.on(lookup, 'SUCCESS', flow.FINISH) flow.on(lookup, 'FAILURE', create)

flow.set_event_variable(create, 'create_event') flow.on(create, 'SUCCESS', save) flow.on(create, 'NEEDS_META', meta)

flow.on(meta, 'SUCCESS', save) flow.on(save, 'SUCCESS', flow.FINISH)

When flow is executed, first we execute WelcomeTask. When it succeeds, we execute FindExistingUserTask (variable named `lookup`). If lookup is executed successfully, we found existing user, so we finish the flow. If lookup failed, we should create new user, using flow created by make_create_user_flow factory function. This flow provides result named 'create_event' which contains next event name. If that result (in storage) is string 'SUCCESS' after create is executed, we run SaveUserTask and finish flow; otherwise, if it is string 'NEEDS_META', we run AddUserMetadataTask, then SaveUserTask, then we finished.

No cycles
While many workflow engines permit cycles, we don't; StateFlow checks that there is no series of events that leads to cycles, and subflows always form DAG.

Requires and provides
Because states form DAG there is finite number of paths to go from StateFlow start to finish. For each of these paths set of provided values MUST be the same; StateFlow ensures that.

Implementation idea
Before and after each subflow special atoms are added; thus, each transition is about selecting one edge of several edges that go out of state node. That will require modification of graph analyzer.

I need cycles! What should I do?
Most workflow engines allow cycles in the states, but we don't want to go that way. We fill that having cycles in execution graph is a road to hell.

Cycles in workflows where I've seen them have two most popular use cases: - retry on failure - repeat some subflow for each element of a collection

In TaskFlow, first case is solved by retry strategies. Second case is interesting and should be done as another pattern -- we still considering how that should be done.

I need complex conditions! What should I do?
For complex conditions, even with expression evaluations and everything, my answer is simple: use tasks. In TaskFlow, when we have a piece of work, we wrap it in task. Compute your complex condition, represent result as a string and let TaskFlow do the rest.

And what about expression evaluation?
In Python, we use Python for expression evaluations (we heard it's extremely good in it). You can also write any other expression evaluator in Python or interface it from Python. Wrap it in task, and it will work beautifully. We feel it should not be part of TaskFlow.

And what about rainbow ponies?
Not in this release, sorry. Do you... want a hug?

What if I want to run two subflows in parallel?
Non-determenistic automata is interesting idea. We may allow to take two transitions to one state simultaneously and execute tasks in parallel if engine supports that. But that will raise questions of joins and things like that. So, we may one day go this way and enhance StateFlow pattern to allow that, but not in first release for sure.

Of course, you can always wrap your two state in pattern that supports parallel execution (unordered flow or graph flow) put it into StateFlow as single state. That will give you more or less what you want, up to the point that makes all this idea not worth to be implemented.