TaskFlow/Patterns and Engines/Persistence
How can we persist the flow? Here is informal description.
We Need the Flow
How do we upgrade? We change the code that creates the flow, then we restart the service.
Then, when flow is restored from storage, what we really want is to load new flow, with updated structure and updated tasks, but preserve task states and results. At least, that is most common case.
So, the code should be run to put the tasks into patterns and re-create the flow, but logbook should be considered to load state and results of tasks.
Flow Factory
Creation of the flow should be put into separate function.
The pattern is already used, look for example at [cinder code].
The thing is that we should be able to call this function again after service restart. For that the simplest way is to save fully qualified name of the function with flow details. Then, the function can be imported by saved name and called.
Discuss: should we allow classes with __call__ method in addition to functions? How do we instantiate them?
Discuss: can flow factories have any arguments? What are the restrictions if they can?
Loading Flow into Engine
The engine gets a flow and flow details, and should reconstruct its internal state.
A task should be somehow matched with TaskDetails. The match should be:
- stable if tasks are added or removed;
- should not change when service is restarted, upgraded;
- should be the same across all server instances in HA setups.
One option is that tasks shuld be matched with TaskDetails by task name. This has several implications:
- the names of tasks should be unique in flow;
- it becomes too hard to change the name of task;
- this is not cool.
Discuss: I don't think it's really problems.
Another option is adding some kind of task id that defaults to utils.get_callable_name() but can be specified explicitly if desired.
Discuss: Alternatives?
If no TaskDetail exists for task, it should be created and task is put to default state (states.PENDING).
If TaskDetail for task exists, task state and result should be loaded from it.
Discuss: What if versions don't match?
When task states are loaded, the engine can run a flow from the point it stopped. For ActionEngine, it can just simulate running flow from the start like taskflow.patterns.LinearFlow does now.
