Difference between revisions of "TaskFlow/Engines"
| Line 1: | Line 1: | ||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
== Engine == | == Engine == | ||
Engine is what really runs the tasks. It should take flow structure (described by patterns) and use it to decide which task to run and when. | Engine is what really runs the tasks. It should take flow structure (described by patterns) and use it to decide which task to run and when. | ||
| − | There may be different implementation of engines. Some may be | + | There may be different implementation of engines. Some may be easier to use (like, require no setup) and understand, others might require more complicated setup but provide better scalability. The idea and ideal is that deployers of a service that uses taskflow can select an engine that suites their setup best without modifying code of said service. This allows for starting off using a simpler implementation and scaling out the service that is powered by taskflow as the service grows. |
| − | + | In concept, all engines should implement same interface to make it easy to replace one engine with another, and provide same guaranties on how patterns are interpreted -- for example, if an engine runs a linear flow, the tasks should be run one after another in order. | |
Possible engines include: | Possible engines include: | ||
| − | * | + | * Simple -- just takes e.g. linear flow and runs tasks from it one after another -- should be useful for debugging tasks and simple use cases; |
| − | * | + | * Threaded -- Runs tasks in separate threads enabling them to run in parallel (even several implementations are possible); |
| − | + | * Distributed -- loads tasks to celery (or some other external service) that uses tasks dependencies to determine ordering; | |
| − | * | ||
| − | Engines might have different capabilities | + | Engines might have different capabilities and different configuration but overall the interface should remain the same. |
== How == | == How == | ||
| Line 72: | Line 22: | ||
Storage is out of scope of [https://blueprints.launchpad.net/taskflow/+spec/patterns-and-engines the blueprint], but it is still worth to point out its role here. | Storage is out of scope of [https://blueprints.launchpad.net/taskflow/+spec/patterns-and-engines the blueprint], but it is still worth to point out its role here. | ||
| − | We already have storage in taskflow -- that's logbook. But it should be | + | We already have storage in taskflow -- that's logbook. But it should be emphasized that logbook should become the authoritative, and, preferably, the '''only''' source of runtime state |
| − | emphasized that logbook should become the authoritative, and, | + | information. When task returns result, it should be written directly to logbook. When task or flow state changes in any way, logbook is first to know. Flow should '''not''' store task results -- there is logbook for that. |
| − | preferably, the '''only''' source of runtime state information. When | ||
| − | task returns result, it should be written directly to logbook. When task | ||
| − | or flow state changes in any way, logbook is first to know. Flow | ||
| − | should '''not''' store task results -- there is logbook for that. | ||
| − | Logbook | + | Logbook and a backend are responsible to store the actual data -- these together specify the persistence mechanism (how data is saved and where -- memory, database, |
| − | persistence mechanism (how data is saved and where -- memory, database, | + | whatever), and persistence policy (when data is saved -- every time it changes or at some particular moments or simply never). |
| − | whatever), and persistence policy (when data is saved -- every time it | ||
| − | changes or at some particular moments or simply never). | ||
Revision as of 05:48, 14 September 2013
Engine
Engine is what really runs the tasks. It should take flow structure (described by patterns) and use it to decide which task to run and when.
There may be different implementation of engines. Some may be easier to use (like, require no setup) and understand, others might require more complicated setup but provide better scalability. The idea and ideal is that deployers of a service that uses taskflow can select an engine that suites their setup best without modifying code of said service. This allows for starting off using a simpler implementation and scaling out the service that is powered by taskflow as the service grows.
In concept, all engines should implement same interface to make it easy to replace one engine with another, and provide same guaranties on how patterns are interpreted -- for example, if an engine runs a linear flow, the tasks should be run one after another in order.
Possible engines include:
- Simple -- just takes e.g. linear flow and runs tasks from it one after another -- should be useful for debugging tasks and simple use cases;
- Threaded -- Runs tasks in separate threads enabling them to run in parallel (even several implementations are possible);
- Distributed -- loads tasks to celery (or some other external service) that uses tasks dependencies to determine ordering;
Engines might have different capabilities and different configuration but overall the interface should remain the same.
How
Blueprint: https://blueprints.launchpad.net/taskflow/+spec/patterns-and-engines
Storage
Storage is out of scope of the blueprint, but it is still worth to point out its role here.
We already have storage in taskflow -- that's logbook. But it should be emphasized that logbook should become the authoritative, and, preferably, the only source of runtime state information. When task returns result, it should be written directly to logbook. When task or flow state changes in any way, logbook is first to know. Flow should not store task results -- there is logbook for that.
Logbook and a backend are responsible to store the actual data -- these together specify the persistence mechanism (how data is saved and where -- memory, database, whatever), and persistence policy (when data is saved -- every time it changes or at some particular moments or simply never).
