Difference between revisions of "TaskFlow/Engines"

Revision as of 06:25, 14 September 2013

Engine

Engines are what really runs your tasks and flows.

An engine takes a flow structure (described by patterns) and uses it to decide which task to run and when.

There may be different implementation of engines. Some may be easier to use (like, require no setup) and understand, others might require more complicated setup but provide better scalability. The idea and ideal is that deployers of a service that uses taskflow can select an engine that suites their setup best without modifying the code of said service. This allows for that deployer to start off using a simpler implementation and scaling out the service that is powered by taskflow as the service grows. In concept, all engines should implement the same interface to make it easy to replace one engine with another, and provide the same guarantees on how patterns are interpreted -- for example, if an engine runs a linear flow, the tasks should be run one after another in order no matter what type of engine is actually running that linear flow.

Note: Engines might have different capabilities and different configuration but overall the interface will remain the same.

Supported Types

Distributed

When you want your applications tasks and flows to be performed in a system that is highly available & resilient to individual failure.

See: DistributedTaskManagement and Celery for more details.

Demo: http://www.youtube.com/watch?v=SJLc3U-KYxQ

Traditional

When you want your tasks and flows to just run inside your applications existing framework and still take advantage of the functionality offered.

Supports the following:

Threaded engine using a provided thread/processor executor.
Single threaded engine using no threads.

How

Blueprint: https://blueprints.launchpad.net/taskflow/+spec/patterns-and-engines

Blueprint: https://blueprints.launchpad.net/taskflow/+spec/distributed-celery

Blueprint: https://blueprints.launchpad.net/taskflow/+spec/eventlet-engine

Storage

Storage is out of scope of the blueprint, but it is still worth to point out its role here.

We already have storage in taskflow -- that's logbook. But it should be emphasized that logbook should become the authoritative, and, preferably, the only source of runtime state information. When task returns result, it should be written directly to logbook. When task or flow state changes in any way, logbook is first to know. Flow should not store task results -- there is logbook for that.

Logbook and a backend are responsible to store the actual data -- these together specify the persistence mechanism (how data is saved and where -- memory, database, whatever), and persistence policy (when data is saved -- every time it changes or at some particular moments or simply never).

@@ Line 5: / Line 5: @@
 [[File:Felipecaparelli_Gears_1.png|frame]]
-It takes a <code>flow</code> structure (described by patterns) and uses it to decide which <code>task</code> to run and when.
+An engine takes a <code>flow</code> structure (described by patterns) and uses it to decide which <code>task</code> to run and when.
 There may be different implementation of engines. Some may be easier to use (like, require no setup) and understand, others might require more complicated setup but provide better scalability. The idea and ''ideal'' is that deployers of a service that uses taskflow can select an engine that suites their setup best without modifying the code of said service. This allows for that deployer to start off using a simpler  implementation and scaling out the service that is powered by taskflow as the service grows. In concept, all engines should implement the same interface to make it easy to replace one engine with another, and provide the same guarantees on how patterns are interpreted -- for example, if an engine runs a linear flow, the tasks should be run one after another in order no matter what type of engine is actually running that linear flow.