TaskFlow/Worker-based Engine

Communication protocol
Worker-based engine communication protocol is described here.

Usage
Let's consider how the  can be applied. First, it is needed to configure  and   to be used with the. After configuration is done,  and   can be started.

Format

 * - broker connection url (see format in kombu documentation);
 * - exchange name in which  /   communication is performed;
 * - topic name under which workers with the same set of tasks are started;
 * - list of tasks worker is able to perform;
 * - can be used to specify the transport explicitly (see available transports);
 * - transport specific options (for instance,  and   parameters for the   transport).

Example with AMQP transport
from taskflow.engines.worker_based import worker as w

config = { 'url': 'amqp://guest:guest@localhost:5672//' 'exchange': 'test-exchange', 'topic': 'test-tasks', 'tasks': ['tasks:TestTask1', 'tasks:TestTask2'] } worker = w.Worker(**config) worker.run

Example filesystem transport
from taskflow.engines.worker_based import worker as w

config = { 'exchange': 'test-exchange', 'topic': 'test-tasks', 'tasks': ['tasks:TestTask1', 'tasks:TestTask2'], 'transport': 'filesystem', 'transport_options': { 'data_folder_in': '/tmp/test', 'data_folder_out': '/tmp/test' } } worker = w.Worker(**config) worker.run

Format

 * - engine type to be used ('worker-based' in our case);
 * - broker connection url (see format in kombu documentation);
 * - exchange name in which  /   communication is performed;
 * - list of workers topics to communicate with;
 * - can be used to specify the transport explicitly (see available transports);
 * - transport specific options (for instance,  and   parameters for the   transport).

Example with AMQP transport
... engine_conf = { 'engine': 'worker-based', 'url': 'amqp://guest:guest@localhost:5672//' 'exchange': 'test-exchange', 'topics': ['topic1', 'topic2'] } flow = lf.Flow('simple-linear').add(...) eng = taskflow.engines.load(flow, engine_conf=engine_conf) eng.run

Example with filesystem transport
... engine_conf = { 'engine': 'worker-based', 'exchange': 'test-exchange', 'topics': ['topic1', 'topic2'], 'transport': 'filesystem', 'transport_options': { 'data_folder_in': '/tmp/test', 'data_folder_out': '/tmp/test' } } flow = lf.Flow('simple-linear').add(...) eng = taskflow.engines.load(flow, engine_conf=engine_conf) eng.run