TaskFlow/Worker-based Engine Protocol
Contents
Worker-based engine protocol
Worker-based flow
Worker-based flow use remote workers to perform high load, computational and time-consuming tasks. There are two communication components - Proxy and Worker.
Proxy
Proxy is a part of the worker-based engine. It is used to publish task requests, so these requests can be accepted and processed by remote workers.
Worker
Worker is started on remote host and has list of tasks it can perform. Worker accepts and processes task requests that are published by proxy. Several requests can be processed simultaneously in separate threads. Workers that can process same set of tasks are grouped under one topic.
Protocol
Proxy and Workers communicate in one common named exchange. Proxy publishes task requests to the named exchange and Workers take requests from it. When Worker gets task request it parses all parameters, dispatches endpoint and starts task processing. During task processing Worker sends responses back to Proxy.
Proxy request has the following format
- task - full task name to be performed
- action - task action to be performed (e.g. execute, revert)
- arguments - arguments the task action to be called with
- result - task execution result (result or misc.Failure) [passed to revert only]
- failures - flow tasks failures map (map of tasks names and misc.Failures) [passed to revert only]
Example
{'task': 'tasks.CallJoe', 'action': 'execute', 'arguments': {'joe_number': 444}}
Additionally, the following parameters are added to the AMQP request message:
- reply_to - Proxy named exchange workers will send responses back to
- correlation_id - Proxy request id (since there can be multiple request being processed simultaneously)
Worker response has the following format
- Task accepted:
{'status': 'RUNNING'} - Task progress:
{'status': 'PROGRESS', 'data': (event, progress)} - Task succeeded:
{'status': 'SUCCESS', 'data': (event, result)} - Task failed:
{'status': 'FAILURE', 'data': (None, misc.Failure)}
