TaskFlow/Worker-based Engine Protocol

Worker-based engine
uses remote workers to perform high load, computational and time-consuming tasks. There are two communication sides -  and   that communicate using   component. is designed to accept/publish messages from/into named exchange.

Executor
is a part of the worker-based engine and is used to publish task requests, so these requests can be accepted and processed by remote.

Worker
is started on remote host and has list of tasks it can perform. accepts and processes task requests that are published by. Several requests can be processed simultaneously in separate threads (executor can be passed to the  and configured). Workers that can process same set of tasks can be grouped under common.

Protocol
and  communicate in one common named exchange. publishes task requests to the named exchange and  take requests from it. When  gets task request it parses all parameters, dispatches endpoint and starts task processing. During task processing  sends responses back to.

Getting topic tasks list by Executor
Initially  has list of worker topics, but does not have list of topic tasks. So, when  is started it begins to send the   messages to all topics. Once topic (actually topic worker) receives the  message it responses back with its   name and   list:

{'topic': 'test-topic', 'tasks': [u'taskflow.tests.utils.TaskOneArgOneReturn', u'taskflow.tests.utils.TaskMultiArgOneReturn']}

Once  received list of tasks it publishes all cached (waiting) requests to the proper topic, so topic worker will start task execution.

Executor and Worker communication
Let's consider how communication between  and   happens. First of all engine resolves all tasks dependencies and schedules tasks that can be performed at the moment. Tasks are executed by worker-based engine. initiates task execution/reversion using.
 * publishes task request (format is described below) into named exchange with, that is used to deliver request to particular workers   and waits for task requests to be accepted and confirmed by  . If   doesn't get task confirmation from   within the given timeout task is considered as timed-out and the   exception is raised.
 * receives a request message and starts a new thread for processing it.
 * dispatches request (gets desired endpoint that actually executes task):


 * If request dispatching  Worker sends confirmation response to the Executor:


 * {'state': 'RUNNING'}
 * If request dispatching  Worker sends failure response to the   with   object:
 * {'state': 'FAILURE', 'result': }


 * gets task request confirmation from the  and task request state changed from the   to the   state. Once task request is in the   state it can't be timed-out (considering that task execution process may take unpredictable time).
 * executes task and once it is finished sends result back to the :
 * If result is of the  type response with the failure state is sent back:
 * {'state': 'FAILURE', 'event':, 'result': )}
 * If result is not of the  type response with the success state is sent back:
 * {'state': 'SUCCESS', 'event':, 'result': )}


 * gets task execution result from  and passes it back to the WorkerTaskExecutor and worker-based engine to finish task processing.
 * is subscribed on the task progress and every time task progress event is triggered it sends progress notification to the  where it is handled by engine:
 * {'state': 'PROGRESS', 'event_data': , 'progress': }

Executor request format

 * task - full task name to be performed
 * action - task action to be performed (e.g. execute, revert)
 * arguments - arguments the task action to be called with
 * result - task execution result (result or misc.Failure) [passed to revert only]
 * failures - flow tasks failures map (map of tasks names and misc.Failures) [passed to revert only]

Example
{'task': 'tasks.CallJoe', 'action': 'execute', 'arguments': {'joe_number': 444}}

Additionally, the following parameters are added to the AMQP request message:
 * reply_to - Executor named exchange workers will send responses back to
 * correlation_id - Executor request id (since there can be multiple request being processed simultaneously)

Worker response format
{'status': 'RUNNING'} {'state': 'PROGRESS', 'event_data': , 'progress': } {'state': 'SUCCESS', 'event':, 'result': )} {'state': 'FAILURE', 'event': , 'result': )}
 * Task running:
 * Task progress:
 * Task succeeded:
 * Task failed: