Distributed Task Management With RPC

Distributed task management built on RPC can be easily integrated with OpenStack. RPC allows to build reliable and scalable system. This document proposes an architecture of distributed flow that can run tasks simultaneously on multiple workers. The main goal is to provide a such architecture that will allow to replace a simple flow with a distributed flow without changing a code.

Architecture

Client is a machine that runs a distributed flow. Worker is a server that executes distributed flows’ tasks. Distributed task is a flow task that performs a remote procedure call. Remote task is a task that runs on the worker side and executes some code to make a flow progress.

The flow runs a task that makes a remote call using a RPC Client on client side. RPC Client calls a remote task and passes task’s arguments and client’s endpoint. The worker accepts this task and returns confirmation to the Client. Then the worker launches remote task and periodically sends its status to the Client. The Client doesn’t know is a worker still alive, so it listens statuses from the Worker. When the remote task is finished, the Worker sends a result to the Client. The Сlient considers Worker as failed if it hadn’t been receiving a task status message for a timeout period.