Difference between revisions of "Swift/ideas/task-execution"

Latest revision as of 21:32, 22 February 2017

Goal

Scaling tasks execution to where data are (object-expirer, container-sync, auto-tiering, metadata-indexation, ...).

Task Registration

Tasks is registered in containers of special accounts by the item-server. There is one account per couple (item, partition), so that each item-servers request only tasks for data they are holding. Containers names are feature dependents:

.tasks-objects-42
  object-expirer-<timestamp>
  auto-tiering-<timestamp>
  metadata-indexation

Task execution

Each item-server fetch the tasks for the partition it is holding and execute them:

• account-server fetch tasks from .tasks-account-<part number>
• container-server fetch tasks from .tasks-container-<part number>
• object-server fetch tasks from .tasks-object-<part number>

Back filling

For feature that are enabled at account or container level (eg: metadata indexation), a task for backfilling is done at feature activation. eg: for metadata indexation, it creates a container task that ask for the creation of metadata-indexation tasks for rows N to M:

.tasks-container-<container partition>
   metadata-indexation
       account/container/start_row-end_row

Once executed, we get a bunch of tasks:

.tasks-object-<object partition>
   metadata-indexation
       account/container/object/timestamp

Same applies to container-sync.