Swift/ideas/task-execution
Goal
Scaling tasks execution to where data are (object-expirer, container-sync, auto-tiering, metadata-indexation, ...).
Task Registration
Tasks is registered in containers of special accounts by the item-server. There is one account per couple (item, partition), so that each item-servers request only tasks for data they are holding. Containers names are feature dependents:
.tasks-objects-42 object-expirer-<timestamp> auto-tiering-<timestamp> metadata-indexation
Task execution
Each item-server fetch the tasks for the partition it is holding and execute them:
- account-server fetch tasks from .tasks-account-<part number>
- container-server fetch tasks from .tasks-container-<part number>
- object-server fetch tasks from .tasks-object-<part number>
Back filling
For feature that are enabled at account or container level (eg: metadata indexation), a task for backfilling is done at feature activation. eg: for metadata indexation, it creates a container task that ask for the creation of metadata-indexation tasks for rows N to M:
.tasks-container-<container partition> metadata-indexation account/container/start_row-end_row
Once executed, we get a bunch of tasks:
.tasks-object-<object partition> metadata-indexation account/container/object/timestamp
Same applies to container-sync.