Zaqar/specs/sharding/v1
Contents
Marconi Storage Sharding: v1 Blueprint
Overview
This proposal introduces another approach to tackle making Marconi highly scalable without losing any of its flexibility. This particular document covers the administrative API for registering storage shards.
Reference
For authoritative header/general details, please refer to the Marconi Queues API v1 Spec.
Endpoints Synopsis
GET /shards?detailed=False&marker=None&limit=10 GET /shards/{shard}?detailed=False PUT /shards/{shard} DELETE /shards/{shard} PATCH /shards/{shard} GET /health
Sharding with Marconi
Marconi supports heterogeneous sharding through the use of:
- stevedore for dynamic storage driver loading
- node weights
As long as an entry point is defined in an installed module that matches the scheme of a shard connection URI, then Marconi will be able to use that shard. For example, a shard entry might look like:
{ "weight": 100, "uri": "mongodb://localhost:27017", "options": { "max_retry_sleep": 1 } }
Register a Shard
Template
PUT /v1/shards/{shard}
Request
PUT /v1/shards/wat HTTP/1.1 Host: marconi.example.com { "weight": 100, "uri": "mongodb://localhost:27017", "options": { "max_retry_sleep": 1 } }
Response
HTTP/1.1 201 Created Location: /v1/shards/wat
Discussion
Registers a shard.
shard
is the name to give the shard entry. The name MUST NOT exceed 64 bytes in length, and is limited to US-ASCII letters, digits, underscores and hyphens.
weight
is the likelihood that this shard will be selected for the next queue allocation. It must be an integer greater than -1.
uri
is a connection string compatible with a storage client (e.g., pymongo) attempting to connect to that shard.
options
An optional request component that gives storage-specific options used by storage driver implementations.
Read a Shard
Template
GET /v1/shards/{shard}?detailed=True
Request
GET /v1/shards/wat HTTP/1.1 Host: marconi.example.com
Response
HTTP/1.1 200 OK Content-Location: /v1/shards/wat { "uri": "mongodb://marconi1.example.com:27017", "weight": 100 }
Discussion
Returns information on a registered shard.
Delete a Shard
Template
DELETE /v1/shard/{shard}
Request
DELETE /v1/shards/wat HTTP/1.1 Host: marconi.example.com
Response
HTTP/1.1 204 No Content
Discussion
Removes a shard from the registry.
List Shards
Template
GET /v1/shards?detailed=True&limit=10&marker=taco
Request
GET /v1/shards HTTP/1.1 Host: marconi.proxy.example.com
Response
HTTP/1.1 200 OK Content-Location: /v1/shards [ {"href": "/v1/shards/wat", "weight": 100, "uri": mongodb://marconi1.example.com:27017"}, {"href": "/v1/shards/wot", "weight": 50, "uri": redis://marconi2.example.com:6379"} ]
Discussion
Lists the registered shards.
detailed
if True, returns the options field in the listing.
marker
used for pagination - what shard do we start listing from?
limit
how many entries to return per request?