Difference between revisions of "Zaqar/specs/sharding/v1"
(→Update a Shard Entry) |
(→Register a Shard) |
||
| Line 59: | Line 59: | ||
"uri": "mongodb://localhost:27017", | "uri": "mongodb://localhost:27017", | ||
"options": { | "options": { | ||
| − | "max_retry_sleep": 1 | + | "max_retry_sleep": 1, |
| + | "partitions": 8 | ||
} | } | ||
} | } | ||
| Line 82: | Line 83: | ||
<code><nowiki>uri</nowiki></code> is a connection string compatible with a storage client (e.g., pymongo) attempting to connect to that shard. | <code><nowiki>uri</nowiki></code> is a connection string compatible with a storage client (e.g., pymongo) attempting to connect to that shard. | ||
| − | <code><nowiki>options</nowiki></code> An optional request component that gives storage-specific options used by storage driver implementations. | + | |
| + | <code><nowiki>options</nowiki></code> An optional request component that gives storage-specific options used by storage driver implementations. Valid parameters come from the registered options for a given storage backend, for example: [https://github.com/openstack/marconi/blob/master/marconi/queues/storage/mongodb/options.py mongodb], [https://github.com/openstack/marconi/blob/master/marconi/queues/storage/sqlite/driver.py#L29 sqlite] | ||
=== Read a Shard === | === Read a Shard === | ||
Revision as of 15:25, 5 November 2013
Contents
Marconi Storage Sharding: v1 Blueprint
Overview
This proposal introduces another approach to tackle making Marconi highly scalable without losing any of its flexibility. This particular document covers the administrative API for registering storage shards.
Reference
For authoritative header/general details, please refer to the Marconi Queues API v1 Spec.
Endpoints Synopsis
GET /shards?detailed=False&marker=None&limit=10
GET /shards/{shard}?detailed=False
PUT /shards/{shard}
DELETE /shards/{shard}
PATCH /shards/{shard}
GET /health
Sharding with Marconi
Marconi supports heterogeneous sharding through the use of:
- stevedore for dynamic storage driver loading
- node weights
As long as an entry point is defined in an installed module that matches the scheme of a shard connection URI, then Marconi will be able to use that shard. For example, a shard entry might look like:
{
"weight": 100,
"uri": "mongodb://localhost:27017",
"options": {
"max_retry_sleep": 1
}
}
Register a Shard
Template
PUT /v1/shards/{shard}
Request
PUT /v1/shards/wat HTTP/1.1
Host: marconi.example.com
{
"weight": 100,
"uri": "mongodb://localhost:27017",
"options": {
"max_retry_sleep": 1,
"partitions": 8
}
}
Response
HTTP/1.1 201 Created Location: /v1/shards/wat
Discussion
Registers a shard.
shard is the name to give the shard entry. The name MUST NOT exceed 64 bytes in length, and is limited to US-ASCII letters, digits, underscores and hyphens.
weight is the likelihood that this shard will be selected for the next queue allocation. It must be an integer greater than -1.
uri is a connection string compatible with a storage client (e.g., pymongo) attempting to connect to that shard.
options An optional request component that gives storage-specific options used by storage driver implementations. Valid parameters come from the registered options for a given storage backend, for example: mongodb, sqlite
Read a Shard
Template
GET /v1/shards/{shard}?detailed=True
Request
GET /v1/shards/wat HTTP/1.1 Host: marconi.example.com
Response
HTTP/1.1 200 OK
Content-Location: /v1/shards/wat
{
"uri": "mongodb://marconi1.example.com:27017",
"weight": 100
}
Discussion
Returns information on a registered shard.
Delete a Shard
Template
DELETE /v1/shard/{shard}
Request
DELETE /v1/shards/wat HTTP/1.1 Host: marconi.example.com
Response
HTTP/1.1 204 No Content
Discussion
Removes a shard from the registry.
Update a Shard
Template
PATCH /v1/shards/{shard}
Request
PATCH /v1/shards/wat HTTP/1.1
Host: marconi.example.com
{
"uri": "mongodb://marconi3.example.com:27018",
"weight": 120
}
Response
HTTP/1.1 204 No Content
Discussion
Allows one to update any or all of: `weight`, `uri`, `options`. At least one of these fields must be specified, else, a HTTP 400 is returned.
List Shards
Template
GET /v1/shards?detailed=True&limit=10&marker=taco
Request
GET /v1/shards HTTP/1.1 Host: marconi.proxy.example.com
Response
HTTP/1.1 200 OK
Content-Location: /v1/shards
[
{"href": "/v1/shards/wat", "weight": 100, "uri": "mongodb://marconi1.example.com:27017"},
{"href": "/v1/shards/wot", "weight": 50, "uri": "redis://marconi2.example.com:6379"}
]
Discussion
Lists the registered shards.
detailed if True, returns the options field in the listing.
marker used for pagination - what shard do we start listing from?
limit how many entries to return per request?
