Jump to: navigation, search

Difference between revisions of "Zaqar/specs/sharding/v1"

(PATCH)
(Replace shards for pools and rename Marconi to Zaqar)
 
(3 intermediate revisions by 2 users not shown)
Line 1: Line 1:
−
= Marconi Storage Sharding: v1 Blueprint =
+
= Zaqar Storage Pooling: v1 Blueprint =
  
 
== Overview ==
 
== Overview ==
  
−
This proposal introduces another approach to tackle making Marconi highly scalable without losing any of its flexibility. This particular document covers the administrative API for registering storage shards.
+
This proposal introduces another approach to tackle making Zaqar highly scalable without losing any of its flexibility. This particular document covers the administrative API for registering storage pools.
  
 
== Reference ==
 
== Reference ==
  
−
For authoritative header/general details, please refer to the [https://wiki.openstack.org/wiki/Marconi/specs/api/v1 Marconi Queues API v1 Spec].
+
For authoritative header/general details, please refer to the [https://wiki.openstack.org/wiki/Zaqar/specs/api/v1 Zaqar Queues API v1 Spec].
  
−
== Endpoints Synopsis ==
+
== Endpoints synopsis ==
  
 
<pre><nowiki>
 
<pre><nowiki>
−
GET /shards?detailed=False&marker=None&limit=10
+
GET /pools?detailed=False&marker=None&limit=10
−
GET /shards/{shard}?detailed=False
+
GET /pools/{pool}?detailed=False
  
−
PUT /shards/{shard}
+
PUT /pools/{pool}
−
DELETE /shards/{shard}
+
DELETE /pools/{pool}
−
PATCH /shards/{shard}
+
PATCH /pools/{pool}
  
 
GET /health
 
GET /health
 
</nowiki></pre>
 
</nowiki></pre>
  
−
== Sharding with Marconi ==
+
== Pooling with Zaqar ==
  
−
Marconi supports heterogeneous sharding through the use of:
+
Zaqar supports heterogeneous pooling through the use of:
  
 
* stevedore for dynamic storage driver loading
 
* stevedore for dynamic storage driver loading
 
* node weights
 
* node weights
  
−
As long as an entry point is defined in an installed module that matches the scheme of a shard connection URI, then Marconi will be able to use that shard. For example, a shard entry might look like:
+
As long as an entry point is defined in an installed module that matches the scheme of a pool connection URI, then Zaqar will be able to use that pool. For example, a pool entry might look like:
  
 
<pre><nowiki>
 
<pre><nowiki>
Line 41: Line 41:
 
</nowiki></pre>
 
</nowiki></pre>
  
−
=== Register a Shard ===
+
=== Register a pool ===
  
 
'''Template'''
 
'''Template'''
  
 
<pre><nowiki>
 
<pre><nowiki>
−
PUT /v1/shards/{shard}
+
PUT /v1/pools/{pool}
 
</nowiki></pre>
 
</nowiki></pre>
  
Line 52: Line 52:
  
 
<pre><nowiki>
 
<pre><nowiki>
−
PUT /v1/shards/wat HTTP/1.1
+
PUT /v1/pools/wat HTTP/1.1
−
Host: marconi.example.com
+
Host: zaqar.example.com
  
 
{
 
{
Line 59: Line 59:
 
   "uri": "mongodb://localhost:27017",
 
   "uri": "mongodb://localhost:27017",
 
   "options": {
 
   "options": {
−
     "max_retry_sleep": 1
+
     "max_retry_sleep": 1,
 +
    "partitions": 8
 
   }
 
   }
 
}
 
}
Line 69: Line 70:
 
<pre><nowiki>
 
<pre><nowiki>
 
HTTP/1.1 201 Created
 
HTTP/1.1 201 Created
−
Location: /v1/shards/wat
+
Location: /v1/pools/wat
 
</nowiki></pre>
 
</nowiki></pre>
  
Line 75: Line 76:
 
'''Discussion'''
 
'''Discussion'''
  
−
Registers a shard.
+
Register a pool.
 +
 
 +
<code><nowiki>pool</nowiki></code> is the name to give the pool entry. The name MUST NOT exceed 64 bytes in length, and is limited to US-ASCII letters, digits, underscores and hyphens.
  
−
<code><nowiki>shard</nowiki></code> is the name to give the shard entry. The name MUST NOT exceed 64 bytes in length, and is limited to US-ASCII letters, digits, underscores and hyphens.
+
<code><nowiki>weight</nowiki></code> is the likelihood that this pool will be selected for the next queue allocation. It must be an integer greater than -1.
  
−
<code><nowiki>weight</nowiki></code> is the likelihood that this shard will be selected for the next queue allocation. It must be an integer greater than -1.
+
<code><nowiki>uri</nowiki></code> is a connection string compatible with a storage client (e.g., pymongo) attempting to connect to that pool.
  
−
<code><nowiki>uri</nowiki></code> is a connection string compatible with a storage client (e.g., pymongo) attempting to connect to that shard.
+
<code><nowiki>options</nowiki></code> An optional request component that gives storage-specific options used by storage driver implementations. Valid parameters come from the registered options for a given storage backend, for example: [https://github.com/openstack/marconi/blob/master/marconi/queues/storage/mongodb/options.py mongodb], [https://github.com/openstack/marconi/blob/master/marconi/queues/storage/sqlite/driver.py#L29 sqlite]
−
<code><nowiki>options</nowiki></code> An optional request component that gives storage-specific options used by storage driver implementations.
 
  
−
=== Read a Shard ===
+
=== Read a pool ===
  
 
'''Template'''
 
'''Template'''
  
 
<pre><nowiki>
 
<pre><nowiki>
−
GET /v1/shards/{shard}?detailed=True
+
GET /v1/pools/{pool}?detailed=True
 
</nowiki></pre>
 
</nowiki></pre>
  
Line 95: Line 97:
  
 
<pre><nowiki>
 
<pre><nowiki>
−
GET /v1/shards/wat HTTP/1.1
+
GET /v1/pools/wat HTTP/1.1
−
Host: marconi.example.com
+
Host: zaqar.example.com
 
</nowiki></pre>
 
</nowiki></pre>
  
Line 104: Line 106:
 
<pre><nowiki>
 
<pre><nowiki>
 
HTTP/1.1 200 OK
 
HTTP/1.1 200 OK
−
Content-Location: /v1/shards/wat
+
Content-Location: /v1/pools/wat
  
 
{
 
{
−
   "uri": "mongodb://marconi1.example.com:27017",
+
   "uri": "mongodb://zaqar1.example.com:27017",
 
   "weight": 100
 
   "weight": 100
 
}
 
}
Line 115: Line 117:
 
'''Discussion'''
 
'''Discussion'''
  
−
Returns information on a registered shard.
+
Returns information on a registered pool.
  
−
=== Delete a Shard ===
+
=== Delete a pool ===
  
 
'''Template'''
 
'''Template'''
  
 
<pre><nowiki>
 
<pre><nowiki>
−
DELETE /v1/shard/{shard}
+
DELETE /v1/pool/{pool}
 
</nowiki></pre>
 
</nowiki></pre>
  
Line 128: Line 130:
  
 
<pre><nowiki>
 
<pre><nowiki>
−
DELETE /v1/shards/wat HTTP/1.1
+
DELETE /v1/pools/wat HTTP/1.1
−
Host: marconi.example.com
+
Host: zaqar.example.com
 
</nowiki></pre>
 
</nowiki></pre>
  
Line 141: Line 143:
 
'''Discussion'''
 
'''Discussion'''
  
−
Removes a shard from the registry.
+
Removes a pool from the registry.
  
−
=== Update a Shard Entry ===
+
=== Update a pool ===
  
 
'''Template'''
 
'''Template'''
  
 
<pre><nowiki>
 
<pre><nowiki>
−
PATCH /v1/shards/{shard}
+
PATCH /v1/pools/{pool}
 
</nowiki></pre>
 
</nowiki></pre>
  
Line 154: Line 156:
  
 
<pre><nowiki>
 
<pre><nowiki>
−
PATCH /v1/shards/wat HTTP/1.1
+
PATCH /v1/pools/wat HTTP/1.1
−
Host: marconi.example.com
+
Host: zaqar.example.com
  
 
{
 
{
−
   "uri": "mongodb://marconi3.example.com:27018",
+
   "uri": "mongodb://zaqar3.example.com:27018",
 
   "weight": 120
 
   "weight": 120
 
}
 
}
Line 168: Line 170:
 
<pre><nowiki>
 
<pre><nowiki>
 
HTTP/1.1 204 No Content
 
HTTP/1.1 204 No Content
−
 
 
</nowiki></pre>
 
</nowiki></pre>
  
Line 175: Line 176:
 
Allows one to update any or all of: `weight`, `uri`, `options`. At least one of these fields must be specified, else, a HTTP 400 is returned.
 
Allows one to update any or all of: `weight`, `uri`, `options`. At least one of these fields must be specified, else, a HTTP 400 is returned.
  
−
 
+
=== List pools ===
−
=== List Shards ===
 
  
 
'''Template'''
 
'''Template'''
  
 
<pre><nowiki>
 
<pre><nowiki>
−
GET /v1/shards?detailed=True&limit=10&marker=taco
+
GET /v1/pools?detailed=True&limit=10&marker=taco
 
</nowiki></pre>
 
</nowiki></pre>
  
Line 187: Line 187:
  
 
<pre><nowiki>
 
<pre><nowiki>
−
GET /v1/shards HTTP/1.1
+
GET /v1/pools HTTP/1.1
−
Host: marconi.proxy.example.com
+
Host: zaqar.proxy.example.com
 
</nowiki></pre>
 
</nowiki></pre>
  
Line 196: Line 196:
 
<pre><nowiki>
 
<pre><nowiki>
 
HTTP/1.1 200 OK
 
HTTP/1.1 200 OK
−
Content-Location: /v1/shards
+
Content-Location: /v1/pools
  
 
[
 
[
−
   {"href": "/v1/shards/wat", "weight": 100, "uri": "mongodb://marconi1.example.com:27017"},
+
   {"href": "/v1/pools/wat", "weight": 100, "uri": "mongodb://zaqar1.example.com:27017"},
−
   {"href": "/v1/shards/wot", "weight": 50, "uri": "redis://marconi2.example.com:6379"}
+
   {"href": "/v1/pools/wot", "weight": 50, "uri": "redis://zaqar2.example.com:6379"}
 
]
 
]
 
</nowiki></pre>
 
</nowiki></pre>
Line 206: Line 206:
 
'''Discussion'''
 
'''Discussion'''
  
−
Lists the registered shards.
+
Lists the registered pools.
  
 
<code><nowiki>detailed</nowiki></code> if True, returns the options field in the listing.
 
<code><nowiki>detailed</nowiki></code> if True, returns the options field in the listing.
−
<code><nowiki>marker</nowiki></code> used for pagination - what shard do we start listing from?
+
<code><nowiki>marker</nowiki></code> used for pagination - what pool do we start listing from?
 
<code><nowiki>limit</nowiki></code> how many entries to return per request?
 
<code><nowiki>limit</nowiki></code> how many entries to return per request?

Latest revision as of 16:41, 21 August 2014

Zaqar Storage Pooling: v1 Blueprint

Overview

This proposal introduces another approach to tackle making Zaqar highly scalable without losing any of its flexibility. This particular document covers the administrative API for registering storage pools.

Reference

For authoritative header/general details, please refer to the Zaqar Queues API v1 Spec.

Endpoints synopsis

GET /pools?detailed=False&marker=None&limit=10
GET /pools/{pool}?detailed=False

PUT /pools/{pool}
DELETE /pools/{pool}
PATCH /pools/{pool}

GET /health

Pooling with Zaqar

Zaqar supports heterogeneous pooling through the use of:

  • stevedore for dynamic storage driver loading
  • node weights

As long as an entry point is defined in an installed module that matches the scheme of a pool connection URI, then Zaqar will be able to use that pool. For example, a pool entry might look like: 

{
  "weight": 100,
  "uri": "mongodb://localhost:27017",
  "options": {
    "max_retry_sleep": 1
  }
}

Register a pool

Template 

PUT /v1/pools/{pool}

Request 

PUT /v1/pools/wat HTTP/1.1
Host: zaqar.example.com

{
  "weight": 100,
  "uri": "mongodb://localhost:27017",
  "options": {
    "max_retry_sleep": 1,
    "partitions": 8
  }
}


Response 

HTTP/1.1 201 Created
Location: /v1/pools/wat


Discussion

Register a pool.

pool is the name to give the pool entry. The name MUST NOT exceed 64 bytes in length, and is limited to US-ASCII letters, digits, underscores and hyphens.

weight is the likelihood that this pool will be selected for the next queue allocation. It must be an integer greater than -1.

uri is a connection string compatible with a storage client (e.g., pymongo) attempting to connect to that pool.

options An optional request component that gives storage-specific options used by storage driver implementations. Valid parameters come from the registered options for a given storage backend, for example: mongodb, sqlite

Read a pool

Template 

GET /v1/pools/{pool}?detailed=True

Request 

GET /v1/pools/wat HTTP/1.1
Host: zaqar.example.com


Response 

HTTP/1.1 200 OK
Content-Location: /v1/pools/wat

{
  "uri": "mongodb://zaqar1.example.com:27017",
  "weight": 100
}


Discussion

Returns information on a registered pool.

Delete a pool

Template 

DELETE /v1/pool/{pool}

Request 

DELETE /v1/pools/wat HTTP/1.1
Host: zaqar.example.com


Response 

HTTP/1.1 204 No Content

Discussion

Removes a pool from the registry.

Update a pool

Template 

PATCH /v1/pools/{pool}

Request 

PATCH /v1/pools/wat HTTP/1.1
Host: zaqar.example.com

{
  "uri": "mongodb://zaqar3.example.com:27018",
  "weight": 120
}


Response 

HTTP/1.1 204 No Content

Discussion

Allows one to update any or all of: `weight`, `uri`, `options`. At least one of these fields must be specified, else, a HTTP 400 is returned.

List pools

Template 

GET /v1/pools?detailed=True&limit=10&marker=taco

Request 

GET /v1/pools HTTP/1.1
Host: zaqar.proxy.example.com


Response 

HTTP/1.1 200 OK
Content-Location: /v1/pools

[
  {"href": "/v1/pools/wat", "weight": 100, "uri": "mongodb://zaqar1.example.com:27017"},
  {"href": "/v1/pools/wot", "weight": 50, "uri": "redis://zaqar2.example.com:6379"}
]

Discussion

Lists the registered pools.

detailed if True, returns the options field in the listing. marker used for pagination - what pool do we start listing from? limit how many entries to return per request?

Retrieved from "https://wiki.openstack.org/w/index.php?title=Zaqar/specs/sharding/v1&oldid=61074"