Zaqar/specs/sharding/v1

Zaqar Storage Pooling: v1 Blueprint

Overview

This proposal introduces another approach to tackle making Zaqar highly scalable without losing any of its flexibility. This particular document covers the administrative API for registering storage pools.

Reference

For authoritative header/general details, please refer to the Zaqar Queues API v1 Spec.

Endpoints synopsis

GET /pools?detailed=False&marker=None&limit=10
GET /pools/{pool}?detailed=False

PUT /pools/{pool}
DELETE /pools/{pool}
PATCH /pools/{pool}

GET /health

Pooling with Zaqar

Zaqar supports heterogeneous pooling through the use of:

• stevedore for dynamic storage driver loading
• node weights

As long as an entry point is defined in an installed module that matches the scheme of a pool connection URI, then Zaqar will be able to use that pool. For example, a pool entry might look like:

{
  "weight": 100,
  "uri": "mongodb://localhost:27017",
  "options": {
    "max_retry_sleep": 1
  }
}

Register a pool

Template

PUT /v1/pools/{pool}

Request

PUT /v1/pools/wat HTTP/1.1
Host: zaqar.example.com

{
  "weight": 100,
  "uri": "mongodb://localhost:27017",
  "options": {
    "max_retry_sleep": 1,
    "partitions": 8
  }
}

Response

HTTP/1.1 201 Created
Location: /v1/pools/wat

Discussion

Register a pool.

pool is the name to give the pool entry. The name MUST NOT exceed 64 bytes in length, and is limited to US-ASCII letters, digits, underscores and hyphens.

weight is the likelihood that this pool will be selected for the next queue allocation. It must be an integer greater than -1.

uri is a connection string compatible with a storage client (e.g., pymongo) attempting to connect to that pool.

options An optional request component that gives storage-specific options used by storage driver implementations. Valid parameters come from the registered options for a given storage backend, for example: mongodb, sqlite

Read a pool

Template

GET /v1/pools/{pool}?detailed=True

Request

GET /v1/pools/wat HTTP/1.1
Host: zaqar.example.com

Response

HTTP/1.1 200 OK
Content-Location: /v1/pools/wat

{
  "uri": "mongodb://zaqar1.example.com:27017",
  "weight": 100
}

Discussion

Returns information on a registered pool.

Delete a pool

Template

DELETE /v1/pool/{pool}

Request

DELETE /v1/pools/wat HTTP/1.1
Host: zaqar.example.com

Response

HTTP/1.1 204 No Content

Discussion

Removes a pool from the registry.

Update a pool

Template

PATCH /v1/pools/{pool}

Request

PATCH /v1/pools/wat HTTP/1.1
Host: zaqar.example.com

{
  "uri": "mongodb://zaqar3.example.com:27018",
  "weight": 120
}

Response

HTTP/1.1 204 No Content

Discussion

Allows one to update any or all of: `weight`, `uri`, `options`. At least one of these fields must be specified, else, a HTTP 400 is returned.

List pools

Template

GET /v1/pools?detailed=True&limit=10&marker=taco

Request

GET /v1/pools HTTP/1.1
Host: zaqar.proxy.example.com

Response

HTTP/1.1 200 OK
Content-Location: /v1/pools

[
  {"href": "/v1/pools/wat", "weight": 100, "uri": "mongodb://zaqar1.example.com:27017"},
  {"href": "/v1/pools/wot", "weight": 50, "uri": "redis://zaqar2.example.com:6379"}
]

Discussion

Lists the registered pools.

detailed if True, returns the options field in the listing. marker used for pagination - what pool do we start listing from? limit how many entries to return per request?