Zaqar/specs/api/v1.1

= Zaqar API v1.1 =

Overview
Zaqar provides an HTTP-based API in the spirit of the REST architecture style.

This guide assumes the reader is familiar with REST, HTTP/1.1 and JSON. URI templates use the same syntax as RFC 6570.

Note: Error responses are enumerated on a separate page: Marconi/specs/api/v1.1/errors

Changes from v1.0
BREAKING


 * Queue existence check endpoint (via HEAD & GET) no longer exists.
 * The  header is now required for every request. This was done in anticipation of identity allowing a single user to manage multiple projects, in which case we will need to know which project this user is requesting (vs. assuming 1:1 mapping between auth token and project). It also affords rate limiting at the LB.
 * The  header is now required for every request.
 * The  header is no longer set in responses
 * The  field is no longer included in responses to a message posting request. When using the official SQL and MongoDB drivers, Zaqar will either succeed in enqueuing all messages in the submitted batch, or fail to enqueue any of them. Any backend driver that supports batch message posting will need to guarantee atomic inserts of the entire batch (v2.0 of the API will define some operations as optional, such as batch message posting).
 * When a collection resource is requested (i.e., queues, messages) and the collection is empty, an empty JSON array is now returned in the response body, rather than HTTP 204 No Content.
 * All top-level JSON arrays in response bodies are now encapsulated within an object. This mitigates a security risk we will have when we later implement support for accessing Zaqar directly from web apps (JavaScript).
 * When posting one or more messages, the list of messages is now encapsulated in a JSON object type. This was done to be consistent with response bodies.
 * Health is now an admin-only endpoint and returns operational stats for a given node. A new "ping" endpoint was added for load balancers to use.

NON-BREAKING


 * MessagePack is now supported in addition to JSON
 * Extraneous path elements from relative URIs were removed
 * Version discovery is now available by accessing the root URI path
 * The  header, when present, now specifies a full (i.e., non-relative) URI
 * A  field has been added to messages. It is omitted if the message is not claimed.
 * An  field has been added to messages
 * Message href's now always include  in the query string if the message is claimed.
 * When posting messages or creating a claim, the  and   fields may be omitted. If it is not given by the client, Zaqar will default to the max values configured by the cloud operator.
 * New pools API and capability
 * Added new "pop" semantics for claiming and deleting messages in a single request. Note that this should only be used when an application is OK with the risk of missing a message when a worker crashes.
 * The server will automatically create a queue when posting a message, if the queue does not already exist.
 * Queries for messages and stats in non-existing queues now return empty lists, etc. rather than 404s.

Clients

 * Clients should follow HTTP redirects
 * Clients should advertise gzip support
 * Clients should identify themselves in the UserAgent request header; e.g., "User-Agent: python/2.7 cloudthing/1.2"
 * Clients should not hard-code URI paths or templates since they may change over time. Instead, clients should cache links and link templates provided by the API.

API Versioning
The Zaqar API uses a URI versioning scheme. The first element of the path contains the target version identifier, e.g.:

https://marconi.example.com/v1.1

The URI version will only be incremented to accommodate major new features or API redesigns that can not be made backwards-compatible. When new API versions are released, older versions are deprecated. Zaqar maintainers will work with developers and partners to ensure there is adequate time to migrate to new versions before deprecated ones are discontinued.

Since the version is only incremented to accommodate major API revisions, sub-versioning of the API will be rare.

Resource Media Types
The API supports `application/json`, encoded as UTF-8. MessagePack is also supported ('application/x-msgpack'), and is the recommended default for client libraries.

Unrecognized protocol elements received from the server should simply be ignored. This includes JSON object properties, link relation types, media types, etc.

Authentication
''May be optionally implemented by middleware or reverse proxy. If implemented, the following applies:''

All requests to the API may only be performed by an authenticated agent.

To authenticate, an agent issues an authentication request to a Keystone Identity Service endpoint.

In response to valid credentials, Keystone responds with an auth token and a service catalog that contains a list of all services and endpoints available for the given token. Multiple endpoints may be returned for Zaqar according to physical locations and performance/availability characteristics of different deployments.

Normally, Keystone middleware provides the X-Project-Id header based on the auth token submitted by the Zaqar client. For this to work, clients MUST specify a valid auth token in the `X-Auth-Token` header for each request to the Zaqar API. The API validates auth tokens against Keystone before servicing each request

If auth is not enabled, clients must provide the X-Project-Id header themselves.

Authorization
''May be optionally implemented by middleware or reverse proxy. If implemented, the following applies:''

The API needs to verify read/write access to all endpoints according to the provided auth token (RBAC). The ACL should be cached with the token.

Endpoints
Clients may choose to interact with any of the service endpoints retrieved from Keystone. Agents use the given endpoint as a home document href from which to discover links to all other API requests. Endpoints may be organized by region

An example endpoint:

https://marconi.example.com/v1.1

Errors
If any request results in an error, the server will return an appropriate 4xx or 5xx HTTP status code, and optionally the following information in the body:


 * Title
 * Description
 * Internal code
 * Link to more information

Example:

HTTP/1.1 400 Bad Request Content-Type: application/json

{ "title": "Unsupported limit", "description": "The given limit cannot be negative, and cannot be greater than 50.", "code": 1092, "link": { "rel": "help", "href": "http://docs.example.com/messages#limit", "text": "API documentation for the limit parameter" } }

Common Headers
Each request to the API must include certain standard and extended HTTP headers. These headers provide host, agent, authentication and other pertinent information to the server.

HTTP Response Codes
See Response Codes

Sample API Request
GET /v1.1/queues/fizbat/messages?marker=1355-237242-783&limit=10 HTTP/1.1 Host: marconi.example.com User-Agent: python/2.7 killer-rabbit/1.2 Date: Wed, 28 Nov 2012 21:14:19 GMT Accept: application/json Accept-Encoding: gzip X-Auth-Token: 7d2f63fd-4dcc-4752-8e9b-1d08f989cc00 X-Project-Id: 518b51ea133c4facadae42c328d6b77b Client-ID: 30387f00-39a0-11e2-be4d-a8d15f34bae2

Endpoints Synopsis
GET /v1.1 GET /v1.1/health GET /v1.1/ping
 * 1) Base endpoints

GET /v1.1/queues{?marker,limit,detailed} PUT /v1.1/queues/{name} DELETE /v1.1/queues/{name}
 * 1) Queues

GET /v1.1/queues/{queue_name}/stats
 * 1) Queue stats

GET /v1.1/queues/{queue_name}/messages{?marker,limit,echo,include_claimed} POST /v1.1/queues/{queue_name}/messages DELETE /v1.1/queues/{queue_name}/messages/{message_id}{?claim_id} DELETE /v1.1/queues/{queue_name}/messages{?ids} DELETE /v1.1/queues/{queue_name}/messages{?pop}
 * 1) Messages

POST /v1.1/queues/{queue_name}/claims{?limit} GET /v1.1/queues/{queue_name}/claims/{claim_id} PATCH /v1.1/queues/{queue_name}/claims/{claim_id} DELETE /v1.1/queues/{queue_name}/claims/{claim_id}
 * 1) Claims

GET /v1.1/pools?detailed=False&marker=None&limit=10 GET /v1.1/pools/{pool}?detailed=False PUT /v1.1/pools/{pool} DELETE /v1.1/pools/{pool} PATCH /v1.1/pools/{pool}
 * 1) Pools

Get Home Document
Template

GET /v1.1

Request

GET /v1.1 HTTP/1.1 Host: marconi.example.com

...

Response

HTTP/1.0 200 OK Cache-Control: max-age=86400 Content-Length: 4345 Content-Type: application/json-home Date: Tue, 06 Aug 2013 16:31:48 GMT Server: WSGIServer/0.1 Python/2.7.3

{   "resources": { "rel/queue-stats": { "href-template": "/v1.1/queues/{queue_name}/stats", "href-vars": { "queue_name": "param/queue_name" },           "hints": { "allow": [ "GET" ],               "formats": { "application/json": {} }           }        },        "rel/post-messages": { "href-template": "/v1.1/queues/{queue_name}/messages", "href-vars": { "queue_name": "param/queue_name" },           "hints": { "accept-post": [ "application/json" ],               "allow": [ "POST" ],               "formats": { "application/json": {} }           }        },        "rel/queue": { "href-template": "/v1.1/queues/{queue_name}", "href-vars": { "queue_name": "param/queue_name" },           "hints": { "allow": [ "PUT", "DELETE" ],               "formats": { "application/json": {} }           }        },        "rel/queues": { "href-template": "/v1.1/queues{?marker,limit,detailed}", "href-vars": { "marker": "param/marker", "detailed": "param/detailed", "limit": "param/queue_limit" },           "hints": { "allow": [ "GET" ],               "formats": { "application/json": {} }           }        },        "rel/messages": { "href-template": "/v1.1/queues/{queue_name}/messages{?marker,limit,echo,include_claimed}", "href-vars": { "marker": "param/marker", "include_claimed": "param/include_claimed", "queue_name": "param/queue_name", "limit": "param/messages_limit", "echo": "param/echo" },           "hints": { "allow": [ "GET" ],               "formats": { "application/json": {} }           }        },        "rel/messages-delete": { "href-template": "/v1.1/queues/{queue_name}/messages{?ids,pop}", "href-vars": { "ids": "param/ids", "pop": "param/pop", },           "hints": { "allow": [ "DELETE" ],               "formats": { "application/json": {} }           }        },        "rel/claim": { "href-template": "/v1.1/queues/{queue_name}/claims{?limit}", "href-vars": { "queue_name": "param/queue_name", "limit": "param/claim_limit" },           "hints": { "accept-post": [ "application/json" ],               "allow": [ "POST" ],               "formats": { "application/json": {} }           }        }    } }

Discussion

The entire Zaqar API is discoverable from a single starting point; agents do not need to know any more than this one URI in order to explore the entire API.

This document is cache-able (and should be cached by proxy and client).

See also: http://tools.ietf.org/html/draft-nottingham-json-home-03

Check Node Health
Template

GET /v1.1/health

or

HEAD /v1.1/health

Request

GET /v1.1/health HTTP/1.1 Host: example.marconi.com

...

Response {   "mongo_pool_1": { "message_volume": { "claimed": 0, "total": 0, "free": 0 },       "storage_reachable": true, "operation_status": { "create_queue": { "seconds": 0.0021300315856933594, "ref": null, "succeeded": true },           "post_messages": { "seconds": 0.033502817153930664, "ref": null, "succeeded": true },           "list_messages": { "seconds": 0.000013113021850585938, "ref": null, "succeeded": true },           "claim_messages": { "seconds": 0.0013759136199951172, "ref": "3f515f37-58a0-4c81-8214-3e92979b82e7", "succeeded": false },           "delete_queue": { "seconds": 0.0030739307403564453, "ref": null, "succeeded": true }       }    },    "mongo_pool_2": { "message_volume": { "claimed": 0, "total": 0, "free": 0 },       "storage_reachable": true, "operation_status": { "create_queue": { "seconds": 0.0011799335479736328, "ref": null, "succeeded": true },           "post_messages": { "seconds": 0.024316072463989258, "ref": null, "succeeded": true },           "list_messages": { "seconds": 0.000008106231689453125, "ref": null, "succeeded": true },           "claim_messages": { "seconds": 0.000576019287109375, "ref": "68629fda-b4ce-4cf9-978a-df0df8df36a7", "succeeded": false },           "delete_queue": { "seconds": 0.003300905227661133, "ref": null, "succeeded": true }       }    },    "catalog_reachable": true }

Discussion

''This is an operator endpoint and should not be implemented in client libraries intended for end users. It should also not be included in the json-home document unless the requesting user is an administrator.''

Use this request to get detailed operational stats for a specific Zaqar node.

Ping a Node
Template

GET /v1.1/ping

or

HEAD /v1.1/ping

Request

GET /v1.1/ping HTTP/1.1 Host: example.marconi.com

...

Response

HTTP/1.1 204 No Content

or

HTTP/1.1 503 Service Unavailable

Discussion

''This is an operator endpoint and should not be implemented in client libraries intended for end users. It should also not be included in the json-home document unless the requesting user is an administrator.''

Use this resource to check whether a given Zaqar node is online. If the node is down, this endpoint will not be reachable and will act as a signal to the load balancer to take the node out of rotation. Alternatively, the endpoint may be reachable but the service is temporarily unavailable due to storage driver failure or some other error.

The health endpoint may be accessed only if the X-Forwarded-For header is present. In that case, it will not require auth, allowing load balancers to query a given node's health.

For requests NOT sent via the LB, Zaqar will return 404 Not Found, since this endpoint is not meant to be accessed by end users and there is no point in exposing another potential DDoS vector to the world.

Get Queue
Template

GET /v1.1/queues/{queue_name}

Request

GET /v1.1/queues/fizbit HTTP/1.1 Host: marconi.example.com

...

Response HTTP/1.1 200 OK

{  "key": { "key2": "value", "key3": [1, 2, 3, 4, 5] } }

Discussion

Returns queue metadata.

Create Queue
Template

PUT /v1.1/queues/{queue_name}

{  ... }

Request

PUT /v1.1/queues/fizbat HTTP/1.1 Host: marconi.example.com

Response

HTTP/1.1 201 Created Location: https://marconi.example.com/v1.1/queues/fizbit

Discussion

Creates a queue.

The body of the request is an arbitrary document which allows storing contextual information about the way a particular application should interact with the queue. The size of this body, in characters and including whitespace must be <= 64 KiB (configurable). The document MUST be valid JSON.

is the name to give the queue. The name MUST NOT exceed 64 bytes in length, and is limited to US-ASCII letters, digits, underscores and hyphens.

List Queues
Template

GET /v1.1/queues{?marker,limit,detailed}

Request

GET /v1.1/queues?marker=baz&detailed=true HTTP/1.1 Host: marconi.example.com

...

Response

HTTP/1.1 200 OK

{ "links": [ {     "rel": "next", "href": "queues?marker=kooleo&detailed=true" } ],  "queues": [ { "name": "boomerang", "href": "queues/boomerang" }, { "name": "fizbit", "href": "queues/fizbit" },

...

{ "name": "kooleo", "href": "queues/kooleo" } ] }

or, if no queues exist:

HTTP/1.1 200 OK

{ "links": [ {     "rel": "next", "href": "queues?marker=baz&detailed=true" } ],  "queues": [] }

Discussion

Query parameters are defined as follows:

name of the last queue returned in a previous response, used to retrieve the next page of results. NOTE: Clients should normally just follow the "next" link rather than attempting to construct URI paths manually.

Maximum number of queue records to return. May not be more than the hard maximum configured by the cloud operator (default 20). If not given, the limit defaults to 10.

Set to "true" to inline queue metadata in the listing (default false).

Delete Queue
Template

DELETE /v1.1/queues/{queue_name}

Request

DELETE /v1.1/queues/fizbat HTTP/1.1 Host: marconi.example.com

Response

HTTP/1.1 204 No Content

Discussion

Use this operation to immediately delete a queue along with all its messages (if any).

Queue Stats
Template

GET /v1.1/queues/{queue_name}/stats

Request

GET /v1.1/queues/fizbit/stats HTTP/1.1 Host: marconi.example.com

...

Response HTTP/1.1 200 OK

{ "messages": { "free": 146929, "claimed": 2409, "total": 149338, "oldest": { "href": "queues/fizbit/messages/50b68a50d6f5b8c8a7c62b01", "age": 63, "created": "2013-08-12T20:44:55Z" },   "newest": { "href": "queues/fizbit/messages/50b68a50d6f5b8c8a7c62b01", "age": 12, "created": "2013-08-12T20:45:46Z" } } }

Discussion

Returns queue statistics, including how many messages are in the queue, broken out by status.

NOTE: If total is 0, then "oldest" and "newest" message stats are not included.

Messages
All message-related operations require  to be included in the headers. This is to ensure that messages are not echoed back to the client that posted them unless the client explicitly requests this.

List Messages
Template

GET /v1.1/queues/{queue_name}/messages{?marker,limit,echo,include_claimed}

Request

GET /v1.1/queues/fizbit/messages?marker=1355-237242-783&limit=10 HTTP/1.1 Host: marconi.example.com

...

Response

HTTP/1.1 200 OK

...

{ "links": [ {     "rel": "next", "href": "messages?marker=6244-244224-783&limit=10" } ],  "messages": [ {     "href": "messages/50b68a50d6f5b8c8a7c62b01", "id": "50b68a50d6f5b8c8a7c62b01", "ttl": 120, "age": 53, "body": { "event": "ActivateAccount", "mode": "active" }   },    {      "href": "messages/50b68a50d6f5b8c8a7c62b02?claim_id=06ef2372-6746-11e3-b311-b3adcbe406e9", "id": "50b68a50d6f5b8c8a7c62b01", "ttl": 800, "age": 790, "body": { "event": "CreateInvoice", "customer_id": "90fd8734-6746-11e3-be3c-7e45c531c7ca" }   }  ] }

or, if no messages are available:

HTTP/1.1 200 OK

...

{ "links": [ {     "rel": "next", "href": "messages?marker=6244-244224-783&limit=10" } ],  "messages": [] }

Discussion

Message IDs and markers are opaque strings; clients should make no assumptions about their format or length. Furthermore, clients should assume there is no relationship between markers and message IDs (i.e., one cannot be derived from the other). This allows for a wide variety of storage driver implementations.

Results are ordered by age, oldest message first.

specifies up to 20 messages (configurable) to return. If not specified, limit defaults to 10. When more messages are available than can be returned in a single request, the client can pick up the next batch by simply using the URI template parameters returned from the previous call in the "next" field (TBD).

is an opaque string that the client can use to request the next batch of messages. It communicates to the server which messages the client has already received.

Note: If  is not specified, the API will return all messages at the head of the queue (up to limit).

is a boolean (i.e., "true" or "false") that determines whether or not the API will return a client's own messages, as determined by the  portion of the Client-ID header. If not specified, echo defaults to "false".

is a boolean (i.e., "true" or "false") that determines whether or not the API will return claimed messages as well as unclaimed ones. If not specified, defaults to "false" (only unclaimed messages are returned).

Get a Specific Message
Template

GET /v1.1/queues/{queue_name}/messages/{message_id}

Request

GET /v1.1/queues/fizbat/messages/50b68a50d6f5b8c8a7c62b01 HTTP/1.1 Host: marconi.example.com

...

Response

HTTP/1.1 200 OK Content-Location: /v1.1/queues/fizbat/messages/50b68a50d6f5b8c8a7c62b01

...

{ "href": "/v1.1/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b01", "id": "50b68a50d6f5b8c8a7c62b01", "ttl": 800, "age": 790, "body": { "event": "ActivateAccount", "mode": "active" } }

Discussion

Message fields are defined as follows:

is an opaque relative URI that the client can use to uniquely identify a message resource, and interact with it.

is the ttl set on the message when it was posted. The message will expire after (ttl - age) seconds.

number of seconds since ts, relative to the server's clock.

Arbitrary document submitted along with the original request to post the message.

If either the message ID or the claim ID is malformed or nonexistent, no message is returned.

Get a Set of Messages by ID
Template

GET /v1.1/queues/{queue_name}/messages{?ids}

Request

GET /v1.1/queues/fizbat/messages?ids=50b68a50d6f5b8c8a7c62b01,f5b8c8a7c62b0150b68a50d6 HTTP/1.1 Host: marconi.example.com

...

Response

HTTP/1.1 200 OK Content-Location: /v1.1/queues/fizbat/messages?ids=50b68a50d6f5b8c8a7c62b01,f5b8c8a7c62b0150b68a50d6

...

{ "messages": [ {     "href": "/v1.1/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b01", "id": "50b68a50d6f5b8c8a7c62b01", "ttl": 800, "age": 32, "body": { "cmd": "EncodeVideo", "jobid": 58229 }   },    {      "href": "/v1.1/queues/fizbit/messages/f5b8c8a7c62b0150b68a50d6", "id": "f5b8c8a7c62b0150b68a50d6", "ttl": 800, "age": 790, "body": { "cmd": "EncodeAudio", "jobid": 58201 }   }  ] }

Discussion

Unlike message listing, a client's own messages are always returned in this approach. Note that the list of ids may not exceed 20 (configurable, shared setting with number of messages that can be posted at once).

This approach returns all the messages with matching IDs. If a malformed ID or a nonexistent message ID is provided, it is ignored and the remaining messages are returned.

Post Message(s)
Template

POST /v1.1/queues/{queue_name}/messages

Request

POST /v1.1/queues/fizbit/messages HTTP/1.1 Host: marconi.example.com

...

{ "messages": [ {     "ttl": 300, "body": { "event": "BackupStarted", "backup_id": "c378813c-3f0b-11e2-ad92-7823d2b0f3ce" }   },    {      "body": { "event": "BackupProgress", "current_bytes": "0", "total_bytes": "99614720" }   }  ] }

Responses

When a single message is submitted:

HTTP/1.1 201 Created Location: https://marconi.example.com/v1.1/queues/fizbit/messages?ids=50b68a50d6f5b8c8a7c62b01 Content-Type: application/json

{   "links": [ {           "rel": "rel/message", "href": "messages/50b68a50d6f5b8c8a7c62b01" }   ] }

When multiple messages are submitted:

HTTP/1.1 201 Created Location: https://marconi.example.com/v1.1/queues/fizbit/messages?ids=50b68a50d6f5b8c8a7c62b01,50b68a50d6f5b8c8a7c62b05 Content-Type: application/json

{   "links": [ {           "rel": "rel/message", "href": "messages/50b68a50d6f5b8c8a7c62b01" },       {            "rel": "rel/message", "href": "messages/50b68a50d6f5b8c8a7c62b05" }   ] }

Discussion

Up to 20 messages (default, but configurable) may be submitted in a single request, but must always be encapsulated in a collection container (e.g., an array in JSON). The resulting value of the Location header or response body may be used to retrieve the created messages for further processing if needed.

The client only specifies the body and ttl for the message; the server will insert metadata such as id and age.

Note: If the specified queue does not already exist, the server will create it. Applications do not need to explicitly manage the lifetimes of their queues, but are encouraged to delete queues that are no longer in use.

The response body contains a list of resource paths corresponding to each message submitted in the request, in the same order. In the case of a server-side error part-way through the processing of the submitted messages, the entire batch will be abandoned (all-or-nothing), and the client will have to retry its request.

The size of the request document, in characters including whitespace, is 256 KiB by default (configurable). The document MUST be a well-formed JSON or MessagePack document (Zaqar will validate).

specifies an arbitrary document which constitutes the body of the message being sent.

is how long the server should wait before expiring and removing the message from the queue. Value MUST be between 60 and 1209600 seconds (14 days, configurable), inclusive. Note that the server may not actually delete the message until it's age has reached up to (ttl + 60) seconds, to allow for flexibility in storage implementations. If the ttl field is omitted from the request, the default value is used, which is normally 3600 seconds unless customized by the service provider.

Delete Message
Template

DELETE /v1.1/queues/{queue_name}/messages/{message_id}{?claim_id}

Request

DELETE /v1.1/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b01 HTTP/1.1 Host: marconi.example.com

...

Response

HTTP/1.1 204 No Content

Discussion

specifies the message to delete.

specifies that the message should only be deleted if it has the specified claim ID, and that claim has not expired. This is useful for ensuring only one agent processes any given message; whenever a worker client's claim expires before it has a chance to delete a message it has processed, the worker must roll back any actions it took based on that message, since another worker will now be able to claim and process the same message.

Note that if claim_id is not specified, but the message is claimed, the operation will fail. In other words, claimed messages can only ever be deleted by providing an appropriate claim_id.

Delete Multiple Messages
Template

DELETE /v1.1/queues/{queue_name}/messages{?ids,pop}

Request

DELETE /v1.1/queues/fizbit/messages?ids=50b68a50d6f5b8c8a7c62b01,50b68a50d6f5b8c8a7c62b02 HTTP/1.1 Host: marconi.example.com

...

Response

HTTP/1.1 204 No Content

or, to atomically pop several messages (when only 2 messages are available):

Request

DELETE /v1.1/queues/fizbit/messages?pop=5 HTTP/1.1 Host: marconi.example.com

...

Response

HTTP/1.1 200 OK Content-Type: application/json

...

{ "messages": [ {     "href": "/v1.1/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b01?claim_id=a28ee94e-6cb4-11e2-b4d5-7703267a7926", "id": "50b68a50d6f5b8c8a7c62b01", "ttl": 800, "age": 100, "body": { "object_id": "8a50d6", "target": "h.264" }   },    {      "href": "/v1.1/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b02?claim_id=a28ee94e-6cb4-11e2-b4d5-7703267a7926", "id": "50b68a50d6f5b8c8a7c62b02", "ttl": 800, "age": 790, "body": { "object_id": "fb8c8a", "target": "h.264" }   }  ] }

pop & ids parameters are mutually exclusive. Using them together in a request will result in HTTP 400.

Request

DELETE /v1.1/queues/fizbit/messages?pop=5&ids=50b68a50d6f5b8c8a7c62b02 HTTP/1.1 Host: marconi.example.com

...

Response

HTTP/1.1 400 Bad Request Content-Type: application/json

...

{ "message": pop and id params cannot be present together in the delete request }

Discussion

Bulk delete for messages.

specifies the messages to delete, up to a maximum of 20 (the limit is configurable, shares the same setting as the maximum for listing messages)

specifies a certain number of messages to pop off the queue, which is equivalent to claiming and deleting those messages atomically (therefore guaranteeing once-and-only-once delivery). The maximum value for this parameter is the same as for the maximum number of messages that can be claimed in a single operation. Note that if the worker crashes after popping the messages, but before processing them, no other workers will be able to reclaim those messages. Therefore, the "pop" operation should only be used when the risk of losing an occasional message can be justified.

Note that the  and   parameters are mutually exclusive. In other words, if both params are present the server will respond with a "400 Bad Request" status.

If any of the message IDs are malformed or non-existent, they are ignored. The remaining valid message IDs are deleted.

WARNING: Claimed messages will not be skipped; they will be deleted along with unclaimed messages. If using the worker pool pattern, in which workers are claiming batches of messages, we encourage deleting messages one at a time. This ensures that at most one message could be processed, but not deleted, if a worker happens to crash at just the wrong moment:

for each message in batch: process message delete message

As opposed to:

for each message in batch: process message bulk-delete batch

Claim Messages
Template

POST /v1.1/queues/{queue_name}/claims{?limit} Content-Type: application/json

...

{   "ttl": {claim_ttl}, "grace": {message_grace} }

Request

POST /v1.1/queues/fizbit/claims?limit=5 HTTP/1.1 Host: marconi.example.com Content-Type: application/json Accept: application/json

...

{   "ttl": 300, "grace": 300 }

Or, to use the default values for ttl and grace, one or both fields may be omitted in the request:

POST /v1.1/queues/fizbit/claims?limit=5 HTTP/1.1 Host: marconi.example.com Content-Type: application/json Accept: application/json

...

{   "ttl": 300, }

NOTE: If omitting both fields, the entire request body may be omitted (although submitting an empty JSON object is also acceptable):

POST /v1.1/queues/fizbit/claims?limit=5 HTTP/1.1 Host: marconi.example.com Content-Length: 0 Accept: application/json

Response

The client receives a claim URI and a list of claimed messages, if any:

HTTP/1.1 201 Created Content-Type: application/json Location: https://marconi.example.com/v1.1/queues/fizbit/claims/a28ee94e-6cb4-11e2-b4d5-7703267a7926

...

{ "messages": [ {     "href": "/v1.1/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b01?claim_id=a28ee94e-6cb4-11e2-b4d5-7703267a7926", "id": "50b68a50d6f5b8c8a7c62b01", "ttl": 800, "age": 100, "body": { "object_id": "8a50d6", "target": "h.264" }   },    {      "href": "/v1.1/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b02?claim_id=a28ee94e-6cb4-11e2-b4d5-7703267a7926", "id": "50b68a50d6f5b8c8a7c62b02", "ttl": 800, "age": 790, "body": { "object_id": "fb8c8a", "target": "h.264" }   }  ] }

Or, if no messages are available to claim:

HTTP/1.1 201 Created Content-Type: application/json Location: https://marconi.example.com/v1.1/queues/fizbit/claims/f9151272-673e-11e3-8e72-43d6a24410d2

...

{ "messages": [] }

Discussion

Claims a set of messages, up to limit, from oldest to newest, skipping any that are already claimed. If no unclaimed messages are available, returns 204 No Content.

The client should delete the message when it has finished processing it, before the claim expires, to ensure the message is processed only once. As part of the delete operation, all worker clients should specify the claim ID (this is best done by simply using the provided href). That way, the server can return an error if the claim just expired (notifying the client of the race condition), giving the worker a chance to roll back its own processing of the given message, since another worker will likely claim the message and process it.

Just as with a message's age, the age given for the claim is relative to the server's clock, and is useful for determining how quickly messages are getting processed, and whether a given message's claim is about to expire.

When a claim expires, it is removed, allowing another client worker to claim the message in the case that the original worker fails to process it.

specifies up to 20 messages (configurable) to claim. If not specified, limit defaults to 10. Note that claim creation is best-effort, meaning the server may claim and return less than the requested number of messages.

is how long the server should wait before releasing the claim. Value MUST be between 60 and 43200 seconds (12 hours, configurable). If omitted, the default value is used, which is normally 300 seconds unless customized by the service provider.

is the message grace period in seconds. Value MUST be between 60 and 43200 seconds (12 hours, configurable). The server will extend the lifetime of claimed messages to be at least as long as the lifetime of the claim itself, plus a specified grace period to deal with crashed workers (up to 1209600 or 14 days including claim lifetime). This gives other workers a chance to reclaim the message after the previous claim expires, but before the message itself expires. Note that If a message's lifetime already extends beyond the claim's TTL plus grace period, the message's lifetime will not be adjusted. If ommitted, the default value is used, which is normally 60 seconds unless customized by the service provider.

Query Claim
Template

GET /v1.1/queues/{queue_name}/claims/{claim_id}

Request

GET /v1.1/queues/foo-bar/claims/a28ee94e-6cb4-11e2-b4d5-7703267a7926 Host: marconi.example.com

...

Response

HTTP/1.1 200 OK

...

{ "age": 19, "ttl": 30, "messages": [ ... ] }

Discussion

Renew Claim
Template

PATCH /v1.1/queues/{queue_name}/claims/{claim_id} Content-Type: application/json

Request

PATCH /v1.1/queues/fizbit/claims/a28ee94e-6cb4-11e2-b4d5-7703267a7926 HTTP/1.1 Host: marconi.example.com Content-Type: application/json

...

{ "ttl": 300 }

Response

HTTP/1.1 204 No Content

Discussion

Clients should periodically renew claims during long-running batches of work to avoid losing a claim in the middle of processing a message. This is done by issuing a PATCH to a specific claim resource and including a new TTL for the claim (which may be different from the original TTL). The server will reset the age of the claim and apply the new TTL.

Release Claim
Template

DELETE /v1.1/queues/{queue_name}/claims/{claim_id}

Request

DELETE /v1.1/queues/foo-bar/claims/a28ee94e-6cb4-11e2-b4d5-7703267a7926 HTTP/1.1 Host: marconi.example.com

Response

HTTP/1.1 204 No Content

Discussion

Use this operation to immediately release a claim, making any (remaining, non-deleted) messages associated with the claim available to other workers.

This operation is useful when a worker is performing a graceful shutdown, fails to process one or more messages, or is taking longer than expected to process messages, and wishes to make the remainder available to other workers.

Message Store Pools
WARNING: in a production deployment, the storage pools API should be restricted to administrators through role-based authentication (e.g., via policy middleware). You can also reduce your attack surface by disabling the pools endpoints completely on public-facing web heads, by setting  in zaqar.conf.

Zaqar supports heterogeneous pools through the use of:


 * stevedore for dynamic storage driver loading
 * node weights

As long as an entry point is defined in an installed module that matches the scheme of a pool connection URI, Zaqar will be able to use that pool. For example, a pool entry might look like:

{ "weight": 100, "uri": "mongodb://localhost:27017", "options": { "max_retry_sleep": 1 } }

Register a Pool
Template

PUT /v1.1/pools/{pool}

Request

PUT /v1.1/pools/wat HTTP/1.1 Host: marconi.example.com

{ "weight": 100, "uri": "mongodb://localhost:27017", "options": { "max_retry_sleep": 1, "partitions": 8 } }

Response

HTTP/1.1 201 Created Location: /v1.1/pools/wat

Discussion

Registers a pool.

is the name to give the storage pool entry. The name MUST NOT exceed 64 bytes in length, and is limited to US-ASCII letters, digits, underscores and hyphens.

is the likelihood that this pool will be selected for the next queue allocation. It must be an integer greater than -1.

is a connection string compatible with a storage client (e.g., pymongo) attempting to connect to that pool.

An optional request component that gives storage-specific options used by storage driver implementations. Valid parameters come from the registered options for a given storage backend, for example: mongodb, sqlite

Get Pool Info
Template

GET /v1.1/pools/{pool}?detailed=True

Request

GET /v1.1/pools/wat HTTP/1.1 Host: marconi.example.com

Response

HTTP/1.1 200 OK Content-Location: /v1.1/pools/wat

{ "uri": "mongodb://marconi1.example.com:27017", "weight": 100 }

Discussion

Returns information on a registered pool.

Delete a Pool
Template

DELETE /v1.1/pools/{pool}

Request

DELETE /v1.1/pools/wat HTTP/1.1 Host: marconi.example.com

Response

HTTP/1.1 204 No Content

Discussion

Removes a storage pool from the registry.

Update a Pool
Template

PATCH /v1.1/pools/{pool}

Request

PATCH /v1.1/pools/wat HTTP/1.1 Host: marconi.example.com

{ "uri": "mongodb://marconi3.example.com:27018", "weight": 120 }

Response

HTTP/1.1 204 No Content

Discussion

Allows one to update any or all of: `weight`, `uri`, `options`. At least one of these fields must be specified, else, a HTTP 400 is returned.

List Pools
Template

GET /v1.1/pools?detailed=True&limit=10&marker=taco

Request

GET /v1.1/pools HTTP/1.1 Host: marconi.proxy.example.com

Response

HTTP/1.1 200 OK Content-Location: /v1.1/pools

{ "links": [ {     "rel": "next", "href": "/v1.1/pools?marker=wot&limit=10&detailed=True   }  ],  "pools": [    {"href": "/v1.1/pools/wat", "weight": 100, "uri": "mongodb://marconi1.example.com:27017"},    {"href": "/v1.1/pools/wot", "weight": 50, "uri": "redis://marconi2.example.com:6379"}  ] }

or if no pools exist:

HTTP/1.1 200 OK Content-Location: /v1.1/pools

{ "links": [ {     "rel": "next", "href": "/v1.1/pools?marker=bar&limit=10&detailed=True   }  ],  "pools": [] }

Discussion

Lists the registered pools.

if True, returns the options field in the listing. used for pagination - what pool do we start listing from? how many entries to return per request?

Queue Flavors
WARNING: in a production deployment, the queue flavors API should be restricted to administrators through role-based authentication (e.g., via policy middleware). You can also reduce your attack surface by disabling the flavors endpoints completely on public-facing web heads, by setting  in zaqar.conf.

Queue flavors allow users to have different types of queues based on the storage capabilities. By using flavors, it's possible to allow consumers of the service to choos between durable storage, fast storage, etc. Flavors must be created by service administrators and they rely on the existence of pools.

A flavor entry might look like:

{ "pool": "", "capabilities": { "durable": true } }

Create a Flavor
Template

PUT /v1.1/flavors/{flavor}

Request

PUT /v1.1/flavors/wat HTTP/1.1 Host: zaqar.example.com

{ "pool": "my_pool", "capabilities": { "durable": true } }

Response

HTTP/1.1 201 Created Location: /v1.1/flavors/wat

Discussion

Create a flavor

is the name to give the queue flavor entry. The name MUST NOT exceed 64 bytes in length, and is limited to US-ASCII letters, digits, underscores and hyphens.

is the name of the pool this flavor sits on top of.

An optional request component that describes flavor-specific capabilities. These capabilities ought to describe what this flavor is capable of base on the storage capabilities. They are used to inform the final user such capabilities.

Get Flavor Info
Template

GET /v1.1/flavors/{flavor}?detailed=True

Request

GET /v1.1/flavors/wat HTTP/1.1 Host: zaqar.example.com

Response

HTTP/1.1 200 OK Content-Location: /v1.1/flavors/wat

{ "pool": "my_pool", }

Discussion

Returns information on a registered pool.

Delete a Flavor
Template

DELETE /v1.1/flavors/{flavor}

Request

DELETE /v1.1/flavors/wat HTTP/1.1 Host: zaqar.example.com

Response

HTTP/1.1 204 No Content

Discussion

Removes a queue flavor from the registry.

Update a Flavor
Template

PATCH /v1.1/flavors/{flavor}

Request

PATCH /v1.1/flavors/wat HTTP/1.1 Host: zaqar.example.com

{ "pool": "my_new_pool", }

Response

HTTP/1.1 204 No Content

Discussion

Allows one to update any or all of: `pool`, `capabilities`. At least one of these fields must be specified, else, a HTTP 400 is returned.

List Flavors
Template

GET /v1.1/flavors?detailed=True&limit=10&marker=taco

Request

GET /v1.1/flavors HTTP/1.1 Host: zaqar.proxy.example.com

Response

HTTP/1.1 200 OK Content-Location: /v1.1/flavors

{ "links": [ {     "rel": "next", "href": "/v1.1/flavors?marker=wot&detailed=True&limit=10" ], "flavors":[ {"href": "/v1.1/flavors/wat", "pool": "pool1", "capabilities": {}}, {"href": "/v1.1/flavors/wot", "pool": "pool1", "capabilities": {"durable": true}} ] }

or if no flavors exist:

HTTP/1.1 200 OK Content-Location: /v1.1/flavors

{ "links": [ {     "rel": "next", "href": "/v1.1/flavors?marker=bar&detailed=True&limit=10" ], "flavors":[] }

Discussion

Lists the registered flavors.

if True, returns the capabilities field in the listing. used for pagination - what flavor do we start listing from? how many entries to return per request?