Zaqar/specs/api/v1.1
Contents
Zaqar API v1.1
Notes
Feedback on the v1.1 spec is welcome (find us in #openstack-zaqar, suggestions tracked on this etherpad). We have also started collecting feedback with an eye toward extensions and v2 of the API here: Zaqar/specs/api/next
TODO
- Document version discovery
Overview
Zaqar provides an HTTP-based API in the spirit of the REST architecture style.
This guide assumes the reader is familiar with REST, HTTP/1.1 and JSON. URI templates use the same syntax as RFC 6570.
Note: Error responses are enumerated on a separate page: Marconi/specs/api/v1.1/errors
Changes from v1.0
BREAKING
- Queue existence check endpoint (via HEAD & GET) no longer exists.
- The
X-Project-Id
header is now required for every request. This was done in anticipation of identity allowing a single user to manage multiple projects, in which case we will need to know which project this user is requesting (vs. assuming 1:1 mapping between auth token and project). It also affords rate limiting at the LB. - The
Client-ID
header is now required for every request. - The
Content-Location
header is no longer set in responses - The
partial
field is no longer included in responses to a message posting request. When using the official SQL and MongoDB drivers, Zaqar will either succeed in enqueuing all messages in the submitted batch, or fail to enqueue any of them. Any backend driver that supports batch message posting will need to guarantee atomic inserts of the entire batch (v2.0 of the API will define some operations as optional, such as batch message posting). - When a collection resource is requested (i.e., queues, messages) and the collection is empty, an empty JSON array is now returned in the response body, rather than HTTP 204 No Content.
- All top-level JSON arrays in response bodies are now encapsulated within an object. This mitigates a security risk we will have when we later implement support for accessing Zaqar directly from web apps (JavaScript).
- When posting one or more messages, the list of messages is now encapsulated in a JSON object type. This was done to be consistent with response bodies.
- Health is now an admin-only endpoint and returns operational stats for a given node. A new "ping" endpoint was added for load balancers to use.
NON-BREAKING
- MessagePack is now supported in addition to JSON
- Extraneous path elements from relative URIs were removed
- Version discovery is now available by accessing the root URI path
- The
Location
header, when present, now specifies a full (i.e., non-relative) URI - A
claim
field has been added to messages. It is omitted if the message is not claimed. - An
id
field has been added to messages - Message href's now always include
claim_id
in the query string if the message is claimed. - When posting messages or creating a claim, the
ttl
andgrace
fields may be omitted. If it is not given by the client, Zaqar will default to the max values configured by the cloud operator. - New pools API and capability
- Added new "pop" semantics for claiming and deleting messages in a single request. Note that this should only be used when an application is OK with the risk of missing a message when a worker crashes.
- The server will automatically create a queue when posting a message, if the queue does not already exist.
- Queries for messages and stats in non-existing queues now return empty lists, etc. rather than 404s.
Common API Elements
Clients
- Clients should follow HTTP redirects
- Clients should advertise gzip support
- Clients should identify themselves in the UserAgent request header; e.g., "User-Agent: python/2.7 cloudthing/1.2"
- Clients should not hard-code URI paths or templates since they may change over time. Instead, clients should cache links and link templates provided by the API.
API Versioning
The Zaqar API uses a URI versioning scheme. The first element of the path contains the target version identifier, e.g.:
https://marconi.example.com/v1.1
The URI version will only be incremented to accommodate major new features or API redesigns that can not be made backwards-compatible. When new API versions are released, older versions are deprecated. Zaqar maintainers will work with developers and partners to ensure there is adequate time to migrate to new versions before deprecated ones are discontinued.
Since the version is only incremented to accommodate major API revisions, sub-versioning of the API will be rare.
Resource Media Types
The API supports `application/json`, encoded as UTF-8. MessagePack is also supported ('application/x-msgpack'), and is the recommended default for client libraries.
Unrecognized protocol elements received from the server should simply be ignored. This includes JSON object properties, link relation types, media types, etc.
Authentication
May be optionally implemented by middleware or reverse proxy. If implemented, the following applies:
All requests to the API may only be performed by an authenticated agent.
To authenticate, an agent issues an authentication request to a Keystone Identity Service endpoint.
In response to valid credentials, Keystone responds with an auth token and a service catalog that contains a list of all services and endpoints available for the given token. Multiple endpoints may be returned for Zaqar according to physical locations and performance/availability characteristics of different deployments.
Normally, Keystone middleware provides the X-Project-Id header based on the auth token submitted by the Zaqar client. For this to work, clients MUST specify a valid auth token in the `X-Auth-Token` header for each request to the Zaqar API. The API validates auth tokens against Keystone before servicing each request
If auth is not enabled, clients must provide the X-Project-Id header themselves.
Authorization
May be optionally implemented by middleware or reverse proxy. If implemented, the following applies:
The API needs to verify read/write access to all endpoints according to the provided auth token (RBAC). The ACL should be cached with the token.
Endpoints
Clients may choose to interact with any of the service endpoints retrieved from Keystone. Agents use the given endpoint as a home document href from which to discover links to all other API requests. Endpoints may be organized by region
An example endpoint:
https://marconi.example.com/v1.1
Errors
If any request results in an error, the server will return an appropriate 4xx or 5xx HTTP status code, and optionally the following information in the body:
- Title
- Description
- Internal code
- Link to more information
Example:
HTTP/1.1 400 Bad Request Content-Type: application/json { "title": "Unsupported limit", "description": "The given limit cannot be negative, and cannot be greater than 50.", "code": 1092, "link": { "rel": "help", "href": "http://docs.example.com/messages#limit", "text": "API documentation for the limit parameter" } }
Common Headers
Each request to the API must include certain standard and extended HTTP headers. These headers provide host, agent, authentication and other pertinent information to the server.
Header | Description |
Host | The host name of the API, as referenced by the Keystone service catalog |
Date | The current date and time, using the standard RFC 1123 HTTP date format |
Accept | Media type desired; only application/json is supported at this time.
|
Accept-Encoding | Specifies that the agent accepts gzip-encoded response bodies. |
Content-Length | For POST or PUT requests, the length in bytes of the message document being submitted |
X-Auth-Token | Keystone auth token, required in every request when auth is enabled |
X-Project-Id | An ID for a project to which the value of X-Auth-Token grants access. Queues will be created under this project. This header is required in every request for multi-tenant deployments. |
Client-ID | A UUID, used to distinguish each client using the service. The UUID must be submitted in its canonical form (e.g., 3381af92-2b9e-11e3-b191-71861300734c ). In Zaqar, this is used to avoid echoing a sender's messages back to the same instance, and may be logged by the server to provide diagnostics and statistics. Should be generated once and persisted between restarts of the client to avoid the race condition where a client sends a message, restarts with a different ID, and then receives its own message, echoed back.
|
HTTP Response Codes
See Response Codes
Sample API Request
GET /v1.1/queues/fizbat/messages?marker=1355-237242-783&limit=10 HTTP/1.1 Host: marconi.example.com User-Agent: python/2.7 killer-rabbit/1.2 Date: Wed, 28 Nov 2012 21:14:19 GMT Accept: application/json Accept-Encoding: gzip X-Auth-Token: 7d2f63fd-4dcc-4752-8e9b-1d08f989cc00 X-Project-Id: 518b51ea133c4facadae42c328d6b77b Client-ID: 30387f00-39a0-11e2-be4d-a8d15f34bae2
Endpoints Synopsis
# Base endpoints GET /v1.1 GET /v1.1/health GET /v1.1/ping # Queues GET /v1.1/queues{?marker,limit,detailed} PUT /v1.1/queues/{name} DELETE /v1.1/queues/{name} # Queue stats GET /v1.1/queues/{queue_name}/stats # Messages GET /v1.1/queues/{queue_name}/messages{?marker,limit,echo,include_claimed} POST /v1.1/queues/{queue_name}/messages DELETE /v1.1/queues/{queue_name}/messages/{message_id}{?claim_id} DELETE /v1.1/queues/{queue_name}/messages{?ids} DELETE /v1.1/queues/{queue_name}/messages{?pop} # Claims POST /v1.1/queues/{queue_name}/claims{?limit} GET /v1.1/queues/{queue_name}/claims/{claim_id} PATCH /v1.1/queues/{queue_name}/claims/{claim_id} DELETE /v1.1/queues/{queue_name}/claims/{claim_id} # Pools GET /v1.1/pools?detailed=False&marker=None&limit=10 GET /v1.1/pools/{pool}?detailed=False PUT /v1.1/pools/{pool} DELETE /v1.1/pools/{pool} PATCH /v1.1/pools/{pool}
Base Endpoints
Get Home Document
Template
GET /v1.1
Request
GET /v1.1 HTTP/1.1 Host: marconi.example.com ...
Response
HTTP/1.0 200 OK Cache-Control: max-age=86400 Content-Length: 4345 Content-Type: application/json-home Date: Tue, 06 Aug 2013 16:31:48 GMT Server: WSGIServer/0.1 Python/2.7.3 { "resources": { "rel/queue-stats": { "href-template": "/v1.1/queues/{queue_name}/stats", "href-vars": { "queue_name": "param/queue_name" }, "hints": { "allow": [ "GET" ], "formats": { "application/json": {} } } }, "rel/post-messages": { "href-template": "/v1.1/queues/{queue_name}/messages", "href-vars": { "queue_name": "param/queue_name" }, "hints": { "accept-post": [ "application/json" ], "allow": [ "POST" ], "formats": { "application/json": {} } } }, "rel/queue": { "href-template": "/v1.1/queues/{queue_name}", "href-vars": { "queue_name": "param/queue_name" }, "hints": { "allow": [ "PUT", "DELETE" ], "formats": { "application/json": {} } } }, "rel/queues": { "href-template": "/v1.1/queues{?marker,limit,detailed}", "href-vars": { "marker": "param/marker", "detailed": "param/detailed", "limit": "param/queue_limit" }, "hints": { "allow": [ "GET" ], "formats": { "application/json": {} } } }, "rel/messages": { "href-template": "/v1.1/queues/{queue_name}/messages{?marker,limit,echo,include_claimed}", "href-vars": { "marker": "param/marker", "include_claimed": "param/include_claimed", "queue_name": "param/queue_name", "limit": "param/messages_limit", "echo": "param/echo" }, "hints": { "allow": [ "GET" ], "formats": { "application/json": {} } } }, "rel/messages-delete": { "href-template": "/v1.1/queues/{queue_name}/messages{?ids,pop}", "href-vars": { "ids": "param/ids", "pop": "param/pop", }, "hints": { "allow": [ "DELETE" ], "formats": { "application/json": {} } } }, "rel/claim": { "href-template": "/v1.1/queues/{queue_name}/claims{?limit}", "href-vars": { "queue_name": "param/queue_name", "limit": "param/claim_limit" }, "hints": { "accept-post": [ "application/json" ], "allow": [ "POST" ], "formats": { "application/json": {} } } } } }
Discussion
The entire Zaqar API is discoverable from a single starting point; agents do not need to know any more than this one URI in order to explore the entire API.
This document is cache-able (and should be cached by proxy and client).
See also: http://tools.ietf.org/html/draft-nottingham-json-home-03
Check Node Health
Template
GET /v1.1/health
or
HEAD /v1.1/health
Request
GET /v1.1/health HTTP/1.1 Host: example.marconi.com ...
Response
{ "mongo_pool_1": { "message_volume": { "claimed": 0, "total": 0, "free": 0 }, "storage_reachable": true, "operation_status": { "create_queue": { "seconds": 0.0021300315856933594, "ref": null, "succeeded": true }, "post_messages": { "seconds": 0.033502817153930664, "ref": null, "succeeded": true }, "list_messages": { "seconds": 0.000013113021850585938, "ref": null, "succeeded": true }, "claim_messages": { "seconds": 0.0013759136199951172, "ref": "3f515f37-58a0-4c81-8214-3e92979b82e7", "succeeded": false }, "delete_queue": { "seconds": 0.0030739307403564453, "ref": null, "succeeded": true } } }, "mongo_pool_2": { "message_volume": { "claimed": 0, "total": 0, "free": 0 }, "storage_reachable": true, "operation_status": { "create_queue": { "seconds": 0.0011799335479736328, "ref": null, "succeeded": true }, "post_messages": { "seconds": 0.024316072463989258, "ref": null, "succeeded": true }, "list_messages": { "seconds": 0.000008106231689453125, "ref": null, "succeeded": true }, "claim_messages": { "seconds": 0.000576019287109375, "ref": "68629fda-b4ce-4cf9-978a-df0df8df36a7", "succeeded": false }, "delete_queue": { "seconds": 0.003300905227661133, "ref": null, "succeeded": true } } }, "catalog_reachable": true }
Discussion
This is an operator endpoint and should not be implemented in client libraries intended for end users. It should also not be included in the json-home document unless the requesting user is an administrator.
Use this request to get detailed operational stats for a specific Zaqar node.
Ping a Node
Template
GET /v1.1/ping
or
HEAD /v1.1/ping
Request
GET /v1.1/ping HTTP/1.1 Host: example.marconi.com ...
Response
HTTP/1.1 204 No Content
or
HTTP/1.1 503 Service Unavailable
Discussion
This is an operator endpoint and should not be implemented in client libraries intended for end users. It should also not be included in the json-home document unless the requesting user is an administrator.
Use this resource to check whether a given Zaqar node is online. If the node is down, this endpoint will not be reachable and will act as a signal to the load balancer to take the node out of rotation. Alternatively, the endpoint may be reachable but the service is temporarily unavailable due to storage driver failure or some other error.
The health endpoint may be accessed only if the X-Forwarded-For header is present. In that case, it will not require auth, allowing load balancers to query a given node's health.
For requests NOT sent via the LB, Zaqar will return 404 Not Found, since this endpoint is not meant to be accessed by end users and there is no point in exposing another potential DDoS vector to the world.
Queues
Get Queue
Template
GET /v1.1/queues/{queue_name}
Request
GET /v1.1/queues/fizbit HTTP/1.1 Host: marconi.example.com ...
Response
HTTP/1.1 200 OK
{ "key": { "key2": "value", "key3": [1, 2, 3, 4, 5] } }
Discussion
Returns queue metadata.
Create Queue
Template
PUT /v1.1/queues/{queue_name} { ... }
Request
PUT /v1.1/queues/fizbat HTTP/1.1 Host: marconi.example.com
Response
HTTP/1.1 201 Created Location: https://marconi.example.com/v1.1/queues/fizbit
Discussion
Creates a queue.
The body of the request is an arbitrary document which allows storing contextual information about the way a particular application should interact with the queue. The size of this body, in characters and including whitespace must be <= 64 KiB (configurable). The document MUST be valid JSON.
queue_name
is the name to give the queue. The name MUST NOT exceed 64 bytes in length, and is limited to US-ASCII letters, digits, underscores and hyphens.
List Queues
Template
GET /v1.1/queues{?marker,limit,detailed}
Request
GET /v1.1/queues?marker=baz&detailed=true HTTP/1.1 Host: marconi.example.com ...
Response
HTTP/1.1 200 OK { "links": [ { "rel": "next", "href": "queues?marker=kooleo&detailed=true" } ], "queues": [ { "name": "boomerang", "href": "queues/boomerang" }, { "name": "fizbit", "href": "queues/fizbit" }, ... { "name": "kooleo", "href": "queues/kooleo" } ] }
or, if no queues exist:
HTTP/1.1 200 OK { "links": [ { "rel": "next", "href": "queues?marker=baz&detailed=true" } ], "queues": [] }
Discussion
Query parameters are defined as follows:
marker
name of the last queue returned in a previous response, used to retrieve the next page of results. NOTE: Clients should normally just follow the "next" link rather than attempting to construct URI paths manually.
limit
Maximum number of queue records to return. May not be more than the hard maximum configured by the cloud operator (default 20). If not given, the limit defaults to 10.
detailed
Set to "true" to inline queue metadata in the listing (default false).
Delete Queue
Template
DELETE /v1.1/queues/{queue_name}
Request
DELETE /v1.1/queues/fizbat HTTP/1.1 Host: marconi.example.com
Response
HTTP/1.1 204 No Content
Discussion
Use this operation to immediately delete a queue along with all its messages (if any).
Queue Stats
Template
GET /v1.1/queues/{queue_name}/stats
Request
GET /v1.1/queues/fizbit/stats HTTP/1.1 Host: marconi.example.com ...
Response
HTTP/1.1 200 OK
{ "messages": { "free": 146929, "claimed": 2409, "total": 149338, "oldest": { "href": "queues/fizbit/messages/50b68a50d6f5b8c8a7c62b01", "age": 63, "created": "2013-08-12T20:44:55Z" }, "newest": { "href": "queues/fizbit/messages/50b68a50d6f5b8c8a7c62b01", "age": 12, "created": "2013-08-12T20:45:46Z" } } }
Discussion
Returns queue statistics, including how many messages are in the queue, broken out by status.
NOTE: If total is 0, then "oldest" and "newest" message stats are not included.
Messages
All message-related operations require Client-Id
to be included in the headers. This is to ensure that messages are not echoed back to the client that posted them unless the client explicitly requests this.
List Messages
Template
GET /v1.1/queues/{queue_name}/messages{?marker,limit,echo,include_claimed}
Request
GET /v1.1/queues/fizbit/messages?marker=1355-237242-783&limit=10 HTTP/1.1 Host: marconi.example.com ...
Response
HTTP/1.1 200 OK ... { "links": [ { "rel": "next", "href": "messages?marker=6244-244224-783&limit=10" } ], "messages": [ { "href": "messages/50b68a50d6f5b8c8a7c62b01", "id": "50b68a50d6f5b8c8a7c62b01", "ttl": 120, "age": 53, "body": { "event": "ActivateAccount", "mode": "active" } }, { "href": "messages/50b68a50d6f5b8c8a7c62b02?claim_id=06ef2372-6746-11e3-b311-b3adcbe406e9", "id": "50b68a50d6f5b8c8a7c62b01", "ttl": 800, "age": 790, "body": { "event": "CreateInvoice", "customer_id": "90fd8734-6746-11e3-be3c-7e45c531c7ca" } } ] }
or, if no messages are available:
HTTP/1.1 200 OK ... { "links": [ { "rel": "next", "href": "messages?marker=6244-244224-783&limit=10" } ], "messages": [] }
Discussion
Message IDs and markers are opaque strings; clients should make no assumptions about their format or length. Furthermore, clients should assume there is no relationship between markers and message IDs (i.e., one cannot be derived from the other). This allows for a wide variety of storage driver implementations.
Results are ordered by age, oldest message first.
limit
specifies up to 20 messages (configurable) to return. If not specified, limit defaults to 10. When more messages are available than can be returned in a single request, the client can pick up the next batch by simply using the URI template parameters returned from the previous call in the "next" field (TBD).
marker
is an opaque string that the client can use to request the next batch of messages. It communicates to the server which messages the client has already received.
Note: If marker
is not specified, the API will return all messages at the head of the queue (up to limit).
echo
is a boolean (i.e., "true" or "false") that determines whether or not the API will return a client's own messages, as determined by the uuid
portion of the Client-ID header. If not specified, echo defaults to "false".
include_claimed
is a boolean (i.e., "true" or "false") that determines whether or not the API will return claimed messages as well as unclaimed ones. If not specified, defaults to "false" (only unclaimed messages are returned).
Get a Specific Message
Template
GET /v1.1/queues/{queue_name}/messages/{message_id}
Request
GET /v1.1/queues/fizbat/messages/50b68a50d6f5b8c8a7c62b01 HTTP/1.1 Host: marconi.example.com ...
Response
HTTP/1.1 200 OK Content-Location: /v1.1/queues/fizbat/messages/50b68a50d6f5b8c8a7c62b01 ... { "href": "/v1.1/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b01", "id": "50b68a50d6f5b8c8a7c62b01", "ttl": 800, "age": 790, "body": { "event": "ActivateAccount", "mode": "active" } }
Discussion
Message fields are defined as follows:
href
is an opaque relative URI that the client can use to uniquely identify a message resource, and interact with it.
ttl
is the ttl set on the message when it was posted. The message will expire after (ttl - age) seconds.
age
number of seconds since ts, relative to the server's clock.
body
Arbitrary document submitted along with the original request to post the message.
If either the message ID or the claim ID is malformed or nonexistent, no message is returned.
Get a Set of Messages by ID
Template
GET /v1.1/queues/{queue_name}/messages{?ids}
Request
GET /v1.1/queues/fizbat/messages?ids=50b68a50d6f5b8c8a7c62b01,f5b8c8a7c62b0150b68a50d6 HTTP/1.1 Host: marconi.example.com ...
Response
HTTP/1.1 200 OK Content-Location: /v1.1/queues/fizbat/messages?ids=50b68a50d6f5b8c8a7c62b01,f5b8c8a7c62b0150b68a50d6 ... { "messages": [ { "href": "/v1.1/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b01", "id": "50b68a50d6f5b8c8a7c62b01", "ttl": 800, "age": 32, "body": { "cmd": "EncodeVideo", "jobid": 58229 } }, { "href": "/v1.1/queues/fizbit/messages/f5b8c8a7c62b0150b68a50d6", "id": "f5b8c8a7c62b0150b68a50d6", "ttl": 800, "age": 790, "body": { "cmd": "EncodeAudio", "jobid": 58201 } } ] }
Discussion
Unlike message listing, a client's own messages are always returned in this approach. Note that the list of ids may not exceed 20 (configurable, shared setting with number of messages that can be posted at once).
This approach returns all the messages with matching IDs. If a malformed ID or a nonexistent message ID is provided, it is ignored and the remaining messages are returned.
Post Message(s)
Template
POST /v1.1/queues/{queue_name}/messages
Request
POST /v1.1/queues/fizbit/messages HTTP/1.1 Host: marconi.example.com ... { "messages": [ { "ttl": 300, "body": { "event": "BackupStarted", "backup_id": "c378813c-3f0b-11e2-ad92-7823d2b0f3ce" } }, { "body": { "event": "BackupProgress", "current_bytes": "0", "total_bytes": "99614720" } } ] }
Responses
When a single message is submitted:
HTTP/1.1 201 Created Location: https://marconi.example.com/v1.1/queues/fizbit/messages?ids=50b68a50d6f5b8c8a7c62b01 Content-Type: application/json { "links": [ { "rel": "rel/message", "href": "messages/50b68a50d6f5b8c8a7c62b01" } ] }
When multiple messages are submitted:
HTTP/1.1 201 Created Location: https://marconi.example.com/v1.1/queues/fizbit/messages?ids=50b68a50d6f5b8c8a7c62b01,50b68a50d6f5b8c8a7c62b05 Content-Type: application/json { "links": [ { "rel": "rel/message", "href": "messages/50b68a50d6f5b8c8a7c62b01" }, { "rel": "rel/message", "href": "messages/50b68a50d6f5b8c8a7c62b05" } ] }
Discussion
Up to 20 messages (default, but configurable) may be submitted in a single request, but must always be encapsulated in a collection container (e.g., an array in JSON). The resulting value of the Location header or response body may be used to retrieve the created messages for further processing if needed.
The client only specifies the body and ttl for the message; the server will insert metadata such as id and age.
Note: If the specified queue does not already exist, the server will create it. Applications do not need to explicitly manage the lifetimes of their queues, but are encouraged to delete queues that are no longer in use.
The response body contains a list of resource paths corresponding to each message submitted in the request, in the same order. In the case of a server-side error part-way through the processing of the submitted messages, the entire batch will be abandoned (all-or-nothing), and the client will have to retry its request.
The size of the request document, in characters including whitespace, is 256 KiB by default (configurable). The document MUST be a well-formed JSON or MessagePack document (Zaqar will validate).
body
specifies an arbitrary document which constitutes the body of the message being sent.
ttl
is how long the server should wait before expiring and removing the message from the queue. Value MUST be between 60 and 1209600 seconds (14 days, configurable), inclusive. Note that the server may not actually delete the message until it's age has reached up to (ttl + 60) seconds, to allow for flexibility in storage implementations. If the ttl field is omitted from the request, the default value is used, which is normally 3600 seconds unless customized by the service provider.
Delete Message
Template
DELETE /v1.1/queues/{queue_name}/messages/{message_id}{?claim_id}
Request
DELETE /v1.1/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b01 HTTP/1.1 Host: marconi.example.com ...
Response
HTTP/1.1 204 No Content
Discussion
message_id
specifies the message to delete.
claim_id
specifies that the message should only be deleted if it has the specified claim ID, and that claim has not expired. This is useful for ensuring only one agent processes any given message; whenever a worker client's claim expires before it has a chance to delete a message it has processed, the worker must roll back any actions it took based on that message, since another worker will now be able to claim and process the same message.
Note that if claim_id is not specified, but the message is claimed, the operation will fail. In other words, claimed messages can only ever be deleted by providing an appropriate claim_id.
Delete Multiple Messages
Template
DELETE /v1.1/queues/{queue_name}/messages{?ids,pop}
Request
DELETE /v1.1/queues/fizbit/messages?ids=50b68a50d6f5b8c8a7c62b01,50b68a50d6f5b8c8a7c62b02 HTTP/1.1 Host: marconi.example.com ...
Response
HTTP/1.1 204 No Content
or, to atomically pop several messages (when only 2 messages are available):
Request
DELETE /v1.1/queues/fizbit/messages?pop=5 HTTP/1.1 Host: marconi.example.com ...
Response
HTTP/1.1 200 OK Content-Type: application/json ... { "messages": [ { "href": "/v1.1/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b01?claim_id=a28ee94e-6cb4-11e2-b4d5-7703267a7926", "id": "50b68a50d6f5b8c8a7c62b01", "ttl": 800, "age": 100, "body": { "object_id": "8a50d6", "target": "h.264" } }, { "href": "/v1.1/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b02?claim_id=a28ee94e-6cb4-11e2-b4d5-7703267a7926", "id": "50b68a50d6f5b8c8a7c62b02", "ttl": 800, "age": 790, "body": { "object_id": "fb8c8a", "target": "h.264" } } ] }
pop & ids parameters are mutually exclusive. Using them together in a request will result in HTTP 400.
Request
DELETE /v1.1/queues/fizbit/messages?pop=5&ids=50b68a50d6f5b8c8a7c62b02 HTTP/1.1 Host: marconi.example.com ...
Response
HTTP/1.1 400 Bad Request Content-Type: application/json ... { "message": pop and id params cannot be present together in the delete request }
Discussion
Bulk delete for messages.
ids
specifies the messages to delete, up to a maximum of 20 (the limit is configurable, shares the same setting as the maximum for listing messages)
pop
specifies a certain number of messages to pop off the queue, which is equivalent to claiming and deleting those messages atomically (therefore guaranteeing once-and-only-once delivery). The maximum value for this parameter is the same as for the maximum number of messages that can be claimed in a single operation. Note that if the worker crashes after popping the messages, but before processing them, no other workers will be able to reclaim those messages. Therefore, the "pop" operation should only be used when the risk of losing an occasional message can be justified.
Note that the ids
and pop
parameters are mutually exclusive. In other words, if both params are present the server will respond with a "400 Bad Request" status.
If any of the message IDs are malformed or non-existent, they are ignored. The remaining valid message IDs are deleted.
WARNING: Claimed messages will not be skipped; they will be deleted along with unclaimed messages. If using the worker pool pattern, in which workers are claiming batches of messages, we encourage deleting messages one at a time. This ensures that at most one message could be processed, but not deleted, if a worker happens to crash at just the wrong moment:
for each message in batch: process message delete message
As opposed to:
for each message in batch: process message bulk-delete batch
Claims
Claim Messages
Template
POST /v1.1/queues/{queue_name}/claims{?limit} Content-Type: application/json ... { "ttl": {claim_ttl}, "grace": {message_grace} }
Request
POST /v1.1/queues/fizbit/claims?limit=5 HTTP/1.1 Host: marconi.example.com Content-Type: application/json Accept: application/json ... { "ttl": 300, "grace": 300 }
Or, to use the default values for ttl and grace, one or both fields may be omitted in the request:
POST /v1.1/queues/fizbit/claims?limit=5 HTTP/1.1 Host: marconi.example.com Content-Type: application/json Accept: application/json ... { "ttl": 300, }
NOTE: If omitting both fields, the entire request body may be omitted (although submitting an empty JSON object is also acceptable):
POST /v1.1/queues/fizbit/claims?limit=5 HTTP/1.1 Host: marconi.example.com Content-Length: 0 Accept: application/json
Response
The client receives a claim URI and a list of claimed messages, if any:
HTTP/1.1 201 Created Content-Type: application/json Location: https://marconi.example.com/v1.1/queues/fizbit/claims/a28ee94e-6cb4-11e2-b4d5-7703267a7926 ... { "messages": [ { "href": "/v1.1/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b01?claim_id=a28ee94e-6cb4-11e2-b4d5-7703267a7926", "id": "50b68a50d6f5b8c8a7c62b01", "ttl": 800, "age": 100, "body": { "object_id": "8a50d6", "target": "h.264" } }, { "href": "/v1.1/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b02?claim_id=a28ee94e-6cb4-11e2-b4d5-7703267a7926", "id": "50b68a50d6f5b8c8a7c62b02", "ttl": 800, "age": 790, "body": { "object_id": "fb8c8a", "target": "h.264" } } ] }
Or, if no messages are available to claim:
HTTP/1.1 201 Created Content-Type: application/json Location: https://marconi.example.com/v1.1/queues/fizbit/claims/f9151272-673e-11e3-8e72-43d6a24410d2 ... { "messages": [] }
Discussion
Claims a set of messages, up to limit, from oldest to newest, skipping any that are already claimed. If no unclaimed messages are available, returns 204 No Content.
The client should delete the message when it has finished processing it, before the claim expires, to ensure the message is processed only once. As part of the delete operation, all worker clients should specify the claim ID (this is best done by simply using the provided href). That way, the server can return an error if the claim just expired (notifying the client of the race condition), giving the worker a chance to roll back its own processing of the given message, since another worker will likely claim the message and process it.
Just as with a message's age, the age given for the claim is relative to the server's clock, and is useful for determining how quickly messages are getting processed, and whether a given message's claim is about to expire.
When a claim expires, it is removed, allowing another client worker to claim the message in the case that the original worker fails to process it.
limit
specifies up to 20 messages (configurable) to claim. If not specified, limit defaults to 10. Note that claim creation is best-effort, meaning the server may claim and return less than the requested number of messages.
ttl
is how long the server should wait before releasing the claim. Value MUST be between 60 and 43200 seconds (12 hours, configurable). If omitted, the default value is used, which is normally 300 seconds unless customized by the service provider.
grace
is the message grace period in seconds. Value MUST be between 60 and 43200 seconds (12 hours, configurable). The server will extend the lifetime of claimed messages to be at least as long as the lifetime of the claim itself, plus a specified grace period to deal with crashed workers (up to 1209600 or 14 days including claim lifetime). This gives other workers a chance to reclaim the message after the previous claim expires, but before the message itself expires. Note that If a message's lifetime already extends beyond the claim's TTL plus grace period, the message's lifetime will not be adjusted. If ommitted, the default value is used, which is normally 60 seconds unless customized by the service provider.
Query Claim
Template
GET /v1.1/queues/{queue_name}/claims/{claim_id}
Request
GET /v1.1/queues/foo-bar/claims/a28ee94e-6cb4-11e2-b4d5-7703267a7926 Host: marconi.example.com ...
Response
HTTP/1.1 200 OK ... { "age": 19, "ttl": 30, "messages": [ ... ] }
Discussion
Renew Claim
Template
PATCH /v1.1/queues/{queue_name}/claims/{claim_id} Content-Type: application/json
Request
PATCH /v1.1/queues/fizbit/claims/a28ee94e-6cb4-11e2-b4d5-7703267a7926 HTTP/1.1 Host: marconi.example.com Content-Type: application/json ... { "ttl": 300 }
Response
HTTP/1.1 204 No Content
Discussion
Clients should periodically renew claims during long-running batches of work to avoid losing a claim in the middle of processing a message. This is done by issuing a PATCH to a specific claim resource and including a new TTL for the claim (which may be different from the original TTL). The server will reset the age of the claim and apply the new TTL.
Release Claim
Template
DELETE /v1.1/queues/{queue_name}/claims/{claim_id}
Request
DELETE /v1.1/queues/foo-bar/claims/a28ee94e-6cb4-11e2-b4d5-7703267a7926 HTTP/1.1 Host: marconi.example.com
Response
HTTP/1.1 204 No Content
Discussion
Use this operation to immediately release a claim, making any (remaining, non-deleted) messages associated with the claim available to other workers.
This operation is useful when a worker is performing a graceful shutdown, fails to process one or more messages, or is taking longer than expected to process messages, and wishes to make the remainder available to other workers.
Message Store Pools
WARNING: in a production deployment, the storage pools API should be restricted to administrators through role-based authentication (e.g., via policy middleware). You can also reduce your attack surface by disabling the pools endpoints completely on public-facing web heads, by setting admin_mode = False
in zaqar.conf.
Zaqar supports heterogeneous pools through the use of:
- stevedore for dynamic storage driver loading
- node weights
As long as an entry point is defined in an installed module that matches the scheme of a pool connection URI, Zaqar will be able to use that pool. For example, a pool entry might look like:
{ "weight": 100, "uri": "mongodb://localhost:27017", "options": { "max_retry_sleep": 1 } }
Register a Pool
Template
PUT /v1.1/pools/{pool}
Request
PUT /v1.1/pools/wat HTTP/1.1 Host: marconi.example.com { "weight": 100, "uri": "mongodb://localhost:27017", "options": { "max_retry_sleep": 1, "partitions": 8 } }
Response
HTTP/1.1 201 Created Location: /v1.1/pools/wat
Discussion
Registers a pool.
pool
is the name to give the storage pool entry. The name MUST NOT exceed 64 bytes in length, and is limited to US-ASCII letters, digits, underscores and hyphens.
weight
is the likelihood that this pool will be selected for the next queue allocation. It must be an integer greater than -1.
uri
is a connection string compatible with a storage client (e.g., pymongo) attempting to connect to that pool.
options
An optional request component that gives storage-specific options used by storage driver implementations. Valid parameters come from the registered options for a given storage backend, for example: mongodb, sqlite
Get Pool Info
Template
GET /v1.1/pools/{pool}?detailed=True
Request
GET /v1.1/pools/wat HTTP/1.1 Host: marconi.example.com
Response
HTTP/1.1 200 OK Content-Location: /v1.1/pools/wat { "uri": "mongodb://marconi1.example.com:27017", "weight": 100 }
Discussion
Returns information on a registered pool.
Delete a Pool
Template
DELETE /v1.1/pools/{pool}
Request
DELETE /v1.1/pools/wat HTTP/1.1 Host: marconi.example.com
Response
HTTP/1.1 204 No Content
Discussion
Removes a storage pool from the registry.
Update a Pool
Template
PATCH /v1.1/pools/{pool}
Request
PATCH /v1.1/pools/wat HTTP/1.1 Host: marconi.example.com { "uri": "mongodb://marconi3.example.com:27018", "weight": 120 }
Response
HTTP/1.1 204 No Content
Discussion
Allows one to update any or all of: `weight`, `uri`, `options`. At least one of these fields must be specified, else, a HTTP 400 is returned.
List Pools
Template
GET /v1.1/pools?detailed=True&limit=10&marker=taco
Request
GET /v1.1/pools HTTP/1.1 Host: marconi.proxy.example.com
Response
HTTP/1.1 200 OK Content-Location: /v1.1/pools { "links": [ { "rel": "next", "href": "/v1.1/pools?marker=wot&limit=10&detailed=True } ], "pools": [ {"href": "/v1.1/pools/wat", "weight": 100, "uri": "mongodb://marconi1.example.com:27017"}, {"href": "/v1.1/pools/wot", "weight": 50, "uri": "redis://marconi2.example.com:6379"} ] }
or if no pools exist:
HTTP/1.1 200 OK Content-Location: /v1.1/pools { "links": [ { "rel": "next", "href": "/v1.1/pools?marker=bar&limit=10&detailed=True } ], "pools": [] }
Discussion
Lists the registered pools.
detailed
if True, returns the options field in the listing.
marker
used for pagination - what pool do we start listing from?
limit
how many entries to return per request?
Queue Flavors
WARNING: in a production deployment, the queue flavors API should be restricted to administrators through role-based authentication (e.g., via policy middleware). You can also reduce your attack surface by disabling the flavors endpoints completely on public-facing web heads, by setting admin_mode = False
in zaqar.conf.
Queue flavors allow users to have different types of queues based on the storage capabilities. By using flavors, it's possible to allow consumers of the service to choos between durable storage, fast storage, etc. Flavors must be created by service administrators and they rely on the existence of pools.
A flavor entry might look like:
{ "pool": "", "capabilities": { durable: true } }
Create a Flavor
Template
PUT /v1.1/flavors/{flavor}
Request
PUT /v1.1/flavors/wat HTTP/1.1 Host: zaqar.example.com { "pool": "my_pool", "capabilities": { "durable": true } }
Response
HTTP/1.1 201 Created Location: /v1.1/flavors/wat
Discussion
Create a flavor
flavor
is the name to give the queue flavor entry. The name MUST NOT exceed 64 bytes in length, and is limited to US-ASCII letters, digits, underscores and hyphens.
pool
is the name of the pool this flavor sits on top of.
capabilities
An optional request component that describes flavor-specific capabilities. These capabilities ought to describe what this flavor is capable of base on the storage capabilities. They are used to inform the final user such capabilities.
Get Flavor Info
Template
GET /v1.1/flavors/{flavor}?detailed=True
Request
GET /v1.1/flavors/wat HTTP/1.1 Host: zaqar.example.com
Response
HTTP/1.1 200 OK Content-Location: /v1.1/flavors/wat { "pool": "my_pool", }
Discussion
Returns information on a registered pool.
Delete a Flavor
Template
DELETE /v1.1/flavors/{flavor}
Request
DELETE /v1.1/flavors/wat HTTP/1.1 Host: zaqar.example.com
Response
HTTP/1.1 204 No Content
Discussion
Removes a queue flavor from the registry.
Update a Flavor
Template
PATCH /v1.1/flavors/{flavor}
Request
PATCH /v1.1/flavors/wat HTTP/1.1 Host: zaqar.example.com { "pool": "my_new_pool", }
Response
HTTP/1.1 204 No Content
Discussion
Allows one to update any or all of: `pool`, `capabilities`. At least one of these fields must be specified, else, a HTTP 400 is returned.
List Flavors
Template
GET /v1.1/flavors?detailed=True&limit=10&marker=taco
Request
GET /v1.1/flavors HTTP/1.1 Host: zaqar.proxy.example.com
Response
HTTP/1.1 200 OK Content-Location: /v1.1/flavors { "links": [ { "rel": "next", "href": "/v1.1/flavors?marker=wot&detailed=True&limit=10" ], "flavors":[ {"href": "/v1.1/flavors/wat", "pool": "pool1", "capabilities": {}}, {"href": "/v1.1/flavors/wot", "pool": "pool1", "capabilities": {"durable": true}} ] }
or if no flavors exist:
HTTP/1.1 200 OK Content-Location: /v1.1/flavors { "links": [ { "rel": "next", "href": "/v1.1/flavors?marker=bar&detailed=True&limit=10" ], "flavors":[] }
Discussion
Lists the registered flavors.
detailed
if True, returns the capabilities field in the listing.
marker
used for pagination - what flavor do we start listing from?
limit
how many entries to return per request?