Zaqar/specs/api/v1

= Marconi API: v1 Blueprint =

The Marconi v1 API is FROZEN

We have started collecting feedback with an eye toward extensions and v2 of the API here: Marconi/specs/api/next

Overview
Marconi provides an HTTP-based API in the spirit of the REST architecture style.

This guide assumes the reader is familiar with REST, HTTP/1.1 and JSON. URI templates use the same syntax as RFC 6570.

Note: Error responses are enumerated on a separate page: Marconi/specs/api/v1/errors

Clients

 * Clients should follow HTTP redirects
 * Clients should advertise gzip support
 * Clients must identify themselves in the UserAgent request header; e.g., "User-Agent: python/2.7 cloudthing/1.2"
 * Clients must not hard-code URI paths or templates since they may change over time. Instead, clients should cache links and link templates provided by the API.

API Versioning
The Marconi API uses a URI versioning scheme. The first element of the path contains the target version identifier, e.g.:

https://marconi.example.com/v1

The URI version will only be incremented to accommodate major new features or API redesigns that can not be made backwards-compatible. When new API versions are released, older versions are deprecated. Marconi maintainers will work with developers and partners to ensure there is adequate time to migrate to new versions before deprecated ones are discontinued.

Since the version is only incremented to accommodate major API revisions, sub-versioning of the API will be rare, but may be used occasionally to release a "polished" major version.

Resource Media Types
The API supports `application/json` (XML is on the road map), encoded as UTF-8. Gzip'd requests are optionally supported by the server.

Unrecognized protocol elements should be ignored. This includes XML elements and attributes, JSON object properties, link relation types, media types, etc.

Authentication
''May be optionally implemented by middleware or reverse proxy. If implemented, the following applies:''

All requests to the API may only be performed by an authenticated agent.

To authenticate, an agent issues an authentication request to a Keystone Identity Service endpoint.

In response to valid credentials, Keystone responds with an auth token and a service catalog that contains a list of all services and endpoints available for the given token. Multiple endpoints may be returned for Marconi according to physical locations and performance/availability characteristics of different deployments.

Normally, Keystone middleware provides the X-Project-Id header based on the auth token submitted by the Marconi client. For this to work, clients MUST specify a valid auth token in the `X-Auth-Token` header for each request to the Marconi API. The API validates auth tokens against Keystone before servicing each request

If auth is not enabled, clients must provide the X-Project-Id header themselves.

Authorization
''May be optionally implemented by middleware or reverse proxy. If implemented, the following applies:''

The API needs to verify read/write access to all endpoints according to the provided auth token (RBAC). The ACL should be cached with the token.

Endpoints
Clients may choose to interact with any of the service endpoints retrieved from Keystone. Agents use the given endpoint as a home document href from which to discover links to all other API requests. Endpoints may be organized by region

An example endpoint:

https://marconi.example.com/v1

Errors
If any request results in an error, the server will return an appropriate 4xx or 5xx HTTP status code, and optionally the following information in the body:


 * Title
 * Description
 * Internal code
 * Link to more information

Example:

HTTP/1.1 400 Bad Request Content-Type: application/json

{ "title": "Unsupported limit", "description": "The given limit cannot be negative, and cannot be greater than 50.", "code": 1092, "link": { "rel": "help", "href": "http://docs.example.com/messages#limit", "text": "API documentation for the limit parameter" } }

Common Headers
Each request to the API must include certain standard and extended HTTP headers. These headers provide host, agent, authentication and other pertinent information to the server.

HTTP Response Codes
See Response Codes

Sample API Request
GET /v1/queues/fizbat/messages?marker=1355-237242-783&limit=10 HTTP/1.1 Host: marconi.example.com User-Agent: python/2.7 killer-rabbit/1.2 Date: Wed, 28 Nov 2012 21:14:19 GMT Accept: application/json Accept-Encoding: gzip X-Auth-Token: 7d2f63fd-4dcc-4752-8e9b-1d08f989cc00 X-Project-Id: 518b51ea133c4facadae42c328d6b77b Client-ID: 30387f00-39a0-11e2-be4d-a8d15f34bae2

Endpoints Synopsis
GET /v1 GET /v1/health
 * 1) Base endpoints

GET /v1/queues{?marker,limit,detailed} GET /v1/queues/{queue_name} # existence check HEAD /v1/queues/{queue_name} # existence check PUT /v1/queues/{name} DELETE /v1/queues/{name}
 * 1) Queues

PUT /v1/queues/{queue_name}/metadata GET /v1/queues/{queue_name}/metadata GET /v1/queues/{queue_name}/stats
 * 1) Queue metadata

GET /v1/queues/{queue_name}/messages{?marker,limit,echo,include_claimed} GET /v1/queues/{queue_name}/messages/{message_id} GET /v1/queues/{queue_name}/messages{?ids} POST /v1/queues/{queue_name}/messages DELETE /v1/queues/{queue_name}/messages/{message_id}{?claim_id} DELETE /v1/queues/{queue_name}/messages{?ids}
 * 1) Messages

POST /v1/queues/{queue_name}/claims{?limit} GET /v1/queues/{queue_name}/claims/{claim_id} PATCH /v1/queues/{queue_name}/claims/{claim_id} DELETE /v1/queues/{queue_name}/claims/{claim_id}
 * 1) Claims

Get Home Document
Template

GET /v1

Request

GET /v1 HTTP/1.1 Host: marconi.example.com

...

Response

HTTP/1.0 200 OK

...

{   "resources": { "rel/queue-stats": { "href-template": "/v1/queues/{queue_name}/stats", "href-vars": { "queue_name": "param/queue_name" },            "hints": { "allow": [ "GET" ],                "formats": { "application/json": {} }           }        },         "rel/post-messages": { "href-template": "/v1/queues/{queue_name}/messages", "href-vars": { "queue_name": "param/queue_name" },            "hints": { "accept-post": [ "application/json" ],                "allow": [ "POST" ],                "formats": { "application/json": {} }           }        },         "rel/queue": { "href-template": "/v1/queues/{queue_name}", "href-vars": { "queue_name": "param/queue_name" },            "hints": { "allow": [ "GET", "HEAD", "PUT", "DELETE" ],                "formats": { "application/json": {} }           }        },         "rel/queue-metadata": { "href-template": "/v1/queues/{queue_name}/metadata", "href-vars": { "queue_name": "param/queue_name" },            "hints": { "allow": [ "GET", "PUT" ],                "formats": { "application/json": {} }           }        },         "rel/queues": { "href-template": "/v1/queues{?marker,limit,detailed}", "href-vars": { "marker": "param/marker", "detailed": "param/detailed", "limit": "param/queue_limit" },            "hints": { "allow": [ "GET" ],                "formats": { "application/json": {} }           }        },         "rel/messages": { "href-template": "/v1/queues/{queue_name}/messages{?marker,limit,echo,include_claimed}", "href-vars": { "marker": "param/marker", "include_claimed": "param/include_claimed", "queue_name": "param/queue_name", "limit": "param/messages_limit", "echo": "param/echo" },            "hints": { "allow": [ "GET" ],                "formats": { "application/json": {} }           }        },         "rel/claim": { "href-template": "/v1/queues/{queue_name}/claims{?limit}", "href-vars": { "queue_name": "param/queue_name", "limit": "param/claim_limit" },            "hints": { "accept-post": [ "application/json" ],                "allow": [ "POST" ],                "formats": { "application/json": {} }           }        }    } }

Discussion

The entire Marconi API is discoverable from a single starting point; agents do not need to know any more than this one URI in order to explore the entire API.

This document is cache-able (and should be cached by proxy and client).

See also: http://tools.ietf.org/html/draft-nottingham-json-home-02

Check Node Health
Template

GET /v1/health

or

HEAD /v1/health

Request

GET /v1/health HTTP/1.1 Host: example.marconi.com

...

Response

HTTP/1.1 204 No Content

or

HTTP/1.1 503 Service Unavailable

Discussion

Use this request to check on the Marconi node status. If the node is down, this endpoint will not be reachable and will act as a signal to the load balancer to take the node out of rotation. Alternatively, the endpoint may be reachable but the service is temporarily unavailable due to storage driver failure or some other error.

Create Queue
Template

PUT /v1/queues/{queue_name}

Request

PUT /v1/queues/fizbat HTTP/1.1 Host: marconi.example.com

Response

HTTP/1.1 201 Created Location: /v1/queues/fizbit

Discussion

Creates a queue.

The body of the PUT is empty.

is the name to give the queue. The name MUST NOT exceed 64 bytes in length, and is limited to US-ASCII letters, digits, underscores and hyphens.

List Queues
Template

GET /v1/queues{?marker,limit,detailed}

Request

GET /v1/queues?marker=baz&detailed=true HTTP/1.1 Host: marconi.example.com

...

Response

HTTP/1.1 200 OK

{ "links": [ {     "rel": "next", "href": "/v1/queues?marker=kooleo&limit=10&detailed=true" } ],  "queues": [ { "name": "boomerang", "href": "/v1/queues/boomerang", "metadata": {} }, { "name": "fizbit", "href": "/v1/queues/fizbit", "metadata": { "handle": "@kgriffs" } },

...

{ "name": "kooleo", "href": "/v1/queues/kooleo", "metadata": { "something": "something_else" } } ] }

Checking Queue Existence
Template

GET /v1/queues/{queue_name}

Request

GET /v1/queues/{queue_name} Host: marconi.example.com

Response

HTTP/1.1 204 OK

Discussion

Returns 204 if the queue exists, else 404.

also works.

Delete Queue
Template

DELETE /v1/queues/{queue_name}

Request

DELETE /v1/queues/fizbat HTTP/1.1 Host: marconi.example.com

Response

HTTP/1.1 204 No Content

Discussion

Use this operation to immediately delete a queue along with all its messages (if any).

Set Queue Metadata
Template

PUT /v1/queues/{queue_name}/metadata

{  ... }

Request

PUT /v1/queues/fizbat/metadata HTTP/1.1 Host: marconi.example.com

...

{  "key": { "key2": "value", "key3": [1, 2, 3, 4, 5] } }

Response

HTTP/1.1 204 No Content

Discussion

This operation allows for setting structured metadata. If PUT is used, this document will replace the existing metadata document in it's entirety, so clients need to be careful they do not accidentally delete fields.

The body of the request is an arbitrary document which allows storing contextual information about the way a particular application should interact with the queue. The size of this body, in characters and including whitespace must be <= 64 KiB (configurable). The document MUST be valid JSON (Marconi will validate).

Get Queue Metadata
Template

GET /v1/queues/{queue_name}/metadata

Request

GET /v1/queues/fizbit/metadata HTTP/1.1 Host: marconi.example.com

...

Response HTTP/1.1 200 OK

{  "key": { "key2": "value", "key3": [1, 2, 3, 4, 5] } }

Discussion

Returns queue metadata.

Get Queue Stats
Template

GET /v1/queues/{queue_name}/stats

Request

GET /v1/queues/fizbit/stats HTTP/1.1 Host: marconi.example.com

...

Response HTTP/1.1 200 OK

{ "messages": { "free": 146929, "claimed": 2409, "total": 149338, "oldest": { "href": "/v1/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b01", "age": 63, "created": "2013-08-12T20:44:55Z" },   "newest": { "href": "/v1/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b01", "age": 12, "created": "2013-08-12T20:45:46Z" } }

Discussion

Returns queue statistics, including how many messages are in the queue, broken out by status.

NOTE: If total is 0, then "oldest" and "newest" message stats are not included.

Messages
All message-related operations require  to be included in the headers. This is to ensure that messages are not echoed back to the client that posted them unless the client explicitly requests this.

List Messages
Template

GET /v1/queues/{queue_name}/messages{?marker,limit,echo,include_claimed}

Request

GET /v1/queues/fizbit/messages?marker=1355-237242-783&limit=10 HTTP/1.1 Host: marconi.example.com

...

Response

HTTP 200 if the query matched 1 or more messages, HTTP 204 otherwise (with no body).

HTTP/1.1 200 OK Content-Location: /v1/queues/fizbit/messages?marker=1355-237242-783&limit=10

...

{ "links": [ {     "rel": "next", "href": "/v1/queues/fizbit/messages?marker=6244-244224-783&limit=10" } ],  "messages": [ {     "href": "/v1/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b01", "ttl": 800, "age": 790, "body": { "event": "ActivateAccount", "mode": "active" }   }  ] }

Discussion

Message IDs and markers are opaque strings; clients should make no assumptions about their format or length. Furthermore, clients should assume there is no relationship between markers and message IDs (i.e., one cannot be derived from the other). This allows for a wide variety of storage driver implementations.

Results are ordered by age, oldest message first.

specifies up to 20 messages (configurable) to return. If not specified, limit defaults to 10. When more messages are available than can be returned in a single request, the client can pick up the next batch by simply using the URI template parameters returned from the previous call in the "next" field (TBD).

is an opaque string that the client can use to request the next batch of messages. It communicates to the server which messages the client has already received.

Note: If  is not specified, the API will return all messages at the head of the queue (up to limit).

is a boolean (i.e., "true" or "false") that determines whether or not the API will return a client's own messages, as determined by the  portion of the Client-ID header. If not specified, echo defaults to "false".

is a boolean (i.e., "true" or "false") that determines whether or not the API will return claimed messages as well as unclaimed ones. If not specified, defaults to "false" (only unclaimed messages are returned).

Get a Specific Message
Template

GET /v1/queues/{queue_name}/messages/{message_id}

Request

GET /v1/queues/fizbat/messages/50b68a50d6f5b8c8a7c62b01 HTTP/1.1 Host: marconi.example.com

...

Response

HTTP/1.1 200 OK Content-Location: /v1/queues/fizbat/messages/50b68a50d6f5b8c8a7c62b01

...

{ "href": "/v1/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b01", "ttl": 800, "age": 790, "body": { "event": "ActivateAccount", "mode": "active" } }

Discussion

Message fields are defined as follows:

is an opaque relative URI that the client can use to uniquely identify a message resource, and interact with it.

is the ttl set on the message when it was posted. The message will expire after (ttl - age) seconds.

number of seconds since ts, relative to the server's clock.

Arbitrary document submitted along with the original request to post the message.

If either the message ID or the claim ID is malformed or nonexistent, no message is returned.

Get a Set of Messages by ID
Template

GET /v1/queues/{queue_name}/messages{?ids}

Request

GET /v1/queues/fizbat/messages?ids=50b68a50d6f5b8c8a7c62b01,f5b8c8a7c62b0150b68a50d6 HTTP/1.1 Host: marconi.example.com

...

Response

HTTP/1.1 200 OK Content-Location: /v1/queues/fizbat/messages?ids=50b68a50d6f5b8c8a7c62b01,f5b8c8a7c62b0150b68a50d6

...

[   {      "href": "/v1/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b01", "ttl": 800, "age": 32, "body": { "cmd": "EncodeVideo", "jobid": 58229 }   },    {      "href": "/v1/queues/fizbit/messages/f5b8c8a7c62b0150b68a50d6", "ttl": 800, "age": 790, "body": { "cmd": "EncodeAudio", "jobid": 58201 }   } ]

Discussion

Unlike message listing, a client's own messages are always returned in this approach. Note that the list of ids may not exceed 20 (configurable, shared setting with number of messages that can be posted at once).

This approach returns all the messages with matching IDs. If a malformed ID or a nonexistent message ID is provided, it is ignored and the remaining messages are returned.

Post Message(s)
Template

POST /v1/queues/{queue_name}/messages

Request

POST /v1/queues/fizbit/messages HTTP/1.1 Host: marconi.example.com

...

[ {    "ttl": 300, "body": { "event": "BackupStarted", "backup_id": "c378813c-3f0b-11e2-ad92-7823d2b0f3ce" } },  {    "ttl": 60, "body": { "event": "BackupProgress", "current_bytes": "0", "total_bytes": "99614720" } } ]

Responses

When a single message is submitted:

HTTP/1.1 201 Created Location: /v1/queues/fizbit/messages?ids=50b68a50d6f5b8c8a7c62b01 Content-Type: application/json

{   "resources": [ "/v1/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b01" ],   "partial": false }

When multiple messages are submitted:

HTTP/1.1 201 Created Location: /v1/queues/fizbit/messages?ids=50b68a50d6f5b8c8a7c62b01,50b68a50d6f5b8c8a7c62b05 Content-Type: application/json

{   "resources": [ "/v1/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b01", "/v1/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b05" ],   "partial": false }

Discussion

Up to 20 messages (default, but configurable) may be submitted in a single request, but must always be encapsulated in a collection container (an array in JSON). The resulting value of the Location header or response body may be used to retrieve the created messages for further processing if needed.

The client only specifies the body and ttl for the message; the server will insert metadata such as id and age.

The response body contains a list of resource paths corresponding to each message submitted in the request, in the same order. In the case of a server-side error part-way through the processing of the submitted messages, a partial list will be returned,  will be true, and the client will need to retry posting the remaining messages. If no messages could be enqueued, the server will return 503 Service Unavailable.

NOTE: The partial field has been deprecated since the icehouse release, and will always be "false"; drivers are not longer allowed to return a "partial" success and so must either succeed or fail the entire batch of messages.

The size of the request document, in characters including whitespace, is 256 KiB by default (configurable). The document MUST be valid JSON (Marconi will validate).

specifies an arbitrary document which constitutes the body of the message being sent.

is how long the server should wait before expiring and removing the message from the queue. Value MUST be between 60 and 1209600 seconds (14 days, configurable), inclusive. Note that the server may not actually delete the message until it's age has reached up to (ttl + 60) seconds, to allow for flexibility in storage implementations.

Delete Message
Template

DELETE /v1/queues/{queue_name}/messages/{message_id}{?claim_id}

Request

DELETE /v1/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b01 HTTP/1.1 Host: marconi.example.com

...

Response

HTTP/1.1 204 No Content

Discussion

specifies the message to delete.

specifies that the message should only be deleted if it has the specified claim ID, and that claim has not expired. This is useful for ensuring only one agent processes any given message; whenever a worker client's claim expires before it has a chance to delete a message it has processed, the worker must roll back any actions it took based on that message, since another worker will now be able to claim and process the same message.

Note that if claim_id is not specified, but the message is claimed, the operation will fail. In other words, claimed messages can only ever be deleted by providing an appropriate claim_id.

Delete a Set of Messages by ID
Template

DELETE /v1/queues/{queue_name}/messages{?ids}

Request

DELETE /v1/queues/fizbit/messages?ids=50b68a50d6f5b8c8a7c62b01,50b68a50d6f5b8c8a7c62b02 HTTP/1.1 Host: marconi.example.com

...

Response

HTTP/1.1 204 No Content

Discussion

Bulk delete for messages.

specifies the messages to delete, up to a maximum of 20 (configurable, shares the same setting as posting and listing messages)

If any of the message IDs are malformed or non-existent, they are ignored. The remaining valid message IDs are deleted.

Claim Messages
Template

POST /v1/queues/{queue_name}/claims{?limit} Content-Type: application/json

...

{    "ttl": {claim_ttl}, "grace": {message_grace} }

Request

POST /v1/queues/fizbit/claims?limit=5 HTTP/1.1 Content-Type: application/json Accept: application/json

...

{    "ttl": 300, "grace": 300 }

Response

The client receives a claim URI and a list of claimed messages, if any:

HTTP/1.1 201 Created Content-Type: application/json Location: /v1/queues/foo-bar/claims/a28ee94e-6cb4-11e2-b4d5-7703267a7926

...

[ {    "href": "/v1/queues/foo-bar/messages/50b68a50d6f5b8c8a7c62b01?claim_id=a28ee94e-6cb4-11e2-b4d5-7703267a7926", "ttl": 800, "age": 100, "body": { "object_id": "8a50d6", "target": "h.264" } },  {    "href": "/v1/queues/foo-bar/messages/50b68a50d6f5b8c8a7c62b02?claim_id=a28ee94e-6cb4-11e2-b4d5-7703267a7926", "ttl": 800, "age": 790, "body": { "object_id": "fb8c8a", "target": "h.264" } } ]

Discussion

Claims a set of messages, up to limit, from oldest to newest, skipping any that are already claimed. If no unclaimed messages are available, returns 204 No Content.

The client should delete the message when it has finished processing it, before the claim expires, to ensure the message is processed only once. As part of the delete operation, all worker clients should specify the claim ID (this is best done by simply using the provided href). That way, the server can return an error if the claim just expired (notifying the client of the race condition), giving the worker a chance to roll back its own processing of the given message, since another worker will likely claim the message and process it.

Just as with a message's age, the age given for the claim is relative to the server's clock, and is useful for determining how quickly messages are getting processed, and whether a given message's claim is about to expire.

When a claim expires, it is removed, allowing another client worker to claim the message in the case that the original worker fails to process it.

specifies up to 20 messages (configurable) to claim. If not specified, limit defaults to 10. Note that claim creation is best-effort, meaning the server may claim and return less than the requested number of messages.

is how long the server should wait before releasing the claim. Value MUST be between 60 and 43200 seconds (12 hours, configurable).

is the message grace period in seconds. Value MUST be between 60 and 43200 seconds (12 hours, configurable). The server will extend the lifetime of claimed messages to be at least as long as the lifetime of the claim itself, plus a specified grace period to deal with crashed workers (up to 1209600 or 14 days including claim lifetime). If a claimed message would normally live longer than the grace period, it's expiration will not be adjusted.

Query Claim
Template

GET /v1/queues/{queue_name}/claims/{claim_id}

Request

GET /v1/queues/foo-bar/claims/a28ee94e-6cb4-11e2-b4d5-7703267a7926 Host: marconi.example.com

...

Response

HTTP/1.1 200 OK Content-Location: /v1/queues/foo-bar/claims/a28ee94e-6cb4-11e2-b4d5-7703267a7926

...

{ "age": 19, "ttl": 30, "messages": [ ... ] }

Discussion

Update Claim
Template

PATCH /v1/queues/{queue_name}/claims/{claim_id} Content-Type: application/json

Request

PATCH /v1/queues/fizbit/claims/a28ee94e-6cb4-11e2-b4d5-7703267a7926 HTTP/1.1 Content-Type: application/json

...

{ "ttl": 300 }

Response

HTTP/1.1 204 No Content

Discussion

Clients should periodically renew claims during long-running batches of work to avoid loosing a claim in the middle of processing a message. This is done by issuing a PATCH to a specific claim resource and including a new TTL for the claim (which may be different from the original TTL). The server will reset the age of the claim and apply the new TTL.

Release Claim
Template

DELETE /v1/queues/{queue_name}/claims/{claim_id}

Request

DELETE /v1/queues/foo-bar/claims/a28ee94e-6cb4-11e2-b4d5-7703267a7926 HTTP/1.1 Host: marconi.example.com

Response

HTTP/1.1 204 No Content

Discussion

Use this operation to immediately release a claim, making any (remaining, non-deleted) messages associated with the claim available to other workers.

This operation is useful when a worker is performing a graceful shutdown, fails to process one or more messages, or is taking longer than expected to process messages, and wishes to make the remainder available to other workers.