Difference between revisions of "Zaqar/specs/api/v1"
(→Post Message(s)) |
(→Post Message(s): Note about "partial" deprecation) |
||
(75 intermediate revisions by 7 users not shown) | |||
Line 1: | Line 1: | ||
= Marconi API: v1 Blueprint = | = Marconi API: v1 Blueprint = | ||
− | |||
− | ''' | + | <big>'''The Marconi v1 API is FROZEN'''</big> |
− | + | We have started collecting feedback with an eye toward extensions and v2 of the API here: [[Marconi/specs/api/next]] | |
− | |||
− | |||
− | |||
Line 18: | Line 14: | ||
Note: Error responses are enumerated on a separate page: [[Marconi/specs/api/v1/errors]] | Note: Error responses are enumerated on a separate page: [[Marconi/specs/api/v1/errors]] | ||
+ | |||
== Common API Elements == | == Common API Elements == | ||
− | |||
− | |||
− | |||
− | |||
− | |||
Line 45: | Line 37: | ||
The URI version will only be incremented to accommodate major new features or API redesigns that can not be made backwards-compatible. When new API versions are released, older versions are deprecated. Marconi maintainers will work with developers and partners to ensure there is adequate time to migrate to new versions before deprecated ones are discontinued. | The URI version will only be incremented to accommodate major new features or API redesigns that can not be made backwards-compatible. When new API versions are released, older versions are deprecated. Marconi maintainers will work with developers and partners to ensure there is adequate time to migrate to new versions before deprecated ones are discontinued. | ||
− | Since the | + | Since the version is only incremented to accommodate major API revisions, sub-versioning of the API will be rare, but may be used occasionally to release a "polished" major version. |
− | |||
=== Resource Media Types === | === Resource Media Types === | ||
Line 65: | Line 56: | ||
In response to valid credentials, Keystone responds with an auth token and a service catalog that contains a list of all services and endpoints available for the given token. Multiple endpoints may be returned for Marconi according to physical locations and performance/availability characteristics of different deployments. | In response to valid credentials, Keystone responds with an auth token and a service catalog that contains a list of all services and endpoints available for the given token. Multiple endpoints may be returned for Marconi according to physical locations and performance/availability characteristics of different deployments. | ||
− | Marconi clients | + | Normally, Keystone middleware provides the X-Project-Id header based on the auth token submitted by the Marconi client. For this to work, clients MUST specify a valid auth token in the `X-Auth-Token` header for each request to the Marconi API. The API validates auth tokens against Keystone before servicing each request |
+ | If auth is not enabled, clients must provide the X-Project-Id header themselves. | ||
=== Authorization === | === Authorization === | ||
Line 75: | Line 67: | ||
− | === | + | === Endpoints === |
− | + | Clients may choose to interact with any of the service endpoints retrieved from Keystone. Agents use the given endpoint as a home document href from which to discover links to all other API requests. Endpoints may be organized by region | |
An example endpoint: | An example endpoint: | ||
<pre><nowiki> | <pre><nowiki> | ||
− | https://marconi.example.com/v1 | + | https://marconi.example.com/v1 |
</nowiki></pre> | </nowiki></pre> | ||
− | |||
− | |||
=== Errors === | === Errors === | ||
− | If any request results in an error, the server will return an appropriate 4xx or 5xx HTTP status code, | + | If any request results in an error, the server will return an appropriate 4xx or 5xx HTTP status code, and optionally the following information in the body: |
* Title | * Title | ||
Line 114: | Line 104: | ||
} | } | ||
</nowiki></pre> | </nowiki></pre> | ||
− | |||
=== Common Headers === | === Common Headers === | ||
Line 131: | Line 120: | ||
|- | |- | ||
| Accept | | Accept | ||
− | | Media type desired; | + | | Media type desired; only <code><nowiki>application/json</nowiki></code> is supported at this time. |
|- | |- | ||
| Accept-Encoding | | Accept-Encoding | ||
Line 141: | Line 130: | ||
| X-Auth-Token | | X-Auth-Token | ||
| Keystone auth token | | Keystone auth token | ||
+ | |- | ||
+ | | X-Project-Id | ||
+ | | An ID for a project to which the value of X-Auth-Token grants access. Queues will be created under this project. | ||
|- | |- | ||
| Client-ID | | Client-ID | ||
− | | A | + | | A UUID, used to distinguish each client using the service. The UUID must be submitted in its canonical form (e.g., <code><nowiki>3381af92-2b9e-11e3-b191-71861300734c</nowiki></code>). In Marconi, this is used to avoid echoing a sender's messages back to the same instance, and may be logged by the server for future use. Should be generated once and persisted between restarts of the client. |
|} | |} | ||
+ | |||
+ | === HTTP Response Codes === | ||
+ | |||
+ | See [[Marconi/specs/api/v1/responsecodes| Response Codes]] | ||
== Sample API Request == | == Sample API Request == | ||
<pre><nowiki> | <pre><nowiki> | ||
− | GET /v1 | + | GET /v1/queues/fizbat/messages?marker=1355-237242-783&limit=10 HTTP/1.1 |
Host: marconi.example.com | Host: marconi.example.com | ||
User-Agent: python/2.7 killer-rabbit/1.2 | User-Agent: python/2.7 killer-rabbit/1.2 | ||
Line 156: | Line 152: | ||
Accept-Encoding: gzip | Accept-Encoding: gzip | ||
X-Auth-Token: 7d2f63fd-4dcc-4752-8e9b-1d08f989cc00 | X-Auth-Token: 7d2f63fd-4dcc-4752-8e9b-1d08f989cc00 | ||
+ | X-Project-Id: 518b51ea133c4facadae42c328d6b77b | ||
Client-ID: 30387f00-39a0-11e2-be4d-a8d15f34bae2 | Client-ID: 30387f00-39a0-11e2-be4d-a8d15f34bae2 | ||
</nowiki></pre> | </nowiki></pre> | ||
− | == | + | == Endpoints Synopsis == |
+ | <pre><nowiki> | ||
+ | # Base endpoints | ||
+ | GET /v1 | ||
+ | GET /v1/health | ||
+ | |||
+ | # Queues | ||
+ | GET /v1/queues{?marker,limit,detailed} | ||
+ | GET /v1/queues/{queue_name} # existence check | ||
+ | HEAD /v1/queues/{queue_name} # existence check | ||
+ | PUT /v1/queues/{name} | ||
+ | DELETE /v1/queues/{name} | ||
+ | |||
+ | # Queue metadata | ||
+ | PUT /v1/queues/{queue_name}/metadata | ||
+ | GET /v1/queues/{queue_name}/metadata | ||
+ | GET /v1/queues/{queue_name}/stats | ||
+ | |||
+ | # Messages | ||
+ | GET /v1/queues/{queue_name}/messages{?marker,limit,echo,include_claimed} | ||
+ | GET /v1/queues/{queue_name}/messages/{message_id} | ||
+ | GET /v1/queues/{queue_name}/messages{?ids} | ||
+ | POST /v1/queues/{queue_name}/messages | ||
+ | DELETE /v1/queues/{queue_name}/messages/{message_id}{?claim_id} | ||
+ | DELETE /v1/queues/{queue_name}/messages{?ids} | ||
+ | |||
+ | # Claims | ||
+ | POST /v1/queues/{queue_name}/claims{?limit} | ||
+ | GET /v1/queues/{queue_name}/claims/{claim_id} | ||
+ | PATCH /v1/queues/{queue_name}/claims/{claim_id} | ||
+ | DELETE /v1/queues/{queue_name}/claims/{claim_id} | ||
+ | </nowiki></pre> | ||
+ | |||
+ | == Base Endpoints == | ||
+ | |||
+ | === Get Home Document === | ||
'''Template''' | '''Template''' | ||
Line 183: | Line 215: | ||
<pre><nowiki> | <pre><nowiki> | ||
− | + | HTTP/1.0 200 OK | |
+ | |||
+ | ... | ||
+ | |||
+ | { | ||
+ | "resources": { | ||
+ | "rel/queue-stats": { | ||
+ | "href-template": "/v1/queues/{queue_name}/stats", | ||
+ | "href-vars": { | ||
+ | "queue_name": "param/queue_name" | ||
+ | }, | ||
+ | "hints": { | ||
+ | "allow": [ | ||
+ | "GET" | ||
+ | ], | ||
+ | "formats": { | ||
+ | "application/json": {} | ||
+ | } | ||
+ | } | ||
+ | }, | ||
+ | "rel/post-messages": { | ||
+ | "href-template": "/v1/queues/{queue_name}/messages", | ||
+ | "href-vars": { | ||
+ | "queue_name": "param/queue_name" | ||
+ | }, | ||
+ | "hints": { | ||
+ | "accept-post": [ | ||
+ | "application/json" | ||
+ | ], | ||
+ | "allow": [ | ||
+ | "POST" | ||
+ | ], | ||
+ | "formats": { | ||
+ | "application/json": {} | ||
+ | } | ||
+ | } | ||
+ | }, | ||
+ | "rel/queue": { | ||
+ | "href-template": "/v1/queues/{queue_name}", | ||
+ | "href-vars": { | ||
+ | "queue_name": "param/queue_name" | ||
+ | }, | ||
+ | "hints": { | ||
+ | "allow": [ | ||
+ | "GET", | ||
+ | "HEAD", | ||
+ | "PUT", | ||
+ | "DELETE" | ||
+ | ], | ||
+ | "formats": { | ||
+ | "application/json": {} | ||
+ | } | ||
+ | } | ||
+ | }, | ||
+ | "rel/queue-metadata": { | ||
+ | "href-template": "/v1/queues/{queue_name}/metadata", | ||
+ | "href-vars": { | ||
+ | "queue_name": "param/queue_name" | ||
+ | }, | ||
+ | "hints": { | ||
+ | "allow": [ | ||
+ | "GET", | ||
+ | "PUT" | ||
+ | ], | ||
+ | "formats": { | ||
+ | "application/json": {} | ||
+ | } | ||
+ | } | ||
+ | }, | ||
+ | "rel/queues": { | ||
+ | "href-template": "/v1/queues{?marker,limit,detailed}", | ||
+ | "href-vars": { | ||
+ | "marker": "param/marker", | ||
+ | "detailed": "param/detailed", | ||
+ | "limit": "param/queue_limit" | ||
+ | }, | ||
+ | "hints": { | ||
+ | "allow": [ | ||
+ | "GET" | ||
+ | ], | ||
+ | "formats": { | ||
+ | "application/json": {} | ||
+ | } | ||
+ | } | ||
+ | }, | ||
+ | "rel/messages": { | ||
+ | "href-template": "/v1/queues/{queue_name}/messages{?marker,limit,echo,include_claimed}", | ||
+ | "href-vars": { | ||
+ | "marker": "param/marker", | ||
+ | "include_claimed": "param/include_claimed", | ||
+ | "queue_name": "param/queue_name", | ||
+ | "limit": "param/messages_limit", | ||
+ | "echo": "param/echo" | ||
+ | }, | ||
+ | "hints": { | ||
+ | "allow": [ | ||
+ | "GET" | ||
+ | ], | ||
+ | "formats": { | ||
+ | "application/json": {} | ||
+ | } | ||
+ | } | ||
+ | }, | ||
+ | "rel/claim": { | ||
+ | "href-template": "/v1/queues/{queue_name}/claims{?limit}", | ||
+ | "href-vars": { | ||
+ | "queue_name": "param/queue_name", | ||
+ | "limit": "param/claim_limit" | ||
+ | }, | ||
+ | "hints": { | ||
+ | "accept-post": [ | ||
+ | "application/json" | ||
+ | ], | ||
+ | "allow": [ | ||
+ | "POST" | ||
+ | ], | ||
+ | "formats": { | ||
+ | "application/json": {} | ||
+ | } | ||
+ | } | ||
+ | } | ||
+ | } | ||
+ | } | ||
</nowiki></pre> | </nowiki></pre> | ||
Line 195: | Line 349: | ||
See also: http://tools.ietf.org/html/draft-nottingham-json-home-02 | See also: http://tools.ietf.org/html/draft-nottingham-json-home-02 | ||
− | + | === Check Node Health === | |
− | == | ||
Line 202: | Line 355: | ||
<pre><nowiki> | <pre><nowiki> | ||
− | + | GET /v1/health | |
+ | </nowiki></pre> | ||
− | + | or | |
− | + | ||
− | + | <pre><nowiki> | |
+ | HEAD /v1/health | ||
</nowiki></pre> | </nowiki></pre> | ||
+ | |||
'''Request''' | '''Request''' | ||
<pre><nowiki> | <pre><nowiki> | ||
− | + | GET /v1/health HTTP/1.1 | |
− | Host: marconi | + | Host: example.marconi.com |
... | ... | ||
− | { | + | </nowiki></pre> |
− | + | ||
− | + | ||
+ | '''Response''' | ||
+ | |||
+ | <pre><nowiki> | ||
+ | HTTP/1.1 204 No Content | ||
+ | </nowiki></pre> | ||
+ | |||
+ | or | ||
+ | |||
+ | <pre><nowiki> | ||
+ | HTTP/1.1 503 Service Unavailable | ||
+ | </nowiki></pre> | ||
+ | |||
+ | |||
+ | |||
+ | '''Discussion''' | ||
+ | |||
+ | Use this request to check on the Marconi node status. If the node is down, this endpoint will not be reachable and will act as a signal to the load balancer to take the node out of rotation. Alternatively, the endpoint may be reachable but the service is temporarily unavailable due to storage driver failure or some other error. | ||
+ | |||
+ | == Queues == | ||
+ | |||
+ | === Create Queue === | ||
+ | |||
+ | |||
+ | '''Template''' | ||
+ | |||
+ | <pre><nowiki> | ||
+ | PUT /v1/queues/{queue_name} | ||
+ | </nowiki></pre> | ||
+ | |||
+ | '''Request''' | ||
+ | <pre><nowiki> | ||
+ | PUT /v1/queues/fizbat HTTP/1.1 | ||
+ | Host: marconi.example.com | ||
</nowiki></pre> | </nowiki></pre> | ||
Line 228: | Line 417: | ||
<pre><nowiki> | <pre><nowiki> | ||
HTTP/1.1 201 Created | HTTP/1.1 201 Created | ||
− | Location: /v1 | + | Location: /v1/queues/fizbit |
</nowiki></pre> | </nowiki></pre> | ||
Line 234: | Line 423: | ||
'''Discussion''' | '''Discussion''' | ||
− | Creates a queue | + | Creates a queue. |
− | The body of the PUT is | + | The body of the PUT is empty. |
− | <code><nowiki>queue_name</nowiki></code> is the name to give the queue. The name MUST NOT exceed 64 bytes in length, and is limited to US-ASCII letters, digits, underscores and hyphens | + | <code><nowiki>queue_name</nowiki></code> is the name to give the queue. The name MUST NOT exceed 64 bytes in length, and is limited to US-ASCII letters, digits, underscores and hyphens. |
− | == List Queues == | + | === List Queues === |
Line 246: | Line 435: | ||
<pre><nowiki> | <pre><nowiki> | ||
− | GET | + | GET /v1/queues{?marker,limit,detailed} |
</nowiki></pre> | </nowiki></pre> | ||
Line 254: | Line 443: | ||
<pre><nowiki> | <pre><nowiki> | ||
− | GET | + | GET /v1/queues?marker=baz&detailed=true HTTP/1.1 |
Host: marconi.example.com | Host: marconi.example.com | ||
Line 270: | Line 459: | ||
{ | { | ||
"rel": "next", | "rel": "next", | ||
− | "href": "/v1 | + | "href": "/v1/queues?marker=kooleo&limit=10&detailed=true" |
} | } | ||
], | ], | ||
"queues": [ | "queues": [ | ||
− | { "name": "boomerang", "href": "/v1 | + | { "name": "boomerang", "href": "/v1/queues/boomerang", "metadata": {} }, |
− | { "name": "fizbit", "href": "/v1 | + | { "name": "fizbit", "href": "/v1/queues/fizbit", "metadata": { "handle": "@kgriffs" } }, |
... | ... | ||
− | { "name": "kooleo", "href": "/v1 | + | { "name": "kooleo", "href": "/v1/queues/kooleo", "metadata": { "something": "something_else" } } |
] | ] | ||
} | } | ||
+ | </nowiki></pre> | ||
+ | |||
+ | === Checking Queue Existence === | ||
+ | |||
+ | '''Template''' | ||
+ | |||
+ | <pre><nowiki> | ||
+ | GET /v1/queues/{queue_name} | ||
+ | </nowiki></pre> | ||
+ | |||
+ | |||
+ | '''Request''' | ||
+ | |||
+ | <pre><nowiki> | ||
+ | GET /v1/queues/{queue_name} | ||
+ | Host: marconi.example.com | ||
</nowiki></pre> | </nowiki></pre> | ||
− | ''' | + | '''Response''' |
− | + | <pre><nowiki> | |
+ | HTTP/1.1 204 OK | ||
+ | </nowiki></pre> | ||
− | |||
− | + | '''Discussion''' | |
− | + | Returns 204 if the queue exists, else 404. | |
− | + | <code><nowiki>HEAD</nowiki></code> also works. | |
+ | === Delete Queue === | ||
'''Template''' | '''Template''' | ||
<pre><nowiki> | <pre><nowiki> | ||
− | + | DELETE /v1/queues/{queue_name} | |
− | |||
− | |||
− | |||
− | } | ||
</nowiki></pre> | </nowiki></pre> | ||
Line 313: | Line 516: | ||
<pre><nowiki> | <pre><nowiki> | ||
− | + | DELETE /v1/queues/fizbat HTTP/1.1 | |
Host: marconi.example.com | Host: marconi.example.com | ||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
</nowiki></pre> | </nowiki></pre> | ||
Line 336: | Line 530: | ||
'''Discussion''' | '''Discussion''' | ||
− | + | Use this operation to immediately delete a queue along with all its messages (if any). | |
− | + | == Queue Metadata == | |
− | + | === Set Queue Metadata === | |
− | == | ||
Line 347: | Line 540: | ||
<pre><nowiki> | <pre><nowiki> | ||
− | + | PUT /v1/queues/{queue_name}/metadata | |
+ | |||
+ | { | ||
+ | ... | ||
+ | } | ||
</nowiki></pre> | </nowiki></pre> | ||
Line 354: | Line 551: | ||
<pre><nowiki> | <pre><nowiki> | ||
− | + | PUT /v1/queues/fizbat/metadata HTTP/1.1 | |
Host: marconi.example.com | Host: marconi.example.com | ||
... | ... | ||
+ | { | ||
+ | "key": { | ||
+ | "key2": "value", | ||
+ | "key3": [1, 2, 3, 4, 5] | ||
+ | } | ||
+ | } | ||
</nowiki></pre> | </nowiki></pre> | ||
'''Response''' | '''Response''' | ||
− | |||
<pre><nowiki> | <pre><nowiki> | ||
− | + | HTTP/1.1 204 No Content | |
− | |||
− | |||
</nowiki></pre> | </nowiki></pre> | ||
Line 374: | Line 574: | ||
'''Discussion''' | '''Discussion''' | ||
− | + | This operation allows for setting structured metadata. If PUT is used, this document will replace the existing metadata document in it's entirety, so clients need to be careful they do not accidentally delete fields. | |
+ | The body of the request is an arbitrary document which allows storing contextual information about the way a particular application should interact with the queue. The size of this body, in characters and including whitespace must be <= 64 KiB (configurable). The document MUST be valid JSON (Marconi will validate). | ||
− | == Get Queue | + | === Get Queue Metadata === |
Line 383: | Line 584: | ||
<pre><nowiki> | <pre><nowiki> | ||
− | GET | + | GET /v1/queues/{queue_name}/metadata |
</nowiki></pre> | </nowiki></pre> | ||
Line 390: | Line 591: | ||
<pre><nowiki> | <pre><nowiki> | ||
− | GET /v1 | + | GET /v1/queues/fizbit/metadata HTTP/1.1 |
Host: marconi.example.com | Host: marconi.example.com | ||
Line 403: | Line 604: | ||
<pre><nowiki> | <pre><nowiki> | ||
{ | { | ||
− | + | "key": { | |
− | + | "key2": "value", | |
− | + | "key3": [1, 2, 3, 4, 5] | |
+ | } | ||
} | } | ||
</nowiki></pre> | </nowiki></pre> | ||
Line 412: | Line 614: | ||
'''Discussion''' | '''Discussion''' | ||
− | Returns queue | + | Returns queue metadata. |
− | == | + | === Get Queue Stats === |
Line 420: | Line 622: | ||
<pre><nowiki> | <pre><nowiki> | ||
− | + | GET /v1/queues/{queue_name}/stats | |
</nowiki></pre> | </nowiki></pre> | ||
Line 427: | Line 629: | ||
<pre><nowiki> | <pre><nowiki> | ||
− | + | GET /v1/queues/fizbit/stats HTTP/1.1 | |
Host: marconi.example.com | Host: marconi.example.com | ||
+ | |||
+ | ... | ||
+ | |||
</nowiki></pre> | </nowiki></pre> | ||
'''Response''' | '''Response''' | ||
+ | HTTP/1.1 200 OK | ||
<pre><nowiki> | <pre><nowiki> | ||
− | + | { | |
+ | "messages": { | ||
+ | "free": 146929, | ||
+ | "claimed": 2409, | ||
+ | "total": 149338, | ||
+ | "oldest": { | ||
+ | "href": "/v1/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b01", | ||
+ | "age": 63, | ||
+ | "created": "2013-08-12T20:44:55Z" | ||
+ | }, | ||
+ | "newest": { | ||
+ | "href": "/v1/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b01", | ||
+ | "age": 12, | ||
+ | "created": "2013-08-12T20:45:46Z" | ||
+ | } | ||
+ | } | ||
</nowiki></pre> | </nowiki></pre> | ||
Line 441: | Line 662: | ||
'''Discussion''' | '''Discussion''' | ||
− | + | Returns queue statistics, including how many messages are in the queue, broken out by status. | |
+ | |||
+ | NOTE: If total is 0, then "oldest" and "newest" message stats are not included. | ||
+ | |||
+ | == Messages == | ||
+ | All message-related operations require <code>Client-Id</code> to be included in the headers. This is to ensure that messages are not echoed back to the client that posted them unless the client explicitly requests this. | ||
− | == | + | === List Messages === |
Line 450: | Line 676: | ||
<pre><nowiki> | <pre><nowiki> | ||
− | GET | + | GET /v1/queues/{queue_name}/messages{?marker,limit,echo,include_claimed} |
</nowiki></pre> | </nowiki></pre> | ||
Line 457: | Line 683: | ||
<pre><nowiki> | <pre><nowiki> | ||
− | GET /v1 | + | GET /v1/queues/fizbit/messages?marker=1355-237242-783&limit=10 HTTP/1.1 |
Host: marconi.example.com | Host: marconi.example.com | ||
Line 471: | Line 697: | ||
<pre><nowiki> | <pre><nowiki> | ||
HTTP/1.1 200 OK | HTTP/1.1 200 OK | ||
− | Content-Location: /v1 | + | Content-Location: /v1/queues/fizbit/messages?marker=1355-237242-783&limit=10 |
... | ... | ||
Line 479: | Line 705: | ||
{ | { | ||
"rel": "next", | "rel": "next", | ||
− | "href": "/v1 | + | "href": "/v1/queues/fizbit/messages?marker=6244-244224-783&limit=10" |
} | } | ||
], | ], | ||
"messages": [ | "messages": [ | ||
{ | { | ||
− | "href": "/v1 | + | "href": "/v1/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b01", |
"ttl": 800, | "ttl": 800, | ||
"age": 790, | "age": 790, | ||
Line 503: | Line 729: | ||
Results are ordered by age, oldest message first. | Results are ordered by age, oldest message first. | ||
− | <code><nowiki>limit</nowiki></code> specifies up to | + | <code><nowiki>limit</nowiki></code> specifies up to 20 messages (configurable) to return. If not specified, limit defaults to 10. When more messages are available than can be returned in a single request, the client can pick up the next batch by simply using the URI template parameters returned from the previous call in the "next" field (TBD). |
<code><nowiki>marker</nowiki></code> is an opaque string that the client can use to request the next batch of messages. It communicates to the server which messages the client has already received. | <code><nowiki>marker</nowiki></code> is an opaque string that the client can use to request the next batch of messages. It communicates to the server which messages the client has already received. | ||
Line 509: | Line 735: | ||
Note: If <code><nowiki>marker</nowiki></code> is not specified, the API will return all messages at the head of the queue (up to limit). | Note: If <code><nowiki>marker</nowiki></code> is not specified, the API will return all messages at the head of the queue (up to limit). | ||
− | <code><nowiki>echo</nowiki></code> is a boolean (i.e., "true" or "false") that determines whether or not the API will return a client's own messages, as determined by the <code><nowiki>uuid</nowiki></code> portion of the | + | <code><nowiki>echo</nowiki></code> is a boolean (i.e., "true" or "false") that determines whether or not the API will return a client's own messages, as determined by the <code><nowiki>uuid</nowiki></code> portion of the Client-ID header. If not specified, echo defaults to "false". |
− | == Get a Specific Message == | + | <code><nowiki>include_claimed</nowiki></code> is a boolean (i.e., "true" or "false") that determines whether or not the API will return claimed messages as well as unclaimed ones. If not specified, defaults to "false" (only unclaimed messages are returned). |
+ | |||
+ | === Get a Specific Message === | ||
Line 517: | Line 745: | ||
<pre><nowiki> | <pre><nowiki> | ||
− | GET | + | GET /v1/queues/{queue_name}/messages/{message_id} |
</nowiki></pre> | </nowiki></pre> | ||
Line 524: | Line 752: | ||
<pre><nowiki> | <pre><nowiki> | ||
− | GET /v1 | + | GET /v1/queues/fizbat/messages/50b68a50d6f5b8c8a7c62b01 HTTP/1.1 |
Host: marconi.example.com | Host: marconi.example.com | ||
Line 536: | Line 764: | ||
<pre><nowiki> | <pre><nowiki> | ||
HTTP/1.1 200 OK | HTTP/1.1 200 OK | ||
− | Content-Location: /v1 | + | Content-Location: /v1/queues/fizbat/messages/50b68a50d6f5b8c8a7c62b01 |
... | ... | ||
{ | { | ||
− | "href": "/v1 | + | "href": "/v1/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b01", |
"ttl": 800, | "ttl": 800, | ||
"age": 790, | "age": 790, | ||
Line 564: | Line 792: | ||
<code><nowiki>body</nowiki></code> Arbitrary document submitted along with the original request to post the message. | <code><nowiki>body</nowiki></code> Arbitrary document submitted along with the original request to post the message. | ||
+ | If either the message ID or the claim ID is malformed or nonexistent, no message is returned. | ||
− | == Get a Set of Messages by ID == | + | === Get a Set of Messages by ID === |
Line 571: | Line 800: | ||
<pre><nowiki> | <pre><nowiki> | ||
− | GET | + | GET /v1/queues/{queue_name}/messages{?ids} |
</nowiki></pre> | </nowiki></pre> | ||
Line 578: | Line 807: | ||
<pre><nowiki> | <pre><nowiki> | ||
− | GET /v1 | + | GET /v1/queues/fizbat/messages?ids=50b68a50d6f5b8c8a7c62b01,f5b8c8a7c62b0150b68a50d6 HTTP/1.1 |
Host: marconi.example.com | Host: marconi.example.com | ||
Line 590: | Line 819: | ||
<pre><nowiki> | <pre><nowiki> | ||
HTTP/1.1 200 OK | HTTP/1.1 200 OK | ||
− | Content-Location: /v1 | + | Content-Location: /v1/queues/fizbat/messages?ids=50b68a50d6f5b8c8a7c62b01,f5b8c8a7c62b0150b68a50d6 |
... | ... | ||
Line 596: | Line 825: | ||
[ | [ | ||
{ | { | ||
− | "href": "/v1 | + | "href": "/v1/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b01", |
"ttl": 800, | "ttl": 800, | ||
"age": 32, | "age": 32, | ||
Line 605: | Line 834: | ||
}, | }, | ||
{ | { | ||
− | "href": "/v1 | + | "href": "/v1/queues/fizbit/messages/f5b8c8a7c62b0150b68a50d6", |
"ttl": 800, | "ttl": 800, | ||
"age": 790, | "age": 790, | ||
Line 619: | Line 848: | ||
'''Discussion''' | '''Discussion''' | ||
− | + | Unlike message listing, a client's own messages are always returned in this approach. Note that the list of ids may not exceed 20 (configurable, shared setting with number of messages that can be posted at once). | |
− | == Post Message(s) == | + | This approach returns all the messages with matching IDs. If a malformed ID or a nonexistent message ID is provided, it is ignored and the remaining messages are returned. |
+ | |||
+ | === Post Message(s) === | ||
Line 627: | Line 858: | ||
<pre><nowiki> | <pre><nowiki> | ||
− | POST | + | POST /v1/queues/{queue_name}/messages |
</nowiki></pre> | </nowiki></pre> | ||
Line 634: | Line 865: | ||
<pre><nowiki> | <pre><nowiki> | ||
− | POST /v1 | + | POST /v1/queues/fizbit/messages HTTP/1.1 |
Host: marconi.example.com | Host: marconi.example.com | ||
Line 665: | Line 896: | ||
<pre><nowiki> | <pre><nowiki> | ||
HTTP/1.1 201 Created | HTTP/1.1 201 Created | ||
− | Location: /v1/ | + | Location: /v1/queues/fizbit/messages?ids=50b68a50d6f5b8c8a7c62b01 |
+ | Content-Type: application/json | ||
+ | |||
+ | { | ||
+ | "resources": [ | ||
+ | "/v1/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b01" | ||
+ | ], | ||
+ | "partial": false | ||
+ | } | ||
</nowiki></pre> | </nowiki></pre> | ||
Line 672: | Line 911: | ||
<pre><nowiki> | <pre><nowiki> | ||
HTTP/1.1 201 Created | HTTP/1.1 201 Created | ||
− | Location: /v1 | + | Location: /v1/queues/fizbit/messages?ids=50b68a50d6f5b8c8a7c62b01,50b68a50d6f5b8c8a7c62b05 |
+ | Content-Type: application/json | ||
+ | |||
+ | { | ||
+ | "resources": [ | ||
+ | "/v1/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b01", | ||
+ | "/v1/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b05" | ||
+ | ], | ||
+ | "partial": false | ||
+ | } | ||
</nowiki></pre> | </nowiki></pre> | ||
'''Discussion''' | '''Discussion''' | ||
− | Up to | + | Up to 20 messages (default, but configurable) may be submitted in a single request, but must always be encapsulated in a collection container (an array in JSON). The resulting value of the Location header or response body may be used to retrieve the created messages for further processing if needed. |
The client only specifies the body and ttl for the message; the server will insert metadata such as id and age. | The client only specifies the body and ttl for the message; the server will insert metadata such as id and age. | ||
− | <code><nowiki> | + | The response body contains a list of resource paths corresponding to each message submitted in the request, in the same order. In the case of a server-side error part-way through the processing of the submitted messages, a partial list will be returned, <code><nowiki>partial</nowiki></code> will be '''true''', and the client will need to retry posting the remaining messages. If no messages could be enqueued, the server will return 503 Service Unavailable. |
+ | |||
+ | NOTE: The ''partial'' field has been deprecated since the icehouse release, and will always be "false"; drivers are not longer allowed to return a "partial" success and so must either succeed or fail the entire batch of messages. | ||
+ | |||
+ | The size of the request document, in characters including whitespace, is 256 KiB by default (configurable). The document MUST be valid JSON (Marconi will validate). | ||
+ | |||
+ | <code><nowiki>body</nowiki></code> specifies an arbitrary document which constitutes the body of the message being sent. | ||
− | <code><nowiki>ttl</nowiki></code> is how long the server should wait before expiring and removing the message from the queue. Value MUST between 60 and 1209600 seconds (14 days). Note that the server may not actually delete the message until it's age has reached up to (ttl + 60) seconds, to allow for flexibility in storage implementations. | + | <code><nowiki>ttl</nowiki></code> is how long the server should wait before expiring and removing the message from the queue. Value MUST be between 60 and 1209600 seconds (14 days, configurable), inclusive. Note that the server may not actually delete the message until it's age has reached up to (ttl + 60) seconds, to allow for flexibility in storage implementations. |
− | == Delete Message == | + | === Delete Message === |
Line 691: | Line 945: | ||
<pre><nowiki> | <pre><nowiki> | ||
− | DELETE | + | DELETE /v1/queues/{queue_name}/messages/{message_id}{?claim_id} |
</nowiki></pre> | </nowiki></pre> | ||
Line 698: | Line 952: | ||
<pre><nowiki> | <pre><nowiki> | ||
− | DELETE /v1 | + | DELETE /v1/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b01 HTTP/1.1 |
Host: marconi.example.com | Host: marconi.example.com | ||
Line 719: | Line 973: | ||
<code><nowiki>claim_id</nowiki></code> specifies that the message should only be deleted if it has the specified claim ID, and that claim has not expired. This is useful for ensuring only one agent processes any given message; whenever a worker client's claim expires before it has a chance to delete a message it has processed, the worker must roll back any actions it took based on that message, since another worker will now be able to claim and process the same message. | <code><nowiki>claim_id</nowiki></code> specifies that the message should only be deleted if it has the specified claim ID, and that claim has not expired. This is useful for ensuring only one agent processes any given message; whenever a worker client's claim expires before it has a chance to delete a message it has processed, the worker must roll back any actions it took based on that message, since another worker will now be able to claim and process the same message. | ||
+ | Note that if claim_id is not specified, but the message is claimed, the operation will fail. In other words, claimed messages can only ever be deleted by providing an appropriate claim_id. | ||
+ | |||
+ | === Delete a Set of Messages by ID === | ||
+ | |||
+ | '''Template''' | ||
+ | |||
+ | <pre><nowiki> | ||
+ | DELETE /v1/queues/{queue_name}/messages{?ids} | ||
+ | </nowiki></pre> | ||
+ | |||
+ | |||
+ | '''Request''' | ||
+ | |||
+ | <pre><nowiki> | ||
+ | DELETE /v1/queues/fizbit/messages?ids=50b68a50d6f5b8c8a7c62b01,50b68a50d6f5b8c8a7c62b02 HTTP/1.1 | ||
+ | Host: marconi.example.com | ||
+ | |||
+ | ... | ||
− | == Claim Messages == | + | </nowiki></pre> |
+ | |||
+ | |||
+ | '''Response''' | ||
+ | |||
+ | <pre><nowiki> | ||
+ | HTTP/1.1 204 No Content | ||
+ | </nowiki></pre> | ||
+ | |||
+ | |||
+ | '''Discussion''' | ||
+ | |||
+ | Bulk delete for messages. | ||
+ | |||
+ | <code><nowiki>ids</nowiki></code> specifies the messages to delete, up to a maximum of 20 (configurable, shares the same setting as posting and listing messages) | ||
+ | |||
+ | If any of the message IDs are malformed or non-existent, they are ignored. The remaining valid message IDs are deleted. | ||
+ | |||
+ | == Claims == | ||
+ | |||
+ | === Claim Messages === | ||
Line 726: | Line 1,018: | ||
<pre><nowiki> | <pre><nowiki> | ||
− | POST | + | POST /v1/queues/{queue_name}/claims{?limit} |
Content-Type: application/json | Content-Type: application/json | ||
Line 741: | Line 1,033: | ||
<pre><nowiki> | <pre><nowiki> | ||
− | POST /v1 | + | POST /v1/queues/fizbit/claims?limit=5 HTTP/1.1 |
Content-Type: application/json | Content-Type: application/json | ||
Accept: application/json | Accept: application/json | ||
Line 759: | Line 1,051: | ||
<pre><nowiki> | <pre><nowiki> | ||
− | HTTP/1.1 | + | HTTP/1.1 201 Created |
Content-Type: application/json | Content-Type: application/json | ||
− | Location: /v1 | + | Location: /v1/queues/foo-bar/claims/a28ee94e-6cb4-11e2-b4d5-7703267a7926 |
... | ... | ||
Line 767: | Line 1,059: | ||
[ | [ | ||
{ | { | ||
− | "href": "/v1 | + | "href": "/v1/queues/foo-bar/messages/50b68a50d6f5b8c8a7c62b01?claim_id=a28ee94e-6cb4-11e2-b4d5-7703267a7926", |
"ttl": 800, | "ttl": 800, | ||
"age": 100, | "age": 100, | ||
Line 776: | Line 1,068: | ||
}, | }, | ||
{ | { | ||
− | "href": "/v1 | + | "href": "/v1/queues/foo-bar/messages/50b68a50d6f5b8c8a7c62b02?claim_id=a28ee94e-6cb4-11e2-b4d5-7703267a7926", |
"ttl": 800, | "ttl": 800, | ||
"age": 790, | "age": 790, | ||
Line 798: | Line 1,090: | ||
When a claim expires, it is removed, allowing another client worker to claim the message in the case that the original worker fails to process it. | When a claim expires, it is removed, allowing another client worker to claim the message in the case that the original worker fails to process it. | ||
− | <code><nowiki>limit</nowiki></code> specifies up to | + | <code><nowiki>limit</nowiki></code> specifies up to 20 messages (configurable) to claim. If not specified, limit defaults to 10. Note that claim creation is best-effort, meaning the server may claim and return less than the requested number of messages. |
− | <code><nowiki>ttl</nowiki></code> is how long the server should wait before releasing the claim. Value MUST be between 60 and 43200 seconds (12 hours). | + | <code><nowiki>ttl</nowiki></code> is how long the server should wait before releasing the claim. Value MUST be between 60 and 43200 seconds (12 hours, configurable). |
− | <code><nowiki>grace</nowiki></code> is the message grace period in seconds. Value MUST be between 60 and 43200 seconds (12 hours). The server will extend the lifetime of claimed messages to be at least as long as the lifetime of the claim itself, plus a specified grace period to deal with crashed workers (up to 1209600 or 14 days including claim lifetime). If a claimed message would normally live longer than the grace period, it's expiration will not be adjusted. | + | <code><nowiki>grace</nowiki></code> is the message grace period in seconds. Value MUST be between 60 and 43200 seconds (12 hours, configurable). The server will extend the lifetime of claimed messages to be at least as long as the lifetime of the claim itself, plus a specified grace period to deal with crashed workers (up to 1209600 or 14 days including claim lifetime). If a claimed message would normally live longer than the grace period, it's expiration will not be adjusted. |
− | == Query Claim == | + | === Query Claim === |
Line 810: | Line 1,102: | ||
<pre><nowiki> | <pre><nowiki> | ||
− | GET | + | GET /v1/queues/{queue_name}/claims/{claim_id} |
</nowiki></pre> | </nowiki></pre> | ||
Line 817: | Line 1,109: | ||
<pre><nowiki> | <pre><nowiki> | ||
− | GET /v1 | + | GET /v1/queues/foo-bar/claims/a28ee94e-6cb4-11e2-b4d5-7703267a7926 |
Host: marconi.example.com | Host: marconi.example.com | ||
Line 829: | Line 1,121: | ||
<pre><nowiki> | <pre><nowiki> | ||
HTTP/1.1 200 OK | HTTP/1.1 200 OK | ||
− | Content-Location: /v1 | + | Content-Location: /v1/queues/foo-bar/claims/a28ee94e-6cb4-11e2-b4d5-7703267a7926 |
... | ... | ||
Line 845: | Line 1,137: | ||
'''Discussion''' | '''Discussion''' | ||
− | == Update Claim == | + | === Update Claim === |
Line 851: | Line 1,143: | ||
<pre><nowiki> | <pre><nowiki> | ||
− | PATCH | + | PATCH /v1/queues/{queue_name}/claims/{claim_id} |
Content-Type: application/json | Content-Type: application/json | ||
</nowiki></pre> | </nowiki></pre> | ||
Line 859: | Line 1,151: | ||
<pre><nowiki> | <pre><nowiki> | ||
− | PATCH /v1 | + | PATCH /v1/queues/fizbit/claims/a28ee94e-6cb4-11e2-b4d5-7703267a7926 HTTP/1.1 |
Content-Type: application/json | Content-Type: application/json | ||
Line 879: | Line 1,171: | ||
Clients should periodically renew claims during long-running batches of work to avoid loosing a claim in the middle of processing a message. This is done by issuing a PATCH to a specific claim resource and including a new TTL for the claim (which may be different from the original TTL). The server will reset the age of the claim and apply the new TTL. | Clients should periodically renew claims during long-running batches of work to avoid loosing a claim in the middle of processing a message. This is done by issuing a PATCH to a specific claim resource and including a new TTL for the claim (which may be different from the original TTL). The server will reset the age of the claim and apply the new TTL. | ||
− | == Release Claim == | + | === Release Claim === |
Line 885: | Line 1,177: | ||
<pre><nowiki> | <pre><nowiki> | ||
− | DELETE | + | DELETE /v1/queues/{queue_name}/claims/{claim_id} |
</nowiki></pre> | </nowiki></pre> | ||
Line 892: | Line 1,184: | ||
<pre><nowiki> | <pre><nowiki> | ||
− | DELETE /v1 | + | DELETE /v1/queues/foo-bar/claims/a28ee94e-6cb4-11e2-b4d5-7703267a7926 HTTP/1.1 |
Host: marconi.example.com | Host: marconi.example.com | ||
</nowiki></pre> | </nowiki></pre> | ||
Line 909: | Line 1,201: | ||
This operation is useful when a worker is performing a graceful shutdown, fails to process one or more messages, or is taking longer than expected to process messages, and wishes to make the remainder available to other workers. | This operation is useful when a worker is performing a graceful shutdown, fails to process one or more messages, or is taking longer than expected to process messages, and wishes to make the remainder available to other workers. | ||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− |
Latest revision as of 19:06, 15 September 2014
Contents
Marconi API: v1 Blueprint
The Marconi v1 API is FROZEN
We have started collecting feedback with an eye toward extensions and v2 of the API here: Marconi/specs/api/next
Overview
Marconi provides an HTTP-based API in the spirit of the REST architecture style.
This guide assumes the reader is familiar with REST, HTTP/1.1 and JSON. URI templates use the same syntax as RFC 6570.
Note: Error responses are enumerated on a separate page: Marconi/specs/api/v1/errors
Common API Elements
Clients
- Clients should follow HTTP redirects
- Clients should advertise gzip support
- Clients must identify themselves in the UserAgent request header; e.g., "User-Agent: python/2.7 cloudthing/1.2"
- Clients must not hard-code URI paths or templates since they may change over time. Instead, clients should cache links and link templates provided by the API.
API Versioning
The Marconi API uses a URI versioning scheme. The first element of the path contains the target version identifier, e.g.:
https://marconi.example.com/v1
The URI version will only be incremented to accommodate major new features or API redesigns that can not be made backwards-compatible. When new API versions are released, older versions are deprecated. Marconi maintainers will work with developers and partners to ensure there is adequate time to migrate to new versions before deprecated ones are discontinued.
Since the version is only incremented to accommodate major API revisions, sub-versioning of the API will be rare, but may be used occasionally to release a "polished" major version.
Resource Media Types
The API supports `application/json` (XML is on the road map), encoded as UTF-8. Gzip'd requests are optionally supported by the server.
Unrecognized protocol elements should be ignored. This includes XML elements and attributes, JSON object properties, link relation types, media types, etc.
Authentication
May be optionally implemented by middleware or reverse proxy. If implemented, the following applies:
All requests to the API may only be performed by an authenticated agent.
To authenticate, an agent issues an authentication request to a Keystone Identity Service endpoint.
In response to valid credentials, Keystone responds with an auth token and a service catalog that contains a list of all services and endpoints available for the given token. Multiple endpoints may be returned for Marconi according to physical locations and performance/availability characteristics of different deployments.
Normally, Keystone middleware provides the X-Project-Id header based on the auth token submitted by the Marconi client. For this to work, clients MUST specify a valid auth token in the `X-Auth-Token` header for each request to the Marconi API. The API validates auth tokens against Keystone before servicing each request
If auth is not enabled, clients must provide the X-Project-Id header themselves.
Authorization
May be optionally implemented by middleware or reverse proxy. If implemented, the following applies:
The API needs to verify read/write access to all endpoints according to the provided auth token (RBAC). The ACL should be cached with the token.
Endpoints
Clients may choose to interact with any of the service endpoints retrieved from Keystone. Agents use the given endpoint as a home document href from which to discover links to all other API requests. Endpoints may be organized by region
An example endpoint:
https://marconi.example.com/v1
Errors
If any request results in an error, the server will return an appropriate 4xx or 5xx HTTP status code, and optionally the following information in the body:
- Title
- Description
- Internal code
- Link to more information
Example:
HTTP/1.1 400 Bad Request Content-Type: application/json { "title": "Unsupported limit", "description": "The given limit cannot be negative, and cannot be greater than 50.", "code": 1092, "link": { "rel": "help", "href": "http://docs.example.com/messages#limit", "text": "API documentation for the limit parameter" } }
Common Headers
Each request to the API must include certain standard and extended HTTP headers. These headers provide host, agent, authentication and other pertinent information to the server.
Header | Description |
Host | The host name of the API, as referenced by the Keystone service catalog |
Date | The current date and time, using the standard RFC 1123 HTTP date format |
Accept | Media type desired; only application/json is supported at this time.
|
Accept-Encoding | Specifies that the agent accepts gzip-encoded response bodies |
Content-Length | For POST or PUT requests, the length in bytes of the message document being submitted |
X-Auth-Token | Keystone auth token |
X-Project-Id | An ID for a project to which the value of X-Auth-Token grants access. Queues will be created under this project. |
Client-ID | A UUID, used to distinguish each client using the service. The UUID must be submitted in its canonical form (e.g., 3381af92-2b9e-11e3-b191-71861300734c ). In Marconi, this is used to avoid echoing a sender's messages back to the same instance, and may be logged by the server for future use. Should be generated once and persisted between restarts of the client.
|
HTTP Response Codes
See Response Codes
Sample API Request
GET /v1/queues/fizbat/messages?marker=1355-237242-783&limit=10 HTTP/1.1 Host: marconi.example.com User-Agent: python/2.7 killer-rabbit/1.2 Date: Wed, 28 Nov 2012 21:14:19 GMT Accept: application/json Accept-Encoding: gzip X-Auth-Token: 7d2f63fd-4dcc-4752-8e9b-1d08f989cc00 X-Project-Id: 518b51ea133c4facadae42c328d6b77b Client-ID: 30387f00-39a0-11e2-be4d-a8d15f34bae2
Endpoints Synopsis
# Base endpoints GET /v1 GET /v1/health # Queues GET /v1/queues{?marker,limit,detailed} GET /v1/queues/{queue_name} # existence check HEAD /v1/queues/{queue_name} # existence check PUT /v1/queues/{name} DELETE /v1/queues/{name} # Queue metadata PUT /v1/queues/{queue_name}/metadata GET /v1/queues/{queue_name}/metadata GET /v1/queues/{queue_name}/stats # Messages GET /v1/queues/{queue_name}/messages{?marker,limit,echo,include_claimed} GET /v1/queues/{queue_name}/messages/{message_id} GET /v1/queues/{queue_name}/messages{?ids} POST /v1/queues/{queue_name}/messages DELETE /v1/queues/{queue_name}/messages/{message_id}{?claim_id} DELETE /v1/queues/{queue_name}/messages{?ids} # Claims POST /v1/queues/{queue_name}/claims{?limit} GET /v1/queues/{queue_name}/claims/{claim_id} PATCH /v1/queues/{queue_name}/claims/{claim_id} DELETE /v1/queues/{queue_name}/claims/{claim_id}
Base Endpoints
Get Home Document
Template
GET /v1
Request
GET /v1 HTTP/1.1 Host: marconi.example.com ...
Response
HTTP/1.0 200 OK ... { "resources": { "rel/queue-stats": { "href-template": "/v1/queues/{queue_name}/stats", "href-vars": { "queue_name": "param/queue_name" }, "hints": { "allow": [ "GET" ], "formats": { "application/json": {} } } }, "rel/post-messages": { "href-template": "/v1/queues/{queue_name}/messages", "href-vars": { "queue_name": "param/queue_name" }, "hints": { "accept-post": [ "application/json" ], "allow": [ "POST" ], "formats": { "application/json": {} } } }, "rel/queue": { "href-template": "/v1/queues/{queue_name}", "href-vars": { "queue_name": "param/queue_name" }, "hints": { "allow": [ "GET", "HEAD", "PUT", "DELETE" ], "formats": { "application/json": {} } } }, "rel/queue-metadata": { "href-template": "/v1/queues/{queue_name}/metadata", "href-vars": { "queue_name": "param/queue_name" }, "hints": { "allow": [ "GET", "PUT" ], "formats": { "application/json": {} } } }, "rel/queues": { "href-template": "/v1/queues{?marker,limit,detailed}", "href-vars": { "marker": "param/marker", "detailed": "param/detailed", "limit": "param/queue_limit" }, "hints": { "allow": [ "GET" ], "formats": { "application/json": {} } } }, "rel/messages": { "href-template": "/v1/queues/{queue_name}/messages{?marker,limit,echo,include_claimed}", "href-vars": { "marker": "param/marker", "include_claimed": "param/include_claimed", "queue_name": "param/queue_name", "limit": "param/messages_limit", "echo": "param/echo" }, "hints": { "allow": [ "GET" ], "formats": { "application/json": {} } } }, "rel/claim": { "href-template": "/v1/queues/{queue_name}/claims{?limit}", "href-vars": { "queue_name": "param/queue_name", "limit": "param/claim_limit" }, "hints": { "accept-post": [ "application/json" ], "allow": [ "POST" ], "formats": { "application/json": {} } } } } }
Discussion
The entire Marconi API is discoverable from a single starting point; agents do not need to know any more than this one URI in order to explore the entire API.
This document is cache-able (and should be cached by proxy and client).
See also: http://tools.ietf.org/html/draft-nottingham-json-home-02
Check Node Health
Template
GET /v1/health
or
HEAD /v1/health
Request
GET /v1/health HTTP/1.1 Host: example.marconi.com ...
Response
HTTP/1.1 204 No Content
or
HTTP/1.1 503 Service Unavailable
Discussion
Use this request to check on the Marconi node status. If the node is down, this endpoint will not be reachable and will act as a signal to the load balancer to take the node out of rotation. Alternatively, the endpoint may be reachable but the service is temporarily unavailable due to storage driver failure or some other error.
Queues
Create Queue
Template
PUT /v1/queues/{queue_name}
Request
PUT /v1/queues/fizbat HTTP/1.1 Host: marconi.example.com
Response
HTTP/1.1 201 Created Location: /v1/queues/fizbit
Discussion
Creates a queue.
The body of the PUT is empty.
queue_name
is the name to give the queue. The name MUST NOT exceed 64 bytes in length, and is limited to US-ASCII letters, digits, underscores and hyphens.
List Queues
Template
GET /v1/queues{?marker,limit,detailed}
Request
GET /v1/queues?marker=baz&detailed=true HTTP/1.1 Host: marconi.example.com ...
Response
HTTP/1.1 200 OK { "links": [ { "rel": "next", "href": "/v1/queues?marker=kooleo&limit=10&detailed=true" } ], "queues": [ { "name": "boomerang", "href": "/v1/queues/boomerang", "metadata": {} }, { "name": "fizbit", "href": "/v1/queues/fizbit", "metadata": { "handle": "@kgriffs" } }, ... { "name": "kooleo", "href": "/v1/queues/kooleo", "metadata": { "something": "something_else" } } ] }
Checking Queue Existence
Template
GET /v1/queues/{queue_name}
Request
GET /v1/queues/{queue_name} Host: marconi.example.com
Response
HTTP/1.1 204 OK
Discussion
Returns 204 if the queue exists, else 404.
HEAD
also works.
Delete Queue
Template
DELETE /v1/queues/{queue_name}
Request
DELETE /v1/queues/fizbat HTTP/1.1 Host: marconi.example.com
Response
HTTP/1.1 204 No Content
Discussion
Use this operation to immediately delete a queue along with all its messages (if any).
Queue Metadata
Set Queue Metadata
Template
PUT /v1/queues/{queue_name}/metadata { ... }
Request
PUT /v1/queues/fizbat/metadata HTTP/1.1 Host: marconi.example.com ... { "key": { "key2": "value", "key3": [1, 2, 3, 4, 5] } }
Response
HTTP/1.1 204 No Content
Discussion
This operation allows for setting structured metadata. If PUT is used, this document will replace the existing metadata document in it's entirety, so clients need to be careful they do not accidentally delete fields.
The body of the request is an arbitrary document which allows storing contextual information about the way a particular application should interact with the queue. The size of this body, in characters and including whitespace must be <= 64 KiB (configurable). The document MUST be valid JSON (Marconi will validate).
Get Queue Metadata
Template
GET /v1/queues/{queue_name}/metadata
Request
GET /v1/queues/fizbit/metadata HTTP/1.1 Host: marconi.example.com ...
Response
HTTP/1.1 200 OK
{ "key": { "key2": "value", "key3": [1, 2, 3, 4, 5] } }
Discussion
Returns queue metadata.
Get Queue Stats
Template
GET /v1/queues/{queue_name}/stats
Request
GET /v1/queues/fizbit/stats HTTP/1.1 Host: marconi.example.com ...
Response
HTTP/1.1 200 OK
{ "messages": { "free": 146929, "claimed": 2409, "total": 149338, "oldest": { "href": "/v1/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b01", "age": 63, "created": "2013-08-12T20:44:55Z" }, "newest": { "href": "/v1/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b01", "age": 12, "created": "2013-08-12T20:45:46Z" } }
Discussion
Returns queue statistics, including how many messages are in the queue, broken out by status.
NOTE: If total is 0, then "oldest" and "newest" message stats are not included.
Messages
All message-related operations require Client-Id
to be included in the headers. This is to ensure that messages are not echoed back to the client that posted them unless the client explicitly requests this.
List Messages
Template
GET /v1/queues/{queue_name}/messages{?marker,limit,echo,include_claimed}
Request
GET /v1/queues/fizbit/messages?marker=1355-237242-783&limit=10 HTTP/1.1 Host: marconi.example.com ...
Response
HTTP 200 if the query matched 1 or more messages, HTTP 204 otherwise (with no body).
HTTP/1.1 200 OK Content-Location: /v1/queues/fizbit/messages?marker=1355-237242-783&limit=10 ... { "links": [ { "rel": "next", "href": "/v1/queues/fizbit/messages?marker=6244-244224-783&limit=10" } ], "messages": [ { "href": "/v1/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b01", "ttl": 800, "age": 790, "body": { "event": "ActivateAccount", "mode": "active" } } ] }
Discussion
Message IDs and markers are opaque strings; clients should make no assumptions about their format or length. Furthermore, clients should assume there is no relationship between markers and message IDs (i.e., one cannot be derived from the other). This allows for a wide variety of storage driver implementations.
Results are ordered by age, oldest message first.
limit
specifies up to 20 messages (configurable) to return. If not specified, limit defaults to 10. When more messages are available than can be returned in a single request, the client can pick up the next batch by simply using the URI template parameters returned from the previous call in the "next" field (TBD).
marker
is an opaque string that the client can use to request the next batch of messages. It communicates to the server which messages the client has already received.
Note: If marker
is not specified, the API will return all messages at the head of the queue (up to limit).
echo
is a boolean (i.e., "true" or "false") that determines whether or not the API will return a client's own messages, as determined by the uuid
portion of the Client-ID header. If not specified, echo defaults to "false".
include_claimed
is a boolean (i.e., "true" or "false") that determines whether or not the API will return claimed messages as well as unclaimed ones. If not specified, defaults to "false" (only unclaimed messages are returned).
Get a Specific Message
Template
GET /v1/queues/{queue_name}/messages/{message_id}
Request
GET /v1/queues/fizbat/messages/50b68a50d6f5b8c8a7c62b01 HTTP/1.1 Host: marconi.example.com ...
Response
HTTP/1.1 200 OK Content-Location: /v1/queues/fizbat/messages/50b68a50d6f5b8c8a7c62b01 ... { "href": "/v1/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b01", "ttl": 800, "age": 790, "body": { "event": "ActivateAccount", "mode": "active" } }
Discussion
Message fields are defined as follows:
href
is an opaque relative URI that the client can use to uniquely identify a message resource, and interact with it.
ttl
is the ttl set on the message when it was posted. The message will expire after (ttl - age) seconds.
age
number of seconds since ts, relative to the server's clock.
body
Arbitrary document submitted along with the original request to post the message.
If either the message ID or the claim ID is malformed or nonexistent, no message is returned.
Get a Set of Messages by ID
Template
GET /v1/queues/{queue_name}/messages{?ids}
Request
GET /v1/queues/fizbat/messages?ids=50b68a50d6f5b8c8a7c62b01,f5b8c8a7c62b0150b68a50d6 HTTP/1.1 Host: marconi.example.com ...
Response
HTTP/1.1 200 OK Content-Location: /v1/queues/fizbat/messages?ids=50b68a50d6f5b8c8a7c62b01,f5b8c8a7c62b0150b68a50d6 ... [ { "href": "/v1/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b01", "ttl": 800, "age": 32, "body": { "cmd": "EncodeVideo", "jobid": 58229 } }, { "href": "/v1/queues/fizbit/messages/f5b8c8a7c62b0150b68a50d6", "ttl": 800, "age": 790, "body": { "cmd": "EncodeAudio", "jobid": 58201 } } ]
Discussion
Unlike message listing, a client's own messages are always returned in this approach. Note that the list of ids may not exceed 20 (configurable, shared setting with number of messages that can be posted at once).
This approach returns all the messages with matching IDs. If a malformed ID or a nonexistent message ID is provided, it is ignored and the remaining messages are returned.
Post Message(s)
Template
POST /v1/queues/{queue_name}/messages
Request
POST /v1/queues/fizbit/messages HTTP/1.1 Host: marconi.example.com ... [ { "ttl": 300, "body": { "event": "BackupStarted", "backup_id": "c378813c-3f0b-11e2-ad92-7823d2b0f3ce" } }, { "ttl": 60, "body": { "event": "BackupProgress", "current_bytes": "0", "total_bytes": "99614720" } } ]
Responses
When a single message is submitted:
HTTP/1.1 201 Created Location: /v1/queues/fizbit/messages?ids=50b68a50d6f5b8c8a7c62b01 Content-Type: application/json { "resources": [ "/v1/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b01" ], "partial": false }
When multiple messages are submitted:
HTTP/1.1 201 Created Location: /v1/queues/fizbit/messages?ids=50b68a50d6f5b8c8a7c62b01,50b68a50d6f5b8c8a7c62b05 Content-Type: application/json { "resources": [ "/v1/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b01", "/v1/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b05" ], "partial": false }
Discussion
Up to 20 messages (default, but configurable) may be submitted in a single request, but must always be encapsulated in a collection container (an array in JSON). The resulting value of the Location header or response body may be used to retrieve the created messages for further processing if needed.
The client only specifies the body and ttl for the message; the server will insert metadata such as id and age.
The response body contains a list of resource paths corresponding to each message submitted in the request, in the same order. In the case of a server-side error part-way through the processing of the submitted messages, a partial list will be returned, partial
will be true, and the client will need to retry posting the remaining messages. If no messages could be enqueued, the server will return 503 Service Unavailable.
NOTE: The partial field has been deprecated since the icehouse release, and will always be "false"; drivers are not longer allowed to return a "partial" success and so must either succeed or fail the entire batch of messages.
The size of the request document, in characters including whitespace, is 256 KiB by default (configurable). The document MUST be valid JSON (Marconi will validate).
body
specifies an arbitrary document which constitutes the body of the message being sent.
ttl
is how long the server should wait before expiring and removing the message from the queue. Value MUST be between 60 and 1209600 seconds (14 days, configurable), inclusive. Note that the server may not actually delete the message until it's age has reached up to (ttl + 60) seconds, to allow for flexibility in storage implementations.
Delete Message
Template
DELETE /v1/queues/{queue_name}/messages/{message_id}{?claim_id}
Request
DELETE /v1/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b01 HTTP/1.1 Host: marconi.example.com ...
Response
HTTP/1.1 204 No Content
Discussion
message_id
specifies the message to delete.
claim_id
specifies that the message should only be deleted if it has the specified claim ID, and that claim has not expired. This is useful for ensuring only one agent processes any given message; whenever a worker client's claim expires before it has a chance to delete a message it has processed, the worker must roll back any actions it took based on that message, since another worker will now be able to claim and process the same message.
Note that if claim_id is not specified, but the message is claimed, the operation will fail. In other words, claimed messages can only ever be deleted by providing an appropriate claim_id.
Delete a Set of Messages by ID
Template
DELETE /v1/queues/{queue_name}/messages{?ids}
Request
DELETE /v1/queues/fizbit/messages?ids=50b68a50d6f5b8c8a7c62b01,50b68a50d6f5b8c8a7c62b02 HTTP/1.1 Host: marconi.example.com ...
Response
HTTP/1.1 204 No Content
Discussion
Bulk delete for messages.
ids
specifies the messages to delete, up to a maximum of 20 (configurable, shares the same setting as posting and listing messages)
If any of the message IDs are malformed or non-existent, they are ignored. The remaining valid message IDs are deleted.
Claims
Claim Messages
Template
POST /v1/queues/{queue_name}/claims{?limit} Content-Type: application/json ... { "ttl": {claim_ttl}, "grace": {message_grace} }
Request
POST /v1/queues/fizbit/claims?limit=5 HTTP/1.1 Content-Type: application/json Accept: application/json ... { "ttl": 300, "grace": 300 }
Response
The client receives a claim URI and a list of claimed messages, if any:
HTTP/1.1 201 Created Content-Type: application/json Location: /v1/queues/foo-bar/claims/a28ee94e-6cb4-11e2-b4d5-7703267a7926 ... [ { "href": "/v1/queues/foo-bar/messages/50b68a50d6f5b8c8a7c62b01?claim_id=a28ee94e-6cb4-11e2-b4d5-7703267a7926", "ttl": 800, "age": 100, "body": { "object_id": "8a50d6", "target": "h.264" } }, { "href": "/v1/queues/foo-bar/messages/50b68a50d6f5b8c8a7c62b02?claim_id=a28ee94e-6cb4-11e2-b4d5-7703267a7926", "ttl": 800, "age": 790, "body": { "object_id": "fb8c8a", "target": "h.264" } } ]
Discussion
Claims a set of messages, up to limit, from oldest to newest, skipping any that are already claimed. If no unclaimed messages are available, returns 204 No Content.
The client should delete the message when it has finished processing it, before the claim expires, to ensure the message is processed only once. As part of the delete operation, all worker clients should specify the claim ID (this is best done by simply using the provided href). That way, the server can return an error if the claim just expired (notifying the client of the race condition), giving the worker a chance to roll back its own processing of the given message, since another worker will likely claim the message and process it.
Just as with a message's age, the age given for the claim is relative to the server's clock, and is useful for determining how quickly messages are getting processed, and whether a given message's claim is about to expire.
When a claim expires, it is removed, allowing another client worker to claim the message in the case that the original worker fails to process it.
limit
specifies up to 20 messages (configurable) to claim. If not specified, limit defaults to 10. Note that claim creation is best-effort, meaning the server may claim and return less than the requested number of messages.
ttl
is how long the server should wait before releasing the claim. Value MUST be between 60 and 43200 seconds (12 hours, configurable).
grace
is the message grace period in seconds. Value MUST be between 60 and 43200 seconds (12 hours, configurable). The server will extend the lifetime of claimed messages to be at least as long as the lifetime of the claim itself, plus a specified grace period to deal with crashed workers (up to 1209600 or 14 days including claim lifetime). If a claimed message would normally live longer than the grace period, it's expiration will not be adjusted.
Query Claim
Template
GET /v1/queues/{queue_name}/claims/{claim_id}
Request
GET /v1/queues/foo-bar/claims/a28ee94e-6cb4-11e2-b4d5-7703267a7926 Host: marconi.example.com ...
Response
HTTP/1.1 200 OK Content-Location: /v1/queues/foo-bar/claims/a28ee94e-6cb4-11e2-b4d5-7703267a7926 ... { "age": 19, "ttl": 30, "messages": [ ... ] }
Discussion
Update Claim
Template
PATCH /v1/queues/{queue_name}/claims/{claim_id} Content-Type: application/json
Request
PATCH /v1/queues/fizbit/claims/a28ee94e-6cb4-11e2-b4d5-7703267a7926 HTTP/1.1 Content-Type: application/json ... { "ttl": 300 }
Response
HTTP/1.1 204 No Content
Discussion
Clients should periodically renew claims during long-running batches of work to avoid loosing a claim in the middle of processing a message. This is done by issuing a PATCH to a specific claim resource and including a new TTL for the claim (which may be different from the original TTL). The server will reset the age of the claim and apply the new TTL.
Release Claim
Template
DELETE /v1/queues/{queue_name}/claims/{claim_id}
Request
DELETE /v1/queues/foo-bar/claims/a28ee94e-6cb4-11e2-b4d5-7703267a7926 HTTP/1.1 Host: marconi.example.com
Response
HTTP/1.1 204 No Content
Discussion
Use this operation to immediately release a claim, making any (remaining, non-deleted) messages associated with the claim available to other workers.
This operation is useful when a worker is performing a graceful shutdown, fails to process one or more messages, or is taking longer than expected to process messages, and wishes to make the remainder available to other workers.