Jump to: navigation, search

Difference between revisions of "Zaqar/specs/api/v1"

(Added todo's, locking semantics.)
m (Text replace - "__NOTOC__" to "")
(11 intermediate revisions by one other user not shown)
Line 1: Line 1:
__NOTOC__
+
 
 
= Marconi API: v1 Blueprint =
 
= Marconi API: v1 Blueprint =
  
Line 6: Line 6:
 
''' To Do '''
 
''' To Do '''
  
 +
* Specify all possible error responses for each type of request
 +
* Define and register message media type/ messages media type - http://www.iana.org/assignments/media-types/index.html
 
* Flesh out examples, incomplete sections (obviously)
 
* Flesh out examples, incomplete sections (obviously)
 
* Clean up this document, add a FAQ section
 
* Clean up this document, add a FAQ section
* Decide whether meta-messages should be a mandatory or optional feature (probably optional)
 
* Decide whether message signing should be a mandatory or optional feature (probably optional)
 
* Clarify use cases around tags, perhaps make them an optional feature
 
* Strongly consider defining top-level queues to facilitate sharing queues between tenants
 
** Makes it easier to layer on top of existing message queueing systems / use them as backing stores.
 
** Define the use case / business case for sharing queues between tenants, figure out how that might work with [[KeyStone]] (i.e., authorizing access to particular queues based on auth token).
 
** Implementations may decide to implement these as syntactic sugar on top of tags
 
* Decide whether client state should be based on timestamps or markers (or make one optional)
 
** Markers assume some kind of monotonic message IDs and/or a sequential storage schema.
 
  
 
== Overview ==
 
== Overview ==
  
Marconi provides an HTTP-based API in the spirit of the [https://en.wikipedia.org/wiki/Representational_state_transfer REST architecture style]
+
Marconi provides an HTTP-based API in the spirit of the [https://en.wikipedia.org/wiki/Representational_state_transfer REST architecture style].
 
 
REST is a natural fit for Marconi's design philosophy, shamelessly borrowed from Donald A. Norman's work regarding The Design of Everyday Things:
 
 
 
  The value of a well-designed object is when it has such a rich set of affordances that the people who use it can do things with it that the designer never imagined.  
 
  
This guide assumes the reader is familiar with REST, HTTP/1.1, JSON and XML. URI templates are specified according to [http://tools.ietf.org/html/rfc6570 RFC 6570].
+
This guide assumes the reader is familiar with REST, HTTP/1.1 and JSON. URI templates use the same syntax as [http://tools.ietf.org/html/rfc6570 RFC 6570].
  
 
== Common API Elements ==
 
== Common API Elements ==
Line 51: Line 40:
  
  
The URI version will only be incremented to accomodate major new features or API redesigns that can not be made backwards-compatible. When new API versions are released, older versions are deprecated. Marconi maintainers will work with developers and partners to ensure there is adequate time to migrate to new versions before deprecated ones are discontinued.
+
The URI version will only be incremented to accommodate major new features or API redesigns that can not be made backwards-compatible. When new API versions are released, older versions are deprecated. Marconi maintainers will work with developers and partners to ensure there is adequate time to migrate to new versions before deprecated ones are discontinued.
  
 
Since the base URI is only incremented to accommodate major API revisions, sub-versioning of the API is meaningless and is therefore not used. For example, the next version after "v1" would be "v2", not "v1.1" or some variant thereof. (HATEOS and media types are used in lieu of minor, URI-based versioning.)
 
Since the base URI is only incremented to accommodate major API revisions, sub-versioning of the API is meaningless and is therefore not used. For example, the next version after "v1" would be "v2", not "v1.1" or some variant thereof. (HATEOS and media types are used in lieu of minor, URI-based versioning.)
Line 57: Line 46:
 
''' Resource Media Types '''
 
''' Resource Media Types '''
  
The API supports `application/json` and `application/xml`, encoded as UTF-8. Gzip'd requests are optionally supported by the server.
+
The API supports `application/json` (XML is on the road map), encoded as UTF-8. Gzip'd requests are optionally supported by the server.
 
 
Unrecognised protocol elements should be ignored. This includes XML elements and attributes, JSON object properties, link relation types, media types, etc.
 
  
TODO: Define and register message media type/ messages media type - http://www.iana.org/assignments/media-types/index.html
+
Unrecognized protocol elements should be ignored. This includes XML elements and attributes, JSON object properties, link relation types, media types, etc.
  
 
''' Authentication '''
 
''' Authentication '''
Line 75: Line 62:
 
''' Authorization '''
 
''' Authorization '''
  
TBD: The API needs to verify read/write access to all endpoints according to the provided auth token.
+
The API needs to verify read/write access to all endpoints according to the provided auth token (RBAC). The ACL should be cached with the token.
 
 
''' Request Signing '''
 
 
 
Messages may optionally contain a signature to verify its authenticity.
 
 
 
TODO: How to register and query public keys? Via Keystone? Shouldn't be part of Marconi, in any case.
 
 
 
How to calculate signature?
 
 
 
[[SecP256K1]] or something using RSA-2048? (encrypt a SHA-512 value). Would be nice to allow different schemes... How is Ceilometer doing it?
 
 
 
See also: http://stackoverflow.com/questions/9897023/asymmetric-hmac#
 
  
 
''' Tenant ID '''
 
''' Tenant ID '''
  
Auth tokens are only valid for a particular tenant ID, which should be reflected in the service endpoints retrieved from Keystone. Agents use the given endpoint as a home document href from which to discover links to all other API requests.  
+
Auth tokens are only valid for a particular tenant ID, which should be reflected in the service endpoints retrieved from Keystone. Agents use the given endpoint as a home document href from which to discover links to all other API requests.
  
 
An example endpoint:
 
An example endpoint:
Line 102: Line 77:
  
 
The client chooses one of the presented endpoints and uses it as the base URL for all subsequent requests.
 
The client chooses one of the presented endpoints and uses it as the base URL for all subsequent requests.
 
''' Caching '''
 
 
@todo
 
 
Can be used to save bandwidth at the expense of server CPU and/or added complexity. Provide an ETag for results, the client submits in subsequent request using the If-None-Match header. Server performs query against data store, dynamically generates ETag, compares to client's ETag, and returns 204 if-and-only-if the two ETags don't match. This is especially helpful in cutting down on duplicate traffic due to the inability of the server to guarantee message ordering.
 
  
 
''' Errors '''
 
''' Errors '''
Line 130: Line 99:
 
   "code": 1092,
 
   "code": 1092,
 
   "link": {
 
   "link": {
     "rel": "api-doc",
+
     "rel": "help",
 
     "href": "http://example.com/marconi/docs/messages#limit",
 
     "href": "http://example.com/marconi/docs/messages#limit",
 
     "text": "API documentation for the limit parameter"
 
     "text": "API documentation for the limit parameter"
Line 156: Line 125:
 
|-
 
|-
 
|  Accept  
 
|  Accept  
|  Media type desired; may be either `application/json` or `application/xml`
+
|  Media type desired; initially, only `application/json` will be supported
 
|-
 
|-
 
|  Accept-Encoding  
 
|  Accept-Encoding  
Line 162: Line 131:
 
|-
 
|-
 
|  Content-Length  
 
|  Content-Length  
|  For POST or PUT requests, the length in bytes of the JSON or XML document being submitted  
+
|  For POST or PUT requests, the length in bytes of the message document being submitted  
 
|-
 
|-
 
|  X-Auth-Token  
 
|  X-Auth-Token  
Line 172: Line 141:
  
 
<pre><nowiki>
 
<pre><nowiki>
GET /v1/480924/messages?tags=channel-foo,topic-bar&ts=1355237242783&limit=10 HTTP/1.1
+
GET /v1/480924/queues/fizbat/messages?marker=1355-237242-783&limit=10 HTTP/1.1
 
Host: marconi.example.com
 
Host: marconi.example.com
 
User-Agent: python/2.7 killer-rabbit/1.2 uuid/30387f00-39a0-11e2-be4d-a8d15f34bae2
 
User-Agent: python/2.7 killer-rabbit/1.2 uuid/30387f00-39a0-11e2-be4d-a8d15f34bae2
Line 184: Line 153:
 
== Get Home Document ==
 
== Get Home Document ==
  
'''Request Template'''
+
'''Template'''
  
  
Line 214: Line 183:
 
'''Discussion'''
 
'''Discussion'''
  
The entire Marconi API is discoverable from a single starting point; agents do not need to know any more than this one URI in order to explore the entire API.  
+
The entire Marconi API is discoverable from a single starting point; agents do not need to know any more than this one URI in order to explore the entire API.
  
 
See also: http://tools.ietf.org/html/draft-nottingham-json-home-02
 
See also: http://tools.ietf.org/html/draft-nottingham-json-home-02
  
== Get a Specific Message ==
+
== Create a Queue ==
  
'''Request Template'''
+
'''Template'''
  
  
 
<pre><nowiki>
 
<pre><nowiki>
GET {base_url}/messages/{messageId}
+
PUT {base_url}/queues/{queue_name}
 +
 
 +
{
 +
  "ttl": {ttl}
 +
}
 
</nowiki></pre>
 
</nowiki></pre>
  
Line 232: Line 205:
  
 
<pre><nowiki>
 
<pre><nowiki>
GET /v1/480924/messages/50b68a50d6f5b8c8a7c62b01 HTTP/1.1
+
PUT /v1/480924/queues/fizbat HTTP/1.1
 
Host: marconi.example.com
 
Host: marconi.example.com
  
 
[...]
 
[...]
 +
 +
{
 +
  "ttl": 3600
 +
}
  
 
</nowiki></pre>
 
</nowiki></pre>
Line 244: Line 221:
  
 
<pre><nowiki>
 
<pre><nowiki>
HTTP/1.1 200 OK
+
HTTP/1.1 201 Created
 +
Location: https://marconi.example.com/v1/480924/queues/fizbit
 +
</nowiki></pre>
 +
 
 +
 
 +
'''Discussion'''
 +
 
 +
<code><nowiki>ttl</nowiki></code> is the number of seconds the server will keep a message in the queue before automatically deleting it. This value Should be long enough to give all observers ample time to retrieve the message (and process it in the case of work queuing / claim semantics, discussed later in this specification.)
 +
 
 +
== Reconfigure a Queue ==
 +
 
 +
'''Template'''
 +
 
 +
 
 +
<pre><nowiki>
 +
PATCH {base_url}/queues/{queue_name}
 +
 
 +
{
 +
  "op": "replace",
 +
  "path": "/ttl",
 +
  "value": {ttl}
 +
}
 +
</nowiki></pre>
 +
 
 +
 
 +
'''Request'''
 +
 
 +
 
 +
<pre><nowiki>
 +
PATCH /v1/480924/queues/fizbat HTTP/1.1
 +
Host: marconi.example.com
  
 
[...]
 
[...]
  
 
{
 
{
   "id": 50b68a50d6f5b8c8a7c62b01,
+
   "op": "replace",
  "userAgent": "python/2.7 killer-rabbit/1.2 uuid/79ed56f8-2519-11e2-b835-acf6018e45a1",
+
   "path": "/ttl",
   "age": 790,
+
   "value": 86400
  "tags": [
 
    foo,
 
    snap,
 
    bar,
 
    bang
 
  ],
 
   "signature": {
 
    "scheme": "SecP256K1",
 
    "salt": "b8c850d6f5b8a7c6",
 
    "value": "T2gJfGQNgUe1XMDHyucwH27zn628it0fWrCFgE2mPWR+oMTOiW7jb1OPNuZtLus5O1IzTzy+5ALyLCyUq7JoLQ=="
 
  },
 
  "body": {
 
    "event": "ActivateAccount",
 
    "mode": "active"
 
  }
 
 
}
 
}
 +
 +
 +
</nowiki></pre>
 +
 +
 +
'''Response'''
 +
 +
 +
<pre><nowiki>
 +
HTTP/1.1 204 No Content
 +
</nowiki></pre>
 +
 +
 +
'''Discussion'''
 +
 +
@todo
 +
 +
== Delete a Queue ==
 +
 +
'''Template'''
 +
 +
 +
<pre><nowiki>
 +
DELETE {base_url}/queues/{queue_name}
 +
</nowiki></pre>
 +
 +
 +
'''Request'''
 +
 +
 +
<pre><nowiki>
 +
DELETE /v1/480924/queues/fizbat HTTP/1.1
 +
Host: marconi.example.com
 +
</nowiki></pre>
 +
 +
 +
'''Response'''
 +
 +
 +
<pre><nowiki>
 +
HTTP/1.1 204 No Content
 
</nowiki></pre>
 
</nowiki></pre>
  
Line 273: Line 306:
 
'''Discussion'''
 
'''Discussion'''
  
@todo Define message fields.
+
Use this operation to immediately delete a queue along with all its messages (if any).
  
 
== Get Messages ==
 
== Get Messages ==
  
'''Request Template'''
+
'''Template'''
  
  
 
<pre><nowiki>
 
<pre><nowiki>
GET {base_url}/messages{?tags,ts,limit,sort,meta,echo}
+
GET {base_url}/queues/{queue_name}/messages/{?marker,limit,echo}
 
</nowiki></pre>
 
</nowiki></pre>
  
Line 289: Line 322:
  
 
<pre><nowiki>
 
<pre><nowiki>
GET /v1/480924/messages?tags=foo,bar,bang&ts=1355237242783&limit=10&sort=desc HTTP/1.1
+
GET /v1/480924/queues/fizbit/messages&marker=1355-237242-783&limit=10 HTTP/1.1
 
Host: marconi.example.com
 
Host: marconi.example.com
  
Line 304: Line 337:
 
<pre><nowiki>
 
<pre><nowiki>
 
HTTP/1.1 200 OK
 
HTTP/1.1 200 OK
ETag: "2:fa0d1a60ef6616bb28038515c8ea4cb2"
+
Content-Location: /v1/480924/queues/fizbit/messages?marker=1355-237242-783&limit=10
  
 
[...]
 
[...]
  
 
{
 
{
  "messageCount": 1,
 
 
   "links": [
 
   "links": [
 
     {
 
     {
       "rel": "self"
+
       "rel": "next",
       "href-template": "{base_url}/messages?tags=foo,bar,bang&ts=1355237242783&limit=10&sort=desc"
+
       "href": "/v1/480924/queues/fizbit/messages?marker=6244-244224-783&limit=10"
    },
 
    {
 
      "rel": "next"
 
      "href-template": "{base_url}/messages?tags=foo,bar,bang&ts=1355237244190&limit=10&sort=desc"
 
 
     }
 
     }
 
   ],
 
   ],
 
   "messages": [
 
   "messages": [
 
     {
 
     {
       "id": 50b68a50d6f5b8c8a7c62b01,
+
       "id": "50b68a50d6f5b8c8a7c62b01",
       "href-template": "{base_url}/messages/{id}"
+
       "href": "/v1/480924/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b01"
 
       "userAgent": "python/2.7 killer-rabbit/1.2 uuid/79ed56f8-2519-11e2-b835-acf6018e45a1",
 
       "userAgent": "python/2.7 killer-rabbit/1.2 uuid/79ed56f8-2519-11e2-b835-acf6018e45a1",
       "ts": 1355237244190
+
       "ts": "2012-12-04 16:53:20Z",
 
       "age": 790,
 
       "age": 790,
      "tags": [
 
        foo,
 
        snap,
 
        bar,
 
        bang
 
      ],
 
      "signature": {
 
        "scheme": "SecP256K1",
 
        "salt": "b8c850d6f5b8a7c6",
 
        "value": "T2gJfGQNgUe1XMDHyucwH27zn628it0fWrCFgE2mPWR+oMTOiW7jb1OPNuZtLus5O1IzTzy+5ALyLCyUq7JoLQ=="
 
      },
 
 
       "body": {
 
       "body": {
 
         "event": "ActivateAccount",
 
         "event": "ActivateAccount",
Line 348: Line 365:
  
  
Example meta-messages response (if a client were to add meta=true to the sample query).
+
'''Discussion'''
 +
 
 +
Message IDs and markers are opaque strings; clients should make no assumptions on their format or length. Furthermore, clients should assume there is no relationships between markers and message IDs (i.e., one cannot be derived from the other).
 +
 
 +
Results are ordered by age, oldest message first.
 +
 
 +
<code><nowiki>limit</nowiki></code> specifies up to x messages to return. Note that x is configurable, but must be at least 10. If not specified, limit defaults to x. When more messages are available than can be returned in a single request, the client can pick up the next batch by simply using the URI template parameters returned from the previous call in the "next" field (TBD).
 +
 
 +
<code><nowiki>marker</nowiki></code> is an opaque string that the client can use to request the next batch of messages. It communicates to the server which messages the client has already received.
 +
 
 +
Note: If <code><nowiki>marker</nowiki></code> is not specified, the API will return all messages (up to limit).
 +
 
 +
<code><nowiki>echo</nowiki></code> is a boolean (i.e., "true" or "false") that determines whether or not the API will return a client's own messages, as determined by the <code><nowiki>uuid</nowiki></code> portion of the User-Agent header. If not specified, echo defaults to "false".
 +
 
 +
== Get Action Messages ==
 +
 
 +
'''Template'''
 +
 
 +
 
 +
<pre><nowiki>
 +
GET {base_url}/queues/{queue_name}/actions/{?marker,limit}
 +
</nowiki></pre>
 +
 
 +
 
 +
'''Request'''
 +
 
 +
 
 +
<pre><nowiki>
 +
GET /v1/480924/queues/fizbit/actions&marker=1355-237242-783&limit=10 HTTP/1.1
 +
Host: marconi.example.com
 +
 
 +
[...]
 +
 
 +
</nowiki></pre>
 +
 
 +
 
 +
'''Response'''
 +
 
 +
HTTP 200 if the query matched 1 or more actions, HTTP 204 otherwise (with no body).
 +
 
  
 
<pre><nowiki>
 
<pre><nowiki>
 
HTTP/1.1 200 OK
 
HTTP/1.1 200 OK
ETag: "2:fa0d1a60ef6616bb28038515c8ea4cb2"
+
Content-Location: /v1/480924/queues/fizbit/actions?marker=1355-237242-783&limit=10
  
 
[...]
 
[...]
  
 
{
 
{
  "messageCount": 2,
 
 
   "links": [
 
   "links": [
 
     {
 
     {
       "rel": "self"
+
       "rel": "next",
       "href-template": "{base_url}/messages?tags=foo,bar,bang&ts=1355237242783&limit=10&sort=desc&meta=true"
+
       "href": "/v1/480924/queues/fizbit/actions?marker=6244-244224-783&limit=10"
    },
 
    {
 
      "rel": "next"
 
      "href-template": "{base_url}/messages?tags=foo,bar,bang&ts=1355237244210&limit=10&sort=desc&meta=true"
 
 
     }
 
     }
 
   ],
 
   ],
 
   "messages": [
 
   "messages": [
 
     {
 
     {
       "id": 10b00a50d6f5b8c8a7c62ccc,
+
       "id": 10b00a50d6f5b8c8a7c62ccb,
       "href-template": "{base_url}/messages/{id}"
+
       "href": "/v1/480924/queues/fizbit/actions/{id}"
 
       "userAgent": "marconi/1 uuid/f2e4b36a-3f05-11e2-b71d-7823d2b0f3ce",
 
       "userAgent": "marconi/1 uuid/f2e4b36a-3f05-11e2-b71d-7823d2b0f3ce",
       "ts": 1355237242783,
+
       "ts": "2012-12-04 16:53:18Z",
      "age": 790,
+
       "age": 792,
       "tags": [
 
        foo,
 
        snap,
 
        bar,
 
        bang
 
      ],
 
 
       "body": {
 
       "body": {
         "event": "MessageCreated",
+
         "event": "LockExpired",
        "timestamp": "2012-12-04 16:53:20Z",
 
 
         "details": {
 
         "details": {
           "messageId": "50b68a50d6f5b8c8a7c62b01",
+
           "messageId": "50b68a50d6f5b8c8a7c62a87",
 
           "userAgent": "python/2.7 killer-rabbit/1.2 uuid/79ed56f8-2519-11e2-b835-acf6018e45a1",
 
           "userAgent": "python/2.7 killer-rabbit/1.2 uuid/79ed56f8-2519-11e2-b835-acf6018e45a1",
 
         }
 
         }
Line 391: Line 435:
 
     },
 
     },
 
     {
 
     {
       "id": 10b00a50d6f5b8c8a7c62ccb,
+
       "id": 10b00a50d6f5b8c8a7c62ccc,
       "href-template": "{base_url}/messages/{id}"
+
       "href": "/v1/480924/queues/fizbit/actions/{id}"
 
       "userAgent": "marconi/1 uuid/f2e4b36a-3f05-11e2-b71d-7823d2b0f3ce",
 
       "userAgent": "marconi/1 uuid/f2e4b36a-3f05-11e2-b71d-7823d2b0f3ce",
       "ts": 1355237244210,
+
       "ts": "2012-12-04 16:53:20Z",
      "age": 792,
+
       "age": 790,
       "tags": [
 
        foo,
 
        snap,
 
        bar,
 
        bang
 
      ],
 
 
       "body": {
 
       "body": {
         "event": "LockExpired",
+
         "event": "MessageCreated",
        "timestamp": "2012-12-04 16:53:18Z",
 
 
         "details": {
 
         "details": {
           "messageId": "50b68a50d6f5b8c8a7c62a87",
+
           "messageId": "50b68a50d6f5b8c8a7c62b01",
 
           "userAgent": "python/2.7 killer-rabbit/1.2 uuid/79ed56f8-2519-11e2-b835-acf6018e45a1",
 
           "userAgent": "python/2.7 killer-rabbit/1.2 uuid/79ed56f8-2519-11e2-b835-acf6018e45a1",
 
         }
 
         }
Line 418: Line 455:
 
'''Discussion'''
 
'''Discussion'''
  
Message IDs are opaque strings; clients should make no assumptions on their format or length, except that IDs may not exceed 50 characters in length.
+
The interactions of various agents/workers with a cloud queuing services can be difficult to audit and debug. Marconi emits action messages to a special queue, from which auditors can retrieve a list of recent actions involving a specific queue's messages. These actions can then be archived by the auditor for future analysis and diagnostics (action messages time out and are deleted according to the TTL configured for the associated queue).
 +
 
 +
Message IDs and markers are opaque strings; clients should make no assumptions on their format or length. Furthermore, clients should assume there is no relationships between markers and message IDs (i.e., one cannot be derived from the other).
 +
 
 +
Results are ordered by age, oldest message first.
 +
 
 +
<code><nowiki>limit</nowiki></code> specifies up to x messages to return. Note that x is configurable, but must be at least 10. If not specified, limit defaults to x. When more messages are available than can be returned in a single request, the client can pick up the next batch by simply using the URI template parameters returned from the previous call in the "next" field.
  
<code><nowiki>tags</nowiki></code> is a list of up to n message tags, where n >= 2. The maximum number of tags supported is configurable. The API will return only those messages containing ALL of the specified tags. If no tags are specified, all messages are returned.
+
<code><nowiki>marker</nowiki></code> is an opaque string that the client can use to request the next batch of messages. It communicates to the server which messages the client has already received.
  
<code><nowiki>limit</nowiki></code> specifies up to x messages to return. Note that x is configurable, but must be at least 10. If not specified, limit defaults to x. When more messages are available than can be returned in a single request, the client can pick up the next batchby simply using the URI template parameters returned from the previous call in the "next" field (TBD).
+
Note: If <code><nowiki>marker</nowiki></code> is not specified, the API will return all messages (up to limit).
 +
 
 +
== Get a Specific Message ==
  
<code><nowiki>ts</nowiki></code> is the timestamp (64-bit UNIX timestamp in milliseconds) from the last message the client saw (relative to the server - use the "ts" message field). The API will return messages that were enqueued after the specified time, minus ''t'' milliseconds, where t is an implementation-defined number of milliseconds within which the server cannot guarantee message ordering (admittedly, a leaky abstraction). The client must cache messages for t milliseconds in order to check for duplicates returned in subsequent requests. Note that the message having the exact given timestamp (ts) will probably be part of the result set unless that particular message has expired or an ETag submitted by the client allows the server to definitively determine that no new messages match the given query string since the last request.
+
'''Template'''
  
Note: If <code><nowiki>ts</nowiki></code> is not specified, the API will return all messages.
 
  
<code><nowiki>sort</nowiki></code> specifies how to chronologically order results. Use "asc" or "desc" for ascending or descending, respectively. If sort is not given, the API will return results in ascending order (oldest message first).
+
<pre><nowiki>
 +
GET {base_url}/queues/{queue_name}/messages/{message_id}
 +
</nowiki></pre>
  
<code><nowiki>meta</nowiki></code> is a boolean value (i.e., "true" or "false") that determines whether the API will return actual messages or meta-messages. Meta-messages are "messages about messages", and are automatically generated by the server. Clients may query for meta-messages to audit business processes or to diagnose data flow issues. The default value for <code><nowiki>meta</nowiki></code> is "false".
 
  
<code><nowiki>echo</nowiki></code> is a boolean (i.e., "true" or "false") that determines whether or not the API will return a client's own messages, as determined by the uuid portion of the User-Agent header. If not specified, echo defaults to "false".  
+
'''Request'''
 +
 
 +
 
 +
<pre><nowiki>
 +
GET /v1/480924/queues/fizbat/messages/50b68a50d6f5b8c8a7c62b01 HTTP/1.1
 +
Host: marconi.example.com
 +
 
 +
[...]
 +
 
 +
</nowiki></pre>
 +
 
 +
 
 +
'''Response'''
 +
 
 +
 
 +
<pre><nowiki>
 +
HTTP/1.1 200 OK
 +
Content-Location: /v1/480924/queues/fizbat/messages/50b68a50d6f5b8c8a7c62b01
 +
 
 +
[...]
 +
 
 +
{
 +
  "id": "50b68a50d6f5b8c8a7c62b01",
 +
  "userAgent": "python/2.7 killer-rabbit/1.2 uuid/79ed56f8-2519-11e2-b835-acf6018e45a1",
 +
  "age": 790,
 +
  "body": {
 +
    "event": "ActivateAccount",
 +
    "mode": "active"
 +
  }
 +
}
 +
</nowiki></pre>
 +
 
 +
 
 +
'''Discussion'''
 +
 
 +
@todo Define message fields.
  
 
== Post Messages ==
 
== Post Messages ==
  
'''Request Template'''
+
'''Template'''
  
  
 
<pre><nowiki>
 
<pre><nowiki>
POST {base_url}/messages
+
POST {base_url}/queues/{queue_name}/messages
 
</nowiki></pre>
 
</nowiki></pre>
  
Line 448: Line 528:
  
 
<pre><nowiki>
 
<pre><nowiki>
POST /v1/480924/messages HTTP/1.1
+
POST /v1/480924/queues/fizbit/messages HTTP/1.1
 
Host: marconi.example.com
 
Host: marconi.example.com
  
Line 456: Line 536:
 
[
 
[
 
   {
 
   {
    "ttl": 10,     
 
    "durability": 3,
 
    "tags": [420D29D6-3F24-11E2-BC14-7823D2B0F3CE, checkpoint]
 
    "signature": {
 
      "scheme": "SecP256K1",
 
      "salt": "b8c850d6f5b8a7c6",
 
      "value": "T2gJfGQNgUe1XMDHyucwH27zn628it0fWrCFgE2mPWR+oMTOiW7jb1OPNuZtLus5O1IzTzy+5ALyLCyUq7JoLQ=="
 
    },
 
 
     "body": {
 
     "body": {
 
       "event": "BackupStarted",
 
       "event": "BackupStarted",
Line 470: Line 542:
 
   },
 
   },
 
   {
 
   {
    "tags": [420D29D6-3F24-11E2-BC14-7823D2B0F3CE, progress]
 
 
     "body": {
 
     "body": {
 
       "event": "BackupProgress",
 
       "event": "BackupProgress",
Line 488: Line 559:
 
<pre><nowiki>
 
<pre><nowiki>
 
HTTP/1.1 201 Created
 
HTTP/1.1 201 Created
Location: https://marconi.example.com/v1/480924/messages/50b68a50d6f5b8c8a7c62b01
+
Location: https://marconi.example.com/v1/480924/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b01
 
</nowiki></pre>
 
</nowiki></pre>
  
  
When multiple messages are submitted:
+
When multiple messages are submitted, the generic messages resource is returned for the location, since Marconi does not support a GET for a list of messages, e.g.:
  
  
 
<pre><nowiki>
 
<pre><nowiki>
 
HTTP/1.1 201 Created
 
HTTP/1.1 201 Created
Location: https://marconi.example.com/v1/480924/messages&ts=1355237242783
+
Location: https://marconi.example.com/v1/480924/queues/fizbit/messages
 
</nowiki></pre>
 
</nowiki></pre>
  
Line 504: Line 575:
  
 
One or more messages may be submitted in a single request, but must always be encapsulated in a collection container (an array in JSON, a parent tag in XML). In the case of a batch POST, querying the returned <code><nowiki>Location</nowiki></code> may return messages posted concurrently by other agents.
 
One or more messages may be submitted in a single request, but must always be encapsulated in a collection container (an array in JSON, a parent tag in XML). In the case of a batch POST, querying the returned <code><nowiki>Location</nowiki></code> may return messages posted concurrently by other agents.
 
<code><nowiki>ttl</nowiki></code> is the number of seconds the server will keep a message before automatically deleting it. Should be long enough to give all observers ample time to retrieve the message. If not specified, defaults to the maximum default set for any of the message's associated tags, or the default set for the tenant (TBD), or the one configured in the deployment (?).
 
 
<code><nowiki>tags</nowiki></code> is a list of up to n tags to associate with a given message, where n >= 2. The maximum number of tags supported is configurable (the default is 5). The maximum length of each tag is likewise configurable (with a default of 150 characters). Tags are case-sensitive.
 
 
<code><nowiki>signature</nowiki></code> - TBD, probably something like sign(hash(salt + payload))
 
  
 
<code><nowiki>body</nowiki></code> specifies a custom document which constitutes the body of the message being sent. The size of this body, in characters and including whitespace, is configurable (the default is 64 KiB).
 
<code><nowiki>body</nowiki></code> specifies a custom document which constitutes the body of the message being sent. The size of this body, in characters and including whitespace, is configurable (the default is 64 KiB).
 
<code><nowiki>durability</nowiki></code> requests a certain durability guarantee from the server. The purpose of this parameter is to allow clients to dynamically make tradeoffs between durability and cost/performance depending on the type of message being sent. Note that the maximum durability level supported by the server is configurable; not all deployments will support all levels. If a level is unsupported, the server will return 400 Bad Request. If a level is supported but the server is unable to complete the request, and appropriate 5xx error will be returned to the client.
 
 
The following levels are defined:
 
 
{| border="1" cellpadding="2" cellspacing="0"
 
|  Level
 
|-
 
|  0
 
|-
 
|  1
 
|-
 
|  2
 
|  Guaranteed to be replicated across two or more storage servers.
 
|-
 
|  3
 
|  Guaranteed to be replicated across two or more zones within the same data center.
 
|-
 
|  4
 
|}
 
 
Note that higher durability levels assume the guarantees (if any) of all lower levels.
 
  
 
== Delete a Single Message ==
 
== Delete a Single Message ==
  
'''Request Template'''
+
'''Template'''
  
  
 
<pre><nowiki>
 
<pre><nowiki>
DELETE {base_url}/messages/{message_id}{?lock_id}
+
DELETE {base_url}/queues/{queue_name}/messages/{message_id}{?claim_id}
 
</nowiki></pre>
 
</nowiki></pre>
  
Line 549: Line 592:
  
 
<pre><nowiki>
 
<pre><nowiki>
DELETE /v1/480924/messages/50b68a50d6f5b8c8a7c62b01 HTTP/1.1
+
DELETE /v1/480924/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b01 HTTP/1.1
 
Host: marconi.example.com
 
Host: marconi.example.com
  
Line 569: Line 612:
 
<code><nowiki>message_id</nowiki></code> specifies the message to delete.
 
<code><nowiki>message_id</nowiki></code> specifies the message to delete.
  
<code><nowiki>lock_id</nowiki></code> specifies that the message should only be deleted if it has the specified transactional lock, and that lock has not expired. This is useful for ensuring only one agent processes any given message; whenever a worker client's lock expires before it has a chance to delete a message it has processed, the worker must roll back any actions it took based on that message, since another worker will likely process the same message as part of a different transaction.  
+
<code><nowiki>claim_id</nowiki></code> specifies that the message should only be deleted if it has the specified claim ID, and that claim has not expired. This is useful for ensuring only one agent processes any given message; whenever a worker client's claim expires before it has a chance to delete a message it has processed, the worker must roll back any actions it took based on that message, since another worker will now be able to claim and process the same message.
  
== Delete Several Messages ==
+
== Claim Messages ==
  
'''Request Template'''
+
'''Template'''
  
  
 
<pre><nowiki>
 
<pre><nowiki>
DELETE {base_url}/messages{?tags,all,lock_id}
+
PATCH {base_url}/queues/{queue_name}/messages{?limit}
 +
Content-Type: application/json-patch
 
</nowiki></pre>
 
</nowiki></pre>
  
Line 585: Line 629:
  
 
<pre><nowiki>
 
<pre><nowiki>
DELETE /v1/480924/messages?tags=420D29D6-3F24-11E2-BC14-7823D2B0F3CE HTTP/1.1
+
PATCH /v1/480924/queues/fizbit/messages/limit=5 HTTP/1.1
Host: marconi.example.com
+
Content-Type: application/json-patch
 +
Accept: application/json
  
 
[...]
 
[...]
  
 +
[
 +
  {
 +
    "op": "add",
 +
    "path": "/claim",
 +
    "value": {
 +
      "ttl": 30
 +
    }
 +
  }
 +
]
 
</nowiki></pre>
 
</nowiki></pre>
  
  
 
'''Response'''
 
'''Response'''
 +
 +
The client receives a list of claimed messages, if any:
  
  
 
<pre><nowiki>
 
<pre><nowiki>
 
HTTP/1.1 200 OK
 
HTTP/1.1 200 OK
 +
Content-Type: application/json
  
 
[...]
 
[...]
  
 
{
 
{
   "messageCount": 43
+
   "messages": [
}
+
    ...
</nowiki></pre>
 
  
 +
    {
 +
      ...
  
'''Discussion'''
+
      "claim": {
 +
        "id": "a28ee94e-6cb4-11e2-b4d5-7703267a7926",
 +
        "ttl": 30,
 +
        "age": 0
 +
      }
  
<code><nowiki>tags</nowiki></code> specifies that only messages having the given tags will be deleted. To avoid accidentally deleting all messages for a given tenant, if <code><nowiki>tags</nowiki></code> is not specified, <code><nowiki>all</nowiki></code> must equal "true"; otherwise, no messages will be deleted and the server will return 400 Bad Request. Tags are case-sensitive.
+
      ...
 
+
     }
<code><nowiki>all</nowiki></code> is a boolean value (i.e., "true" or "false") that determines whether all messages will be deleted in the case that no tags are specified. If tags ''are'' specified, however, the server simply ignores <code><nowiki>all</nowiki></code>.
 
 
 
<code><nowiki>lock_id</nowiki></code> specifies that messages should only be deleted if they have the specified transactional lock, and that lock has not expired.
 
 
 
The server returns the number of messages deleted.
 
 
 
== Lock Messages ==
 
 
 
'''Request Template'''
 
 
 
 
 
<pre><nowiki>
 
POST {base_url}/messages/locks{?tags,limit}
 
</nowiki></pre>
 
 
 
 
 
'''Request'''
 
 
 
 
 
<pre><nowiki>
 
POST /v1/480924/messages/locks?tags=686897696a7c876b7e&limit=5
 
 
 
[...]
 
 
 
{
 
  "ttl": 30 
 
}
 
</nowiki></pre>
 
 
 
 
 
'''Response'''
 
 
 
The client receives a lock ID and a list of locked messages, if any:
 
 
 
 
 
<pre><nowiki>
 
@todo
 
 
 
{
 
  "lock": {
 
    "id": "9206ebcc0939d18c351e994555de97b9",
 
    "ttl": 30,
 
     "age": 0
 
  },
 
  "messages": [
 
    ...
 
 
   ]
 
   ]
 
}
 
}
Line 664: Line 680:
 
'''Discussion'''
 
'''Discussion'''
  
Locks a set of messages, up to limit, from oldest to newest, skipping any that are already locked.
+
Claims a set of messages, up to limit, from oldest to newest, skipping any that are already claimed. Uses JSON patch ([http://tools.ietf.org/html/draft-ietf-appsawg-json-patch-10 RFC Draft]).
  
Lock TTL should be <= message TTLs. If it is not, the messages will expire and be garbage collected before the lock expires, which is not what clients will expect. (Should this constraint be enforced server-side?)
+
The client should delete the message when it has finished processing it, before the claim expires, to ensure the message is processed only one time. As part of the delete operation, all worker clients should specify the claim ID. That way, the server can return an error if the claim just expired (notifying it of the race condition), giving the worker a chance to roll back its own processing of the given message, since another worker will likely claim the message and process it.
  
<code><nowiki>limit</nowiki></code> specifies up to x messages to lock. Note that x is configurable, but must be at least 10. If not specified, limit defaults to x.  
+
Just as with a message's age, the age given for each claim is relative to the server's clock, and is useful for determining how quickly messages are getting processed, and whether a given message's claim is about to expire.
  
== Check the Status of a Lock ==
+
When a claim expires, it is removed, allowing another client worker to claim the message in the case that the original worker fails to process it.
  
'''Request Template'''
+
<code><nowiki>limit</nowiki></code> specifies up to x messages to claim. Note that x is configurable, but must be at least 10. If not specified, limit defaults to x.
  
 +
== Renew a Claim ==
  
<pre><nowiki>
+
'''Template'''
GET {base_url}/messages/locks/{lock_id}
 
</nowiki></pre>
 
 
 
  
  
 
<pre><nowiki>
 
<pre><nowiki>
GET /v1/480924/messages/locks/9206ebcc0939d18c351e994555de97b9
+
PATCH {base_url}/queues/{queue_name}/messages/{message_id}
 
+
Content-Type: application/json-patch
[...]
 
 
 
</nowiki></pre>
 
 
 
 
 
'''Response'''
 
 
 
 
 
<pre><nowiki>
 
{
 
  "id": "9206ebcc0939d18c351e994555de97b9",
 
  "ttl": 30,
 
  "age": 17
 
}
 
</nowiki></pre>
 
 
 
 
 
'''Discussion'''
 
 
 
This query may be used to check the age of a given lock (relative to the server's clock).
 
 
 
== Renew a Lock ==
 
 
 
'''Request Template'''
 
 
 
 
 
<pre><nowiki>
 
PATCH {base_url}/messages/locks/9206ebcc0939d18c351e994555de97b9
 
 
</nowiki></pre>
 
</nowiki></pre>
  
Line 719: Line 705:
  
 
<pre><nowiki>
 
<pre><nowiki>
PATCH /v1/480924/messages/locks/9206ebcc0939d18c351e994555de97b9
+
PATCH /v1/480924/queues/fizbit/messages/10b00a50d6f5b8c8a7c62ccb HTTP/1.1
 +
Content-Type: application/json-patch
  
 
[...]
 
[...]
 
+
 
 
[
 
[
 
   {
 
   {
     "replace": "/age",
+
     "op": "test", "path": "/claim/id", "value": "a28ee94e-6cb4-11e2-b4d5-7703267a7926"
    "value": 0
+
  },
 +
  {
 +
    "op": "replace", "path": "/claim/age", "value": 0
 
   }
 
   }
 
]
 
]
Line 742: Line 731:
 
'''Discussion'''
 
'''Discussion'''
  
Clients should periodically renew locks during long-running batches of work to avoid loosing a lock in the middle of processing a message. The server only accepts "0" for the age value.
+
Clients should periodically renew claims during long-running batches of work to avoid loosing a claim in the middle of processing a message. The server only accepts "0" for the age value, and the test operation is required to ensure only the client that owns a given claim can renew it.
  
 
== Count Messages ==
 
== Count Messages ==
  
'''Request Template'''
+
'''Template'''
  
  
 
<pre><nowiki>
 
<pre><nowiki>
HEAD /{version}/{tenant}/messages{?tags,ts,meta,echo}
+
GET {base_url}/queues/{queue_name}/stats
 
</nowiki></pre>
 
</nowiki></pre>
  
Line 758: Line 747:
  
 
<pre><nowiki>
 
<pre><nowiki>
HEAD /v1/480924/messages?tags=foo,bar,bang&ts=1355237242783 HTTP/1.1
+
GET /v1/480924/queues/fizbit/stats HTTP/1.1
 
Host: marconi.example.com
 
Host: marconi.example.com
  
Line 772: Line 761:
 
<pre><nowiki>
 
<pre><nowiki>
 
{
 
{
   "messageCount": 46
+
   "messages": 46,
 +
  "actions": 122
 
}
 
}
 
</nowiki></pre>
 
</nowiki></pre>
Line 779: Line 769:
 
'''Discussion'''
 
'''Discussion'''
  
See ''Get Messages'' for definitions of the query parameters.
+
Returns queue statistics, including how many messages are in the queue, and how many actions have been recorded (actions can be retrieved by performing a GET on {queue_name}/actions).
  
 
== Check Health ==
 
== Check Health ==
  
'''Request Template'''
+
'''Template'''
  
  
Line 803: Line 793:
  
 
<pre><nowiki>
 
<pre><nowiki>
GET /v1/480924/health
+
GET /v1/480924/health HTTP/1.1
 
Host: example.marconi.com
 
Host: example.marconi.com
  
Line 822: Line 812:
 
   "code": "green",
 
   "code": "green",
 
   "link": {
 
   "link": {
     "rel": "status",
+
     "text": "Service status page"
 
     "href": "http://marconi.example.com/status",
 
     "href": "http://marconi.example.com/status",
     "text": "Service status page"
+
     "rel": "help",
 
   }
 
   }
 
}
 
}
Line 842: Line 832:
 
|-
 
|-
 
|  red  
 
|  red  
|  Degraded availability
 
 
|}
 
|}
  
Line 855: Line 844:
 
|}
 
|}
  
== Poll Stats ==
+
== TBD ==
  
@todo - also how authenticate?
+
'''Template'''
 
 
== Set Message Defaults ==
 
 
 
'''Request Template'''
 
  
  
 
<pre><nowiki>
 
<pre><nowiki>
PUT {base_url}/config
+
GET /{version}/{tenant}
 
 
{
 
  "message": {
 
    "ttl": {ttl},
 
    "durability": {durability},
 
    "lock": {
 
      "ttl": {lock_ttl}
 
    }
 
  }
 
}
 
 
</nowiki></pre>
 
</nowiki></pre>
  
  
 
'''Request'''
 
'''Request'''
 
<pre><nowiki>
 
PUT /v1/480924/config HTTP/1.1
 
Host: marconi.example.com
 
 
[...]
 
 
{
 
  "message": {
 
    "ttl": 120,
 
    "durability": 1,
 
    "lock": {
 
      "ttl": 60
 
    }
 
  }
 
}
 
</nowiki></pre>
 
 
 
'''Response'''
 
  
  
 
<pre><nowiki>
 
<pre><nowiki>
HTTP/1.1 204 No Content
+
GET /{version}/{tenant} HTTP/1.1
</nowiki></pre>
 
 
 
 
 
'''Discussion'''
 
 
 
Set per-tenant default message options, such as ttl and durability. Lock TTL must be <= message TTL.
 
 
 
== Get Message Defaults ==
 
 
 
'''Request Template'''
 
 
 
 
 
<pre><nowiki>
 
GET {base_url}/config
 
</nowiki></pre>
 
 
 
 
 
'''Request'''
 
 
 
<pre><nowiki>
 
GET /v1/480924/config HTTP/1.1
 
Host: marconi.example.com
 
 
 
[...]
 
 
 
</nowiki></pre>
 
 
 
 
 
'''Response'''
 
 
 
 
 
<pre><nowiki>
 
HTTP/1.1 200 OK
 
 
 
[...]
 
 
 
{
 
  "message": {
 
    "ttl": 120,
 
    "durability": 1,
 
    "lock": {
 
      "ttl": 60
 
    }
 
  }
 
}
 
</nowiki></pre>
 
 
 
 
 
'''Discussion'''
 
 
 
Get per-tenant default message options, such as ttl and durability. If not set explicitly by a client, a default configuration is used, and can vary between API deployments.
 
 
 
== TBD ==
 
 
 
'''Request Template'''
 
 
 
 
 
<pre><nowiki>
 
GET /{version}/{tenant}
 
 
</nowiki></pre>
 
</nowiki></pre>
  

Revision as of 23:29, 17 February 2013

Marconi API: v1 Blueprint

See also: marconi/specs/grizzly

To Do

  • Specify all possible error responses for each type of request
  • Define and register message media type/ messages media type - http://www.iana.org/assignments/media-types/index.html
  • Flesh out examples, incomplete sections (obviously)
  • Clean up this document, add a FAQ section

Overview

Marconi provides an HTTP-based API in the spirit of the REST architecture style.

This guide assumes the reader is familiar with REST, HTTP/1.1 and JSON. URI templates use the same syntax as RFC 6570.

Common API Elements

HTTPS

All requests to authenticate and operate against the API in production environments use SSL/TLS over HTTP (HTTPS) on TCP port 443. Web servers should only accept high-quality cipher suites, such as AES256_SHA and ECDHE_RSA_AES256_SHA. If hardware acceleration (e.g., AES_NI) is not available, RC4_SHA and ECDHE_RSA_RC4_SHA may be used with some discretion.

Clients

  • Clients should follow HTTP redirects
  • Clients should advertise gzip support
  • Clients must identify themselves in the UserAgent request header; e.g., "User-Agent: python/2.7 cloudthing/1.2"
  • Clients must not hard-code URI paths or templates since they may change over time. Instead, clients should cache links and link templates provided by the API.

API Versioning

The Marconi API uses a URI versioning scheme. The first element of the path contains the target version identifier, e.g.:


  https://marconi.example.com/v1


The URI version will only be incremented to accommodate major new features or API redesigns that can not be made backwards-compatible. When new API versions are released, older versions are deprecated. Marconi maintainers will work with developers and partners to ensure there is adequate time to migrate to new versions before deprecated ones are discontinued.

Since the base URI is only incremented to accommodate major API revisions, sub-versioning of the API is meaningless and is therefore not used. For example, the next version after "v1" would be "v2", not "v1.1" or some variant thereof. (HATEOS and media types are used in lieu of minor, URI-based versioning.)

Resource Media Types

The API supports `application/json` (XML is on the road map), encoded as UTF-8. Gzip'd requests are optionally supported by the server.

Unrecognized protocol elements should be ignored. This includes XML elements and attributes, JSON object properties, link relation types, media types, etc.

Authentication

All requests to the API may only be performed by an authenticated agent.

To authenticate, an agent issues an authentication request to a Keystone Identity Service endpoint.

In response to valid credentials, Keystone responds with an auth token and a service catalog that contains a list of all services and endpoints available for the given token. Multiple endpoints may be returned for Marconi according to physical locations and performance/availability characteristics of different deployments.

Marconi clients must specify a valid auth token in the `X-Auth-Token` header for each request to the Marconi API. The API validates auth tokens against Keystone before servicing each request.

Authorization

The API needs to verify read/write access to all endpoints according to the provided auth token (RBAC). The ACL should be cached with the token.

Tenant ID

Auth tokens are only valid for a particular tenant ID, which should be reflected in the service endpoints retrieved from Keystone. Agents use the given endpoint as a home document href from which to discover links to all other API requests.

An example endpoint:


https://marconi.example.com/v1/480924


The client chooses one of the presented endpoints and uses it as the base URL for all subsequent requests.

Errors

If any request results in an error, the server will return an appropriate 4xx or 5xx HTTP status code, as well as the following information in the body:

  • Title
  • Description
  • Internal code
  • Link to more information

Example:


HTTP/1.1 400 Bad Request
Content-Type: application/json

{
  "title": "Unsupported limit",
  "description": "The given limit cannot be negative, and cannot be greater than 50.",
  "code": 1092,
  "link": {
    "rel": "help",
    "href": "http://example.com/marconi/docs/messages#limit",
    "text": "API documentation for the limit parameter"
  }
}


Common Headers

Each request to the API must include certain standard and extended HTTP headers. These headers provide host, agent, authentication and other pertinent information to the server.

Header Description
Host The host name of the API, as referenced by the Keystone service catalog
User-Agent The name and version of the Marconi client, as well as a UUID for that client. Marconi uses the UUID to distinguish publishers from subscribers, i.e., to avoid echoing an agent's own messages back to it.
Date The current date and time, using the standard RFC 1123 HTTP date format
Accept Media type desired; initially, only `application/json` will be supported
Accept-Encoding Specifies that the agent accepts gzip-encoded response bodies
Content-Length For POST or PUT requests, the length in bytes of the message document being submitted
X-Auth-Token Keystone auth token

Sample API Request

GET /v1/480924/queues/fizbat/messages?marker=1355-237242-783&limit=10 HTTP/1.1
Host: marconi.example.com
User-Agent: python/2.7 killer-rabbit/1.2 uuid/30387f00-39a0-11e2-be4d-a8d15f34bae2
Date: Wed, 28 Nov 2012 21:14:19 GMT
Accept: application/json
Accept-Encoding: gzip
X-Auth-Token: 7d2f63fd-4dcc-4752-8e9b-1d08f989cc00


Get Home Document

Template


GET {base_url}


Request


GET /v1/480924 HTTP/1.1
Host: marconi.example.com

[...]


Response


@todo


Discussion

The entire Marconi API is discoverable from a single starting point; agents do not need to know any more than this one URI in order to explore the entire API.

See also: http://tools.ietf.org/html/draft-nottingham-json-home-02

Create a Queue

Template


PUT {base_url}/queues/{queue_name}

{
  "ttl": {ttl}
}


Request


PUT /v1/480924/queues/fizbat HTTP/1.1
Host: marconi.example.com

[...]

{
  "ttl": 3600
}


Response


HTTP/1.1 201 Created
Location: https://marconi.example.com/v1/480924/queues/fizbit


Discussion

ttl is the number of seconds the server will keep a message in the queue before automatically deleting it. This value Should be long enough to give all observers ample time to retrieve the message (and process it in the case of work queuing / claim semantics, discussed later in this specification.)

Reconfigure a Queue

Template


PATCH {base_url}/queues/{queue_name}

{
  "op": "replace",
  "path": "/ttl",
  "value": {ttl}
}


Request


PATCH /v1/480924/queues/fizbat HTTP/1.1
Host: marconi.example.com

[...]

{
  "op": "replace",
  "path": "/ttl",
  "value": 86400
}



Response


HTTP/1.1 204 No Content


Discussion

@todo

Delete a Queue

Template


DELETE {base_url}/queues/{queue_name}


Request


DELETE /v1/480924/queues/fizbat HTTP/1.1
Host: marconi.example.com


Response


HTTP/1.1 204 No Content


Discussion

Use this operation to immediately delete a queue along with all its messages (if any).

Get Messages

Template


GET {base_url}/queues/{queue_name}/messages/{?marker,limit,echo}


Request


GET /v1/480924/queues/fizbit/messages&marker=1355-237242-783&limit=10 HTTP/1.1
Host: marconi.example.com

[...]


Response

HTTP 200 if the query matched 1 or more messages, HTTP 204 otherwise (with no body).


HTTP/1.1 200 OK
Content-Location: /v1/480924/queues/fizbit/messages?marker=1355-237242-783&limit=10

[...]

{
  "links": [
    {
      "rel": "next",
      "href": "/v1/480924/queues/fizbit/messages?marker=6244-244224-783&limit=10"
    }
  ],
  "messages": [
    {
      "id": "50b68a50d6f5b8c8a7c62b01",
      "href": "/v1/480924/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b01"
      "userAgent": "python/2.7 killer-rabbit/1.2 uuid/79ed56f8-2519-11e2-b835-acf6018e45a1",
      "ts": "2012-12-04 16:53:20Z",
      "age": 790,
      "body": {
        "event": "ActivateAccount",
        "mode": "active"
      }
    }
  ]
}


Discussion

Message IDs and markers are opaque strings; clients should make no assumptions on their format or length. Furthermore, clients should assume there is no relationships between markers and message IDs (i.e., one cannot be derived from the other).

Results are ordered by age, oldest message first.

limit specifies up to x messages to return. Note that x is configurable, but must be at least 10. If not specified, limit defaults to x. When more messages are available than can be returned in a single request, the client can pick up the next batch by simply using the URI template parameters returned from the previous call in the "next" field (TBD).

marker is an opaque string that the client can use to request the next batch of messages. It communicates to the server which messages the client has already received.

Note: If marker is not specified, the API will return all messages (up to limit).

echo is a boolean (i.e., "true" or "false") that determines whether or not the API will return a client's own messages, as determined by the uuid portion of the User-Agent header. If not specified, echo defaults to "false".

Get Action Messages

Template


GET {base_url}/queues/{queue_name}/actions/{?marker,limit}


Request


GET /v1/480924/queues/fizbit/actions&marker=1355-237242-783&limit=10 HTTP/1.1
Host: marconi.example.com

[...]


Response

HTTP 200 if the query matched 1 or more actions, HTTP 204 otherwise (with no body).


HTTP/1.1 200 OK
Content-Location: /v1/480924/queues/fizbit/actions?marker=1355-237242-783&limit=10

[...]

{
  "links": [
    {
      "rel": "next",
      "href": "/v1/480924/queues/fizbit/actions?marker=6244-244224-783&limit=10"
    }
  ],
  "messages": [
    {
      "id": 10b00a50d6f5b8c8a7c62ccb,
      "href": "/v1/480924/queues/fizbit/actions/{id}"
      "userAgent": "marconi/1 uuid/f2e4b36a-3f05-11e2-b71d-7823d2b0f3ce",
      "ts": "2012-12-04 16:53:18Z",
      "age": 792,
      "body": {
        "event": "LockExpired",
        "details": {
          "messageId": "50b68a50d6f5b8c8a7c62a87",
          "userAgent": "python/2.7 killer-rabbit/1.2 uuid/79ed56f8-2519-11e2-b835-acf6018e45a1",
        }
      }
    },
    {
      "id": 10b00a50d6f5b8c8a7c62ccc,
      "href": "/v1/480924/queues/fizbit/actions/{id}"
      "userAgent": "marconi/1 uuid/f2e4b36a-3f05-11e2-b71d-7823d2b0f3ce",
      "ts": "2012-12-04 16:53:20Z",
      "age": 790,
      "body": {
        "event": "MessageCreated",
        "details": {
          "messageId": "50b68a50d6f5b8c8a7c62b01",
          "userAgent": "python/2.7 killer-rabbit/1.2 uuid/79ed56f8-2519-11e2-b835-acf6018e45a1",
        }
      }
    }
  ]
}


Discussion

The interactions of various agents/workers with a cloud queuing services can be difficult to audit and debug. Marconi emits action messages to a special queue, from which auditors can retrieve a list of recent actions involving a specific queue's messages. These actions can then be archived by the auditor for future analysis and diagnostics (action messages time out and are deleted according to the TTL configured for the associated queue).

Message IDs and markers are opaque strings; clients should make no assumptions on their format or length. Furthermore, clients should assume there is no relationships between markers and message IDs (i.e., one cannot be derived from the other).

Results are ordered by age, oldest message first.

limit specifies up to x messages to return. Note that x is configurable, but must be at least 10. If not specified, limit defaults to x. When more messages are available than can be returned in a single request, the client can pick up the next batch by simply using the URI template parameters returned from the previous call in the "next" field.

marker is an opaque string that the client can use to request the next batch of messages. It communicates to the server which messages the client has already received.

Note: If marker is not specified, the API will return all messages (up to limit).

Get a Specific Message

Template


GET {base_url}/queues/{queue_name}/messages/{message_id}


Request


GET /v1/480924/queues/fizbat/messages/50b68a50d6f5b8c8a7c62b01 HTTP/1.1
Host: marconi.example.com

[...]


Response


HTTP/1.1 200 OK
Content-Location: /v1/480924/queues/fizbat/messages/50b68a50d6f5b8c8a7c62b01

[...]

{
  "id": "50b68a50d6f5b8c8a7c62b01",
  "userAgent": "python/2.7 killer-rabbit/1.2 uuid/79ed56f8-2519-11e2-b835-acf6018e45a1",
  "age": 790,
  "body": {
    "event": "ActivateAccount",
    "mode": "active"
  }
}


Discussion

@todo Define message fields.

Post Messages

Template


POST {base_url}/queues/{queue_name}/messages


Request


POST /v1/480924/queues/fizbit/messages HTTP/1.1
Host: marconi.example.com

[...]


[
  {
    "body": {
      "event": "BackupStarted",
      "backupId": "c378813c-3f0b-11e2-ad92-7823d2b0f3ce"
    }
  },
  {
    "body": {
      "event": "BackupProgress",
      "currentBytes": "0",
      "totalBytes": "99614720"
    }
  }
]


Responses

When a single message is submitted:


HTTP/1.1 201 Created
Location: https://marconi.example.com/v1/480924/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b01


When multiple messages are submitted, the generic messages resource is returned for the location, since Marconi does not support a GET for a list of messages, e.g.:


HTTP/1.1 201 Created
Location: https://marconi.example.com/v1/480924/queues/fizbit/messages


Discussion

One or more messages may be submitted in a single request, but must always be encapsulated in a collection container (an array in JSON, a parent tag in XML). In the case of a batch POST, querying the returned Location may return messages posted concurrently by other agents.

body specifies a custom document which constitutes the body of the message being sent. The size of this body, in characters and including whitespace, is configurable (the default is 64 KiB).

Delete a Single Message

Template


DELETE {base_url}/queues/{queue_name}/messages/{message_id}{?claim_id}


Request


DELETE /v1/480924/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b01 HTTP/1.1
Host: marconi.example.com

[...]


Response


HTTP/1.1 204 No Content


Discussion

message_id specifies the message to delete.

claim_id specifies that the message should only be deleted if it has the specified claim ID, and that claim has not expired. This is useful for ensuring only one agent processes any given message; whenever a worker client's claim expires before it has a chance to delete a message it has processed, the worker must roll back any actions it took based on that message, since another worker will now be able to claim and process the same message.

Claim Messages

Template


PATCH {base_url}/queues/{queue_name}/messages{?limit}
Content-Type: application/json-patch


Request


PATCH /v1/480924/queues/fizbit/messages/limit=5 HTTP/1.1
Content-Type: application/json-patch
Accept: application/json

[...]

[
  {
    "op": "add",
    "path": "/claim",
    "value": {
      "ttl": 30
    }
  }
]


Response

The client receives a list of claimed messages, if any:


HTTP/1.1 200 OK
Content-Type: application/json

[...]

{
  "messages": [
    ...

    {
      ...

      "claim": {
        "id": "a28ee94e-6cb4-11e2-b4d5-7703267a7926",
        "ttl": 30,
        "age": 0
      }

      ...
    }
  ]
}


Discussion

Claims a set of messages, up to limit, from oldest to newest, skipping any that are already claimed. Uses JSON patch (RFC Draft).

The client should delete the message when it has finished processing it, before the claim expires, to ensure the message is processed only one time. As part of the delete operation, all worker clients should specify the claim ID. That way, the server can return an error if the claim just expired (notifying it of the race condition), giving the worker a chance to roll back its own processing of the given message, since another worker will likely claim the message and process it.

Just as with a message's age, the age given for each claim is relative to the server's clock, and is useful for determining how quickly messages are getting processed, and whether a given message's claim is about to expire.

When a claim expires, it is removed, allowing another client worker to claim the message in the case that the original worker fails to process it.

limit specifies up to x messages to claim. Note that x is configurable, but must be at least 10. If not specified, limit defaults to x.

Renew a Claim

Template


PATCH {base_url}/queues/{queue_name}/messages/{message_id}
Content-Type: application/json-patch


Request


PATCH /v1/480924/queues/fizbit/messages/10b00a50d6f5b8c8a7c62ccb HTTP/1.1
Content-Type: application/json-patch

[...]

[
  {
    "op": "test", "path": "/claim/id", "value": "a28ee94e-6cb4-11e2-b4d5-7703267a7926"
  },
  {
    "op": "replace", "path": "/claim/age", "value": 0
  }
]


Response


HTTP/1.1 204 No Content


Discussion

Clients should periodically renew claims during long-running batches of work to avoid loosing a claim in the middle of processing a message. The server only accepts "0" for the age value, and the test operation is required to ensure only the client that owns a given claim can renew it.

Count Messages

Template


GET {base_url}/queues/{queue_name}/stats


Request


GET /v1/480924/queues/fizbit/stats HTTP/1.1
Host: marconi.example.com

[...]


Response HTTP/1.1 200 OK


{
  "messages": 46,
  "actions": 122
}


Discussion

Returns queue statistics, including how many messages are in the queue, and how many actions have been recorded (actions can be retrieved by performing a GET on {queue_name}/actions).

Check Health

Template


GET {base_url}/health


or


HEAD {base_url}/health


Request


GET /v1/480924/health HTTP/1.1
Host: example.marconi.com

[...]


Response


HTTP/1.1 200 OK

[...]

{
  "code": "green",
  "link": {
    "text": "Service status page"
    "href": "http://marconi.example.com/status",
    "rel": "help",
  }
}


Discussion

Use this request to check on the Marconi service status as a whole. The following status values are defined:

Code
green
yellow
red

Note that if a client performs HEAD instead of GET, the server will return one of the following:

Status
HTTP/1.1 204 No Content
HTTP/1.1 503 Service Unavailable

TBD

Template


GET /{version}/{tenant}


Request


GET /{version}/{tenant} HTTP/1.1


Response


@todo


Discussion

@todo