Jump to: navigation, search

Difference between revisions of "Zaqar/specs/api/v1"

(Move decisions into their own section)
(Post Message(s): Note about "partial" deprecation)
 
(147 intermediate revisions by 9 users not shown)
Line 1: Line 1:
__NOTOC__
 
 
= Marconi API: v1 Blueprint =
 
= Marconi API: v1 Blueprint =
  
See also: [[marconi/specs/grizzly]]
 
  
''' Decisions '''
+
<big>'''The Marconi v1 API is FROZEN'''</big>
  
* Consider splitting API into two namespaces, one for work queuing and one for eventing
+
We have started collecting feedback with an eye toward extensions and v2 of the API here: [[Marconi/specs/api/next]]
** Allows for different scaling architectures, storage backends, etc.
 
** Simplifies semantics for the two major use cases
 
** Downside is we remove affordances - clients won't be able to mix/match semantics (we would be effectively removing features)
 
* Decide whether message signing should be a mandatory or optional feature (probably optional)
 
* Decide whether client state should be based on timestamps or markers (or make one optional)
 
** Markers assume some kind of monotonic message IDs and/or a sequential storage schema.
 
* Decide whether Marconi should be responsible for cross-availability zone and even cross-DC replication/availability
 
  
''' To Do '''
 
 
* Clarify use cases around tags
 
* Strongly consider defining top-level queues to facilitate sharing queues between tenants
 
** Makes it easier to layer on top of existing message queueing systems / use them as backing stores.
 
** Define the use case / business case for sharing queues between tenants, figure out how that might work with [[KeyStone]] (i.e., authorizing access to particular queues based on auth token).
 
** Implementations may decide to implement these as syntactic sugar on top of tags
 
* Flesh out examples, incomplete sections (obviously)
 
* Clean up this document, add a FAQ section
 
  
 
== Overview ==
 
== Overview ==
  
Marconi provides an HTTP-based API in the spirit of the [https://en.wikipedia.org/wiki/Representational_state_transfer REST architecture style].
+
Marconi provides an HTTP-based API in the spirit of the [https://en.wikipedia.org/wiki/Representational_state_transfer REST architecture style].
  
REST is a natural fit for Marconi's design philosophy, shamelessly borrowed from Donald A. Norman's work regarding The Design of Everyday Things:
+
This guide assumes the reader is familiar with REST, HTTP/1.1 and JSON. URI templates use the same syntax as [http://tools.ietf.org/html/rfc6570 RFC 6570].
  
  The value of a well-designed object is when it has such a rich set of affordances that the people who use it can do things with it that the designer never imagined.
+
Note: Error responses are enumerated on a separate page: [[Marconi/specs/api/v1/errors]]
  
This guide assumes the reader is familiar with REST, HTTP/1.1, JSON and XML. URI templates are specified according to [http://tools.ietf.org/html/rfc6570 RFC 6570].
 
  
 
== Common API Elements ==
 
== Common API Elements ==
  
''' HTTPS '''
 
 
All requests to authenticate and operate against the API in production environments use SSL/TLS over HTTP (HTTPS) on TCP port 443. Web servers should only accept high-quality cipher suites, such as AES256_SHA and ECDHE_RSA_AES256_SHA. If hardware acceleration (e.g., AES_NI) is not available, RC4_SHA and ECDHE_RSA_RC4_SHA may be used with some discretion.
 
  
''' Clients '''
+
=== Clients ===
  
 
* Clients should follow HTTP redirects
 
* Clients should follow HTTP redirects
Line 48: Line 26:
 
* Clients must not hard-code URI paths or templates since they may change over time. Instead, clients should cache links and link templates provided by the API.
 
* Clients must not hard-code URI paths or templates since they may change over time. Instead, clients should cache links and link templates provided by the API.
  
''' API Versioning '''
+
 
 +
=== API Versioning ===
  
 
The Marconi API uses a URI versioning scheme. The first element of the path contains the target version identifier, e.g.:
 
The Marconi API uses a URI versioning scheme. The first element of the path contains the target version identifier, e.g.:
 
  
 
<pre><nowiki>
 
<pre><nowiki>
Line 57: Line 35:
 
</nowiki></pre>
 
</nowiki></pre>
  
 +
The URI version will only be incremented to accommodate major new features or API redesigns that can not be made backwards-compatible. When new API versions are released, older versions are deprecated. Marconi maintainers will work with developers and partners to ensure there is adequate time to migrate to new versions before deprecated ones are discontinued.
  
The URI version will only be incremented to accomodate major new features or API redesigns that can not be made backwards-compatible. When new API versions are released, older versions are deprecated. Marconi maintainers will work with developers and partners to ensure there is adequate time to migrate to new versions before deprecated ones are discontinued.
+
Since the version is only incremented to accommodate major API revisions, sub-versioning of the API will be rare, but may be used occasionally to release a "polished" major version.
  
Since the base URI is only incremented to accommodate major API revisions, sub-versioning of the API is meaningless and is therefore not used. For example, the next version after "v1" would be "v2", not "v1.1" or some variant thereof. (HATEOS and media types are used in lieu of minor, URI-based versioning.)
+
=== Resource Media Types ===
  
''' Resource Media Types '''
+
The API supports `application/json` (XML is on the road map), encoded as UTF-8. Gzip'd requests are optionally supported by the server.
  
The API supports `application/json` and `application/xml`, encoded as UTF-8. Gzip'd requests are optionally supported by the server.
+
Unrecognized protocol elements should be ignored. This includes XML elements and attributes, JSON object properties, link relation types, media types, etc.
  
Unrecognised protocol elements should be ignored. This includes XML elements and attributes, JSON object properties, link relation types, media types, etc.
 
  
TODO: Define and register message media type/ messages media type - http://www.iana.org/assignments/media-types/index.html
+
=== Authentication ===
  
''' Authentication '''
+
''May be optionally implemented by middleware or reverse proxy. If implemented, the following applies:''
  
 
All requests to the API may only be performed by an authenticated agent.
 
All requests to the API may only be performed by an authenticated agent.
Line 78: Line 56:
 
In response to valid credentials, Keystone responds with an auth token and a service catalog that contains a list of all services and endpoints available for the given token. Multiple endpoints may be returned for Marconi according to physical locations and performance/availability characteristics of different deployments.
 
In response to valid credentials, Keystone responds with an auth token and a service catalog that contains a list of all services and endpoints available for the given token. Multiple endpoints may be returned for Marconi according to physical locations and performance/availability characteristics of different deployments.
  
Marconi clients must specify a valid auth token in the `X-Auth-Token` header for each request to the Marconi API. The API validates auth tokens against Keystone before servicing each request.
+
Normally, Keystone middleware provides the X-Project-Id header based on the auth token submitted by the Marconi client. For this to work, clients MUST specify a valid auth token in the `X-Auth-Token` header for each request to the Marconi API. The API validates auth tokens against Keystone before servicing each request
  
''' Authorization '''
+
If auth is not enabled, clients must provide the X-Project-Id header themselves.
  
TBD: The API needs to verify read/write access to all endpoints according to the provided auth token.
+
=== Authorization ===
  
''' Request Signing '''
+
''May be optionally implemented by middleware or reverse proxy. If implemented, the following applies:''
  
Messages may optionally contain a signature to verify its authenticity.
+
The API needs to verify read/write access to all endpoints according to the provided auth token (RBAC). The ACL should be cached with the token.
  
TODO: How to register and query public keys? Via Keystone? Shouldn't be part of Marconi, in any case.
 
  
How to calculate signature?
+
=== Endpoints ===
  
[[SecP256K1]] or something using RSA-2048? (encrypt a SHA-512 value). Would be nice to allow different schemes... How is Ceilometer doing it?
+
Clients may choose to interact with any of the service endpoints retrieved from Keystone. Agents use the given endpoint as a home document href from which to discover links to all other API requests. Endpoints may be organized by region
 
 
See also: http://stackoverflow.com/questions/9897023/asymmetric-hmac#
 
 
 
''' Tenant ID '''
 
 
 
Auth tokens are only valid for a particular tenant ID, which should be reflected in the service endpoints retrieved from Keystone. Agents use the given endpoint as a home document href from which to discover links to all other API requests.  
 
  
 
An example endpoint:
 
An example endpoint:
 
  
 
<pre><nowiki>
 
<pre><nowiki>
https://marconi.example.com/v1/480924
+
https://marconi.example.com/v1
 
</nowiki></pre>
 
</nowiki></pre>
  
 +
=== Errors ===
  
The client chooses one of the presented endpoints and uses it as the base URL for all subsequent requests.
+
If any request results in an error, the server will return an appropriate 4xx or 5xx HTTP status code, and optionally the following information in the body:
 
 
''' Caching '''
 
 
 
@todo
 
 
 
Can be used to save bandwidth at the expense of server CPU and/or added complexity. Provide an ETag for results, the client submits in subsequent request using the If-None-Match header. Server performs query against data store, dynamically generates ETag, compares to client's ETag, and returns 204 if-and-only-if the two ETags don't match. This is especially helpful in cutting down on duplicate traffic due to the inability of the server to guarantee message ordering.
 
 
 
''' Errors '''
 
 
 
If any request results in an error, the server will return an appropriate 4xx or 5xx HTTP status code, as well as the following information in the body:
 
  
 
* Title
 
* Title
Line 124: Line 85:
 
* Internal code
 
* Internal code
 
* Link to more information
 
* Link to more information
 +
  
 
Example:
 
Example:
 
  
 
<pre><nowiki>
 
<pre><nowiki>
Line 137: Line 98:
 
   "code": 1092,
 
   "code": 1092,
 
   "link": {
 
   "link": {
     "rel": "api-doc",
+
     "rel": "help",
     "href": "http://example.com/marconi/docs/messages#limit",
+
     "href": "http://docs.example.com/messages#limit",
 
     "text": "API documentation for the limit parameter"
 
     "text": "API documentation for the limit parameter"
 
   }
 
   }
Line 144: Line 105:
 
</nowiki></pre>
 
</nowiki></pre>
  
 
+
=== Common Headers ===
''' Common Headers '''
 
  
 
Each request to the API must include certain standard and extended HTTP headers. These headers provide host, agent, authentication and other pertinent information to the server.
 
Each request to the API must include certain standard and extended HTTP headers. These headers provide host, agent, authentication and other pertinent information to the server.
  
 
{| border="1" cellpadding="2" cellspacing="0"
 
{| border="1" cellpadding="2" cellspacing="0"
|  '''Header'''  
+
|  '''Header'''
|  '''Description'''  
+
|  '''Description'''
 
|-
 
|-
|  Host  
+
|  Host
|  The host name of the API, as referenced by the Keystone service catalog  
+
|  The host name of the API, as referenced by the Keystone service catalog
 
|-
 
|-
User-Agent
+
Date
|  The name and version of the Marconi client, as well as a UUID for that client. Marconi uses the UUID to distinguish publishers from subscribers, i.e., to avoid echoing an agent's own messages back to it.
+
|  The current date and time, using the standard RFC 1123 HTTP date format
 
|-
 
|-
Date
+
Accept
The current date and time, using the standard RFC 1123 HTTP date format
+
Media type desired; only <code><nowiki>application/json</nowiki></code> is supported at this time.
 
|-
 
|-
|  Accept  
+
|  Accept-Encoding
Media type desired; may be either `application/json` or `application/xml`
+
Specifies that the agent accepts gzip-encoded response bodies
 
|-
 
|-
Accept-Encoding
+
Content-Length
Specifies that the agent accepts gzip-encoded response bodies
+
For POST or PUT requests, the length in bytes of the message document being submitted
 
|-
 
|-
Content-Length
+
X-Auth-Token
For POST or PUT requests, the length in bytes of the JSON or XML document being submitted
+
Keystone auth token
 
|-
 
|-
|  X-Auth-Token  
+
X-Project-Id
Keystone auth token
+
|  An ID for a project to which the value of X-Auth-Token grants access. Queues will be created under this project.
 +
|-
 +
|  Client-ID
 +
A UUID, used to distinguish each client using the service. The UUID must be submitted in its canonical form (e.g., <code><nowiki>3381af92-2b9e-11e3-b191-71861300734c</nowiki></code>). In Marconi, this is used to avoid echoing a sender's messages back to the same instance, and may be logged by the server for future use. Should be generated once and persisted between restarts of the client.
 
|}
 
|}
 +
 +
=== HTTP Response Codes ===
 +
 +
See [[Marconi/specs/api/v1/responsecodes| Response Codes]]
  
 
== Sample API Request ==
 
== Sample API Request ==
 
  
 
<pre><nowiki>
 
<pre><nowiki>
GET /v1/480924/messages?tags=channel-foo,topic-bar&ts=1355237242783&limit=10 HTTP/1.1
+
GET /v1/queues/fizbat/messages?marker=1355-237242-783&limit=10 HTTP/1.1
 
Host: marconi.example.com
 
Host: marconi.example.com
User-Agent: python/2.7 killer-rabbit/1.2 uuid/30387f00-39a0-11e2-be4d-a8d15f34bae2
+
User-Agent: python/2.7 killer-rabbit/1.2
 
Date: Wed, 28 Nov 2012 21:14:19 GMT
 
Date: Wed, 28 Nov 2012 21:14:19 GMT
 
Accept: application/json
 
Accept: application/json
 
Accept-Encoding: gzip
 
Accept-Encoding: gzip
 
X-Auth-Token: 7d2f63fd-4dcc-4752-8e9b-1d08f989cc00
 
X-Auth-Token: 7d2f63fd-4dcc-4752-8e9b-1d08f989cc00
 +
X-Project-Id: 518b51ea133c4facadae42c328d6b77b
 +
Client-ID: 30387f00-39a0-11e2-be4d-a8d15f34bae2
 
</nowiki></pre>
 
</nowiki></pre>
  
 +
== Endpoints Synopsis ==
 +
 +
<pre><nowiki>
 +
# Base endpoints
 +
GET /v1
 +
GET /v1/health
 +
 +
# Queues
 +
GET /v1/queues{?marker,limit,detailed}
 +
GET /v1/queues/{queue_name}  # existence check
 +
HEAD /v1/queues/{queue_name}  # existence check
 +
PUT /v1/queues/{name}
 +
DELETE /v1/queues/{name}
 +
 +
# Queue metadata
 +
PUT /v1/queues/{queue_name}/metadata
 +
GET /v1/queues/{queue_name}/metadata
 +
GET /v1/queues/{queue_name}/stats
 +
 +
# Messages
 +
GET /v1/queues/{queue_name}/messages{?marker,limit,echo,include_claimed}
 +
GET /v1/queues/{queue_name}/messages/{message_id}
 +
GET /v1/queues/{queue_name}/messages{?ids}
 +
POST /v1/queues/{queue_name}/messages
 +
DELETE /v1/queues/{queue_name}/messages/{message_id}{?claim_id}
 +
DELETE /v1/queues/{queue_name}/messages{?ids}
 +
 +
# Claims
 +
POST /v1/queues/{queue_name}/claims{?limit}
 +
GET /v1/queues/{queue_name}/claims/{claim_id}
 +
PATCH /v1/queues/{queue_name}/claims/{claim_id}
 +
DELETE /v1/queues/{queue_name}/claims/{claim_id}
 +
</nowiki></pre>
  
== Get Home Document ==
+
== Base Endpoints ==
  
'''Request Template'''
+
=== Get Home Document ===
  
 +
'''Template'''
  
 
<pre><nowiki>
 
<pre><nowiki>
GET {base_url}
+
GET /v1
 
</nowiki></pre>
 
</nowiki></pre>
  
  
 
'''Request'''
 
'''Request'''
 
  
 
<pre><nowiki>
 
<pre><nowiki>
GET /v1/480924 HTTP/1.1
+
GET /v1 HTTP/1.1
 
Host: marconi.example.com
 
Host: marconi.example.com
  
[...]
+
...
  
 
</nowiki></pre>
 
</nowiki></pre>
Line 213: Line 214:
 
'''Response'''
 
'''Response'''
  
 +
<pre><nowiki>
 +
HTTP/1.0 200 OK
 +
 +
...
  
<pre><nowiki>
+
{
@todo
+
    "resources": {
 +
        "rel/queue-stats": {
 +
            "href-template": "/v1/queues/{queue_name}/stats",
 +
            "href-vars": {
 +
                "queue_name": "param/queue_name"
 +
            },
 +
            "hints": {
 +
                "allow": [
 +
                    "GET"
 +
                ],
 +
                "formats": {
 +
                    "application/json": {}
 +
                }
 +
            }
 +
        },
 +
        "rel/post-messages": {
 +
            "href-template": "/v1/queues/{queue_name}/messages",
 +
            "href-vars": {
 +
                "queue_name": "param/queue_name"
 +
            },
 +
            "hints": {
 +
                "accept-post": [
 +
                    "application/json"
 +
                ],
 +
                "allow": [
 +
                    "POST"
 +
                ],
 +
                "formats": {
 +
                    "application/json": {}
 +
                }
 +
            }
 +
        },
 +
        "rel/queue": {
 +
            "href-template": "/v1/queues/{queue_name}",
 +
            "href-vars": {
 +
                "queue_name": "param/queue_name"
 +
            },
 +
            "hints": {
 +
                "allow": [
 +
                    "GET",
 +
                    "HEAD",
 +
                    "PUT",
 +
                    "DELETE"
 +
                ],
 +
                "formats": {
 +
                    "application/json": {}
 +
                }
 +
            }
 +
        },
 +
        "rel/queue-metadata": {
 +
            "href-template": "/v1/queues/{queue_name}/metadata",
 +
            "href-vars": {
 +
                "queue_name": "param/queue_name"
 +
            },
 +
            "hints": {
 +
                "allow": [
 +
                    "GET",
 +
                    "PUT"
 +
                ],
 +
                "formats": {
 +
                    "application/json": {}
 +
                }
 +
            }
 +
        },
 +
        "rel/queues": {
 +
            "href-template": "/v1/queues{?marker,limit,detailed}",
 +
            "href-vars": {
 +
                "marker": "param/marker",
 +
                "detailed": "param/detailed",
 +
                "limit": "param/queue_limit"
 +
            },
 +
            "hints": {
 +
                "allow": [
 +
                    "GET"
 +
                ],
 +
                "formats": {
 +
                    "application/json": {}
 +
                }
 +
            }
 +
        },
 +
        "rel/messages": {
 +
            "href-template": "/v1/queues/{queue_name}/messages{?marker,limit,echo,include_claimed}",
 +
            "href-vars": {
 +
                "marker": "param/marker",
 +
                "include_claimed": "param/include_claimed",
 +
                "queue_name": "param/queue_name",
 +
                "limit": "param/messages_limit",
 +
                "echo": "param/echo"
 +
            },
 +
            "hints": {
 +
                "allow": [
 +
                    "GET"
 +
                ],
 +
                "formats": {
 +
                    "application/json": {}
 +
                }
 +
            }
 +
        },
 +
        "rel/claim": {
 +
            "href-template": "/v1/queues/{queue_name}/claims{?limit}",
 +
            "href-vars": {
 +
                "queue_name": "param/queue_name",
 +
                "limit": "param/claim_limit"
 +
            },
 +
            "hints": {
 +
                "accept-post": [
 +
                    "application/json"
 +
                ],
 +
                "allow": [
 +
                    "POST"
 +
                ],
 +
                "formats": {
 +
                    "application/json": {}
 +
                }
 +
            }
 +
        }
 +
    }
 +
}
 
</nowiki></pre>
 
</nowiki></pre>
  
Line 221: Line 343:
 
'''Discussion'''
 
'''Discussion'''
  
The entire Marconi API is discoverable from a single starting point; agents do not need to know any more than this one URI in order to explore the entire API.  
+
The entire Marconi API is discoverable from a single starting point; agents do not need to know any more than this one URI in order to explore the entire API.
 +
 
 +
This document is cache-able (and should be cached by proxy and client).
  
 
See also: http://tools.ietf.org/html/draft-nottingham-json-home-02
 
See also: http://tools.ietf.org/html/draft-nottingham-json-home-02
  
== Get a Specific Message ==
+
=== Check Node Health ===
 +
 
  
'''Request Template'''
+
'''Template'''
 +
 
 +
<pre><nowiki>
 +
GET /v1/health
 +
</nowiki></pre>
  
 +
or
  
 
<pre><nowiki>
 
<pre><nowiki>
GET {base_url}/messages/{messageId}
+
HEAD /v1/health
 
</nowiki></pre>
 
</nowiki></pre>
  
  
 
'''Request'''
 
'''Request'''
 
  
 
<pre><nowiki>
 
<pre><nowiki>
GET /v1/480924/messages/50b68a50d6f5b8c8a7c62b01 HTTP/1.1
+
GET /v1/health HTTP/1.1
Host: marconi.example.com
+
Host: example.marconi.com
  
[...]
+
...
  
 
</nowiki></pre>
 
</nowiki></pre>
Line 248: Line 377:
  
 
'''Response'''
 
'''Response'''
 
  
 
<pre><nowiki>
 
<pre><nowiki>
HTTP/1.1 200 OK
+
HTTP/1.1 204 No Content
 +
</nowiki></pre>
  
[...]
+
or
  
{
+
<pre><nowiki>
  "id": 50b68a50d6f5b8c8a7c62b01,
+
HTTP/1.1 503 Service Unavailable
  "userAgent": "python/2.7 killer-rabbit/1.2 uuid/79ed56f8-2519-11e2-b835-acf6018e45a1",
 
  "age": 790,
 
  "tags": [
 
    foo,
 
    snap,
 
    bar,
 
    bang
 
  ],
 
  "signature": {
 
    "scheme": "SecP256K1",
 
    "salt": "b8c850d6f5b8a7c6",
 
    "value": "T2gJfGQNgUe1XMDHyucwH27zn628it0fWrCFgE2mPWR+oMTOiW7jb1OPNuZtLus5O1IzTzy+5ALyLCyUq7JoLQ=="
 
  },
 
  "body": {
 
    "event": "ActivateAccount",
 
    "mode": "active"
 
  }
 
}
 
 
</nowiki></pre>
 
</nowiki></pre>
 +
  
  
 
'''Discussion'''
 
'''Discussion'''
  
@todo Define message fields.
+
Use this request to check on the Marconi node status. If the node is down, this endpoint will not be reachable and will act as a signal to the load balancer to take the node out of rotation. Alternatively, the endpoint may be reachable but the service is temporarily unavailable due to storage driver failure or some other error.
  
== Get Messages ==
+
== Queues ==
  
'''Request Template'''
+
=== Create Queue ===
  
 +
 +
'''Template'''
  
 
<pre><nowiki>
 
<pre><nowiki>
GET {base_url}/messages{?tags,ts,limit,sort,meta,echo}
+
PUT /v1/queues/{queue_name}
 
</nowiki></pre>
 
</nowiki></pre>
 
  
 
'''Request'''
 
'''Request'''
 
  
 
<pre><nowiki>
 
<pre><nowiki>
GET /v1/480924/messages?tags=foo,bar,bang&ts=1355237242783&limit=10&sort=desc HTTP/1.1
+
PUT /v1/queues/fizbat HTTP/1.1
 
Host: marconi.example.com
 
Host: marconi.example.com
 +
</nowiki></pre>
  
[...]
 
  
 +
'''Response'''
 +
 +
<pre><nowiki>
 +
HTTP/1.1 201 Created
 +
Location: /v1/queues/fizbit
 
</nowiki></pre>
 
</nowiki></pre>
  
  
'''Response'''
+
'''Discussion'''
 +
 
 +
Creates a queue.
 +
 
 +
The body of the PUT is empty.
 +
 
 +
<code><nowiki>queue_name</nowiki></code> is the name to give the queue. The name MUST NOT exceed 64 bytes in length, and is limited to US-ASCII letters, digits, underscores and hyphens.
 +
 
 +
=== List Queues ===
  
HTTP 200 if the query matched 1 or more messages, HTTP 204 otherwise (with no body).
 
  
 +
'''Template'''
  
 
<pre><nowiki>
 
<pre><nowiki>
HTTP/1.1 200 OK
+
GET /v1/queues{?marker,limit,detailed}
ETag: "2:fa0d1a60ef6616bb28038515c8ea4cb2"
+
 
 +
</nowiki></pre>
 +
 
 +
 
 +
'''Request'''
  
[...]
+
<pre><nowiki>
 +
GET /v1/queues?marker=baz&detailed=true HTTP/1.1
 +
Host: marconi.example.com
  
{
+
...
  "messageCount": 1,
 
  "links": [
 
    {
 
      "rel": "self"
 
      "href-template": "{base_url}/messages?tags=foo,bar,bang&ts=1355237242783&limit=10&sort=desc"
 
    },
 
    {
 
      "rel": "next"
 
      "href-template": "{base_url}/messages?tags=foo,bar,bang&ts=1355237244190&limit=10&sort=desc"
 
    }
 
  ],
 
  "messages": [
 
    {
 
      "id": 50b68a50d6f5b8c8a7c62b01,
 
      "href-template": "{base_url}/messages/{id}"
 
      "userAgent": "python/2.7 killer-rabbit/1.2 uuid/79ed56f8-2519-11e2-b835-acf6018e45a1",
 
      "ts": 1355237244190
 
      "age": 790,
 
      "tags": [
 
        foo,
 
        snap,
 
        bar,
 
        bang
 
      ],
 
      "signature": {
 
        "scheme": "SecP256K1",
 
        "salt": "b8c850d6f5b8a7c6",
 
        "value": "T2gJfGQNgUe1XMDHyucwH27zn628it0fWrCFgE2mPWR+oMTOiW7jb1OPNuZtLus5O1IzTzy+5ALyLCyUq7JoLQ=="
 
      },
 
      "body": {
 
        "event": "ActivateAccount",
 
        "mode": "active"
 
      }
 
    }
 
  ]
 
}
 
 
</nowiki></pre>
 
</nowiki></pre>
  
  
Example meta-messages response (if a client were to add meta=true to the sample query).
+
'''Response'''
  
 
<pre><nowiki>
 
<pre><nowiki>
 
HTTP/1.1 200 OK
 
HTTP/1.1 200 OK
ETag: "2:fa0d1a60ef6616bb28038515c8ea4cb2"
 
 
[...]
 
  
 
{
 
{
  "messageCount": 2,
 
 
   "links": [
 
   "links": [
 
     {
 
     {
       "rel": "self"
+
       "rel": "next",
       "href-template": "{base_url}/messages?tags=foo,bar,bang&ts=1355237242783&limit=10&sort=desc&meta=true"
+
       "href": "/v1/queues?marker=kooleo&limit=10&detailed=true"
    },
 
    {
 
      "rel": "next"
 
      "href-template": "{base_url}/messages?tags=foo,bar,bang&ts=1355237244210&limit=10&sort=desc&meta=true"
 
 
     }
 
     }
 
   ],
 
   ],
   "messages": [
+
   "queues": [
     {
+
     { "name": "boomerang", "href": "/v1/queues/boomerang", "metadata": {} },
      "id": 10b00a50d6f5b8c8a7c62ccc,
+
     { "name": "fizbit", "href": "/v1/queues/fizbit", "metadata": { "handle": "@kgriffs" } },
      "href-template": "{base_url}/messages/{id}"
+
 
      "userAgent": "marconi/1 uuid/f2e4b36a-3f05-11e2-b71d-7823d2b0f3ce",
+
    ...
      "ts": 1355237242783,
+
 
      "age": 790,
+
    { "name": "kooleo", "href": "/v1/queues/kooleo", "metadata": { "something": "something_else" } }
      "tags": [
 
        foo,
 
        snap,
 
        bar,
 
        bang
 
      ],
 
      "body": {
 
        "event": "MessageCreated",
 
        "timestamp": "2012-12-04 16:53:20Z",
 
        "details": {
 
          "messageId": "50b68a50d6f5b8c8a7c62b01",
 
          "userAgent": "python/2.7 killer-rabbit/1.2 uuid/79ed56f8-2519-11e2-b835-acf6018e45a1",
 
        }
 
      }
 
    },
 
     {
 
      "id": 10b00a50d6f5b8c8a7c62ccb,
 
      "href-template": "{base_url}/messages/{id}"
 
      "userAgent": "marconi/1 uuid/f2e4b36a-3f05-11e2-b71d-7823d2b0f3ce",
 
      "ts": 1355237244210,
 
      "age": 792,
 
      "tags": [
 
        foo,
 
        snap,
 
        bar,
 
        bang
 
      ],
 
      "body": {
 
        "event": "LockExpired",
 
        "timestamp": "2012-12-04 16:53:18Z",
 
        "details": {
 
          "messageId": "50b68a50d6f5b8c8a7c62a87",
 
          "userAgent": "python/2.7 killer-rabbit/1.2 uuid/79ed56f8-2519-11e2-b835-acf6018e45a1",
 
        }
 
      }
 
    }
 
 
   ]
 
   ]
 
}
 
}
 +
 
</nowiki></pre>
 
</nowiki></pre>
  
 +
=== Checking Queue Existence ===
  
'''Discussion'''
+
'''Template'''
  
Message IDs are opaque strings; clients should make no assumptions on their format or length, except that IDs may not exceed 50 characters in length.
+
<pre><nowiki>
 +
GET /v1/queues/{queue_name}
 +
</nowiki></pre>
  
<code><nowiki>tags</nowiki></code> is a list of up to n message tags, where n >= 2. The maximum number of tags supported is configurable. The API will return only those messages containing ALL of the specified tags. If no tags are specified, all messages are returned.
 
  
<code><nowiki>limit</nowiki></code> specifies up to x messages to return. Note that x is configurable, but must be at least 10. If not specified, limit defaults to x. When more messages are available than can be returned in a single request, the client can pick up the next batchby simply using the URI template parameters returned from the previous call in the "next" field (TBD).
+
'''Request'''
  
<code><nowiki>ts</nowiki></code> is the timestamp (64-bit UNIX timestamp in milliseconds) from the last message the client saw (relative to the server - use the "ts" message field). The API will return messages that were enqueued after the specified time, minus ''t'' milliseconds, where t is an implementation-defined number of milliseconds within which the server cannot guarantee message ordering (admittedly, a leaky abstraction). The client must cache messages for t milliseconds in order to check for duplicates returned in subsequent requests. Note that the message having the exact given timestamp (ts) will probably be part of the result set unless that particular message has expired or an ETag submitted by the client allows the server to definitively determine that no new messages match the given query string since the last request.
+
<pre><nowiki>
 +
GET /v1/queues/{queue_name}
 +
Host: marconi.example.com
 +
</nowiki></pre>
 +
 
 +
 
 +
'''Response'''
  
Note: If <code><nowiki>ts</nowiki></code> is not specified, the API will return all messages.
+
<pre><nowiki>
 +
HTTP/1.1 204 OK
 +
</nowiki></pre>
  
<code><nowiki>sort</nowiki></code> specifies how to chronologically order results. Use "asc" or "desc" for ascending or descending, respectively. If sort is not given, the API will return results in ascending order (oldest message first).
 
  
<code><nowiki>meta</nowiki></code> is a boolean value (i.e., "true" or "false") that determines whether the API will return actual messages or meta-messages. Meta-messages are "messages about messages", and are automatically generated by the server. Clients may query for meta-messages to audit business processes or to diagnose data flow issues. The default value for <code><nowiki>meta</nowiki></code> is "false".
+
'''Discussion'''
  
<code><nowiki>echo</nowiki></code> is a boolean (i.e., "true" or "false") that determines whether or not the API will return a client's own messages, as determined by the uuid portion of the User-Agent header. If not specified, echo defaults to "false".  
+
Returns 204 if the queue exists, else 404.
  
== Post Messages ==
+
<code><nowiki>HEAD</nowiki></code> also works.
  
'''Request Template'''
+
=== Delete Queue ===
  
 +
'''Template'''
  
 
<pre><nowiki>
 
<pre><nowiki>
POST {base_url}/messages
+
DELETE /v1/queues/{queue_name}
 
</nowiki></pre>
 
</nowiki></pre>
  
  
 
'''Request'''
 
'''Request'''
 
  
 
<pre><nowiki>
 
<pre><nowiki>
POST /v1/480924/messages HTTP/1.1
+
DELETE /v1/queues/fizbat HTTP/1.1
 
Host: marconi.example.com
 
Host: marconi.example.com
 +
</nowiki></pre>
  
[...]
 
  
 +
'''Response'''
  
[
+
<pre><nowiki>
  {
+
HTTP/1.1 204 No Content
    "ttl": 10,     
 
    "durability": 3,
 
    "tags": [420D29D6-3F24-11E2-BC14-7823D2B0F3CE, checkpoint]
 
    "signature": {
 
      "scheme": "SecP256K1",
 
      "salt": "b8c850d6f5b8a7c6",
 
      "value": "T2gJfGQNgUe1XMDHyucwH27zn628it0fWrCFgE2mPWR+oMTOiW7jb1OPNuZtLus5O1IzTzy+5ALyLCyUq7JoLQ=="
 
    },
 
    "body": {
 
      "event": "BackupStarted",
 
      "backupId": "c378813c-3f0b-11e2-ad92-7823d2b0f3ce"
 
    }
 
  },
 
  {
 
    "tags": [420D29D6-3F24-11E2-BC14-7823D2B0F3CE, progress]
 
    "body": {
 
      "event": "BackupProgress",
 
      "currentBytes": "0",
 
      "totalBytes": "99614720"
 
    }
 
  }
 
]
 
 
</nowiki></pre>
 
</nowiki></pre>
  
  
'''Responses'''
+
'''Discussion'''
 
 
When a single message is submitted:
 
  
 +
Use this operation to immediately delete a queue along with all its messages (if any).
  
<pre><nowiki>
+
== Queue Metadata ==
HTTP/1.1 201 Created
 
Location: https://marconi.example.com/v1/480924/messages/50b68a50d6f5b8c8a7c62b01
 
</nowiki></pre>
 
  
 +
=== Set Queue Metadata ===
  
When multiple messages are submitted:
 
  
 +
'''Template'''
  
 
<pre><nowiki>
 
<pre><nowiki>
HTTP/1.1 201 Created
+
PUT /v1/queues/{queue_name}/metadata
Location: https://marconi.example.com/v1/480924/messages&ts=1355237242783
+
 
 +
{
 +
  ...
 +
}
 
</nowiki></pre>
 
</nowiki></pre>
  
  
'''Discussion'''
+
'''Request'''
 +
 
 +
<pre><nowiki>
 +
PUT /v1/queues/fizbat/metadata HTTP/1.1
 +
Host: marconi.example.com
  
One or more messages may be submitted in a single request, but must always be encapsulated in a collection container (an array in JSON, a parent tag in XML). In the case of a batch POST, querying the returned <code><nowiki>Location</nowiki></code> may return messages posted concurrently by other agents.
+
...
  
<code><nowiki>ttl</nowiki></code> is the number of seconds the server will keep a message before automatically deleting it. Should be long enough to give all observers ample time to retrieve the message. If not specified, defaults to the maximum default set for any of the message's associated tags, or the default set for the tenant (TBD), or the one configured in the deployment (?).
+
{
 +
  "key": {
 +
      "key2": "value",
 +
      "key3": [1, 2, 3, 4, 5]
 +
    }
 +
}
 +
</nowiki></pre>
  
<code><nowiki>tags</nowiki></code> is a list of up to n tags to associate with a given message, where n >= 2. The maximum number of tags supported is configurable (the default is 5). The maximum length of each tag is likewise configurable (with a default of 150 characters). Tags are case-sensitive.
 
  
<code><nowiki>signature</nowiki></code> - TBD, probably something like sign(hash(salt + payload))
+
'''Response'''
  
<code><nowiki>body</nowiki></code> specifies a custom document which constitutes the body of the message being sent. The size of this body, in characters and including whitespace, is configurable (the default is 64 KiB).
+
<pre><nowiki>
 +
HTTP/1.1 204 No Content
 +
</nowiki></pre>
  
<code><nowiki>durability</nowiki></code> requests a certain durability guarantee from the server. The purpose of this parameter is to allow clients to dynamically make tradeoffs between durability and cost/performance depending on the type of message being sent. Note that the maximum durability level supported by the server is configurable; not all deployments will support all levels. If a level is unsupported, the server will return 400 Bad Request. If a level is supported but the server is unable to complete the request, and appropriate 5xx error will be returned to the client.
 
  
The following levels are defined:
+
'''Discussion'''
  
{| border="1" cellpadding="2" cellspacing="0"
+
This operation allows for setting structured metadata. If PUT is used, this document will replace the existing metadata document in it's entirety, so clients need to be careful they do not accidentally delete fields.
|  Level
 
|-
 
|  0
 
|-
 
|  1
 
|-
 
|  2
 
|  Guaranteed to be replicated across two or more storage servers.  
 
|-
 
|  3
 
|  Guaranteed to be replicated across two or more zones within the same data center.  
 
|-
 
|  4
 
|}
 
  
Note that higher durability levels assume the guarantees (if any) of all lower levels.  
+
The body of the request is an arbitrary document which allows storing contextual information about the way a particular application should interact with the queue. The size of this body, in characters and including whitespace must be <= 64 KiB (configurable). The document MUST be valid JSON (Marconi will validate).
  
== Delete a Single Message ==
+
=== Get Queue Metadata ===
  
'''Request Template'''
 
  
 +
'''Template'''
  
 
<pre><nowiki>
 
<pre><nowiki>
DELETE {base_url}/messages/{message_id}{?lock_id}
+
GET /v1/queues/{queue_name}/metadata
 
</nowiki></pre>
 
</nowiki></pre>
  
  
 
'''Request'''
 
'''Request'''
 
  
 
<pre><nowiki>
 
<pre><nowiki>
DELETE /v1/480924/messages/50b68a50d6f5b8c8a7c62b01 HTTP/1.1
+
GET /v1/queues/fizbit/metadata HTTP/1.1
 
Host: marconi.example.com
 
Host: marconi.example.com
  
[...]
+
...
  
 
</nowiki></pre>
 
</nowiki></pre>
Line 565: Line 600:
  
 
'''Response'''
 
'''Response'''
 
+
HTTP/1.1 200 OK
  
 
<pre><nowiki>
 
<pre><nowiki>
HTTP/1.1 204 No Content
+
{
 +
  "key": {
 +
      "key2": "value",
 +
      "key3": [1, 2, 3, 4, 5]
 +
    }
 +
}
 
</nowiki></pre>
 
</nowiki></pre>
  
Line 574: Line 614:
 
'''Discussion'''
 
'''Discussion'''
  
<code><nowiki>message_id</nowiki></code> specifies the message to delete.
+
Returns queue metadata.
  
<code><nowiki>lock_id</nowiki></code> specifies that the message should only be deleted if it has the specified transactional lock, and that lock has not expired. This is useful for ensuring only one agent processes any given message; whenever a worker client's lock expires before it has a chance to delete a message it has processed, the worker must roll back any actions it took based on that message, since another worker will likely process the same message as part of a different transaction.
+
=== Get Queue Stats ===
  
== Delete Several Messages ==
 
 
'''Request Template'''
 
  
 +
'''Template'''
  
 
<pre><nowiki>
 
<pre><nowiki>
DELETE {base_url}/messages{?tags,all,lock_id}
+
GET /v1/queues/{queue_name}/stats
 
</nowiki></pre>
 
</nowiki></pre>
  
  
 
'''Request'''
 
'''Request'''
 
  
 
<pre><nowiki>
 
<pre><nowiki>
DELETE /v1/480924/messages?tags=420D29D6-3F24-11E2-BC14-7823D2B0F3CE HTTP/1.1
+
GET /v1/queues/fizbit/stats HTTP/1.1
 
Host: marconi.example.com
 
Host: marconi.example.com
  
[...]
+
...
  
 
</nowiki></pre>
 
</nowiki></pre>
Line 601: Line 638:
  
 
'''Response'''
 
'''Response'''
 
+
HTTP/1.1 200 OK
  
 
<pre><nowiki>
 
<pre><nowiki>
HTTP/1.1 200 OK
 
 
[...]
 
 
 
{
 
{
   "messageCount": 43
+
   "messages": {
 +
    "free": 146929,
 +
    "claimed": 2409,
 +
    "total": 149338,
 +
    "oldest": {
 +
        "href": "/v1/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b01",
 +
        "age": 63,
 +
        "created": "2013-08-12T20:44:55Z"
 +
    },
 +
    "newest": {
 +
        "href": "/v1/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b01",
 +
        "age": 12,
 +
        "created": "2013-08-12T20:45:46Z"
 +
    }
 
}
 
}
 
</nowiki></pre>
 
</nowiki></pre>
Line 616: Line 662:
 
'''Discussion'''
 
'''Discussion'''
  
<code><nowiki>tags</nowiki></code> specifies that only messages having the given tags will be deleted. To avoid accidentally deleting all messages for a given tenant, if <code><nowiki>tags</nowiki></code> is not specified, <code><nowiki>all</nowiki></code> must equal "true"; otherwise, no messages will be deleted and the server will return 400 Bad Request. Tags are case-sensitive.
+
Returns queue statistics, including how many messages are in the queue, broken out by status.
  
<code><nowiki>all</nowiki></code> is a boolean value (i.e., "true" or "false") that determines whether all messages will be deleted in the case that no tags are specified. If tags ''are'' specified, however, the server simply ignores <code><nowiki>all</nowiki></code>.
+
NOTE: If total is 0, then "oldest" and "newest" message stats are not included.
  
<code><nowiki>lock_id</nowiki></code> specifies that messages should only be deleted if they have the specified transactional lock, and that lock has not expired.
+
== Messages ==
  
The server returns the number of messages deleted.
+
All message-related operations require <code>Client-Id</code> to be included in the headers. This is to ensure that messages are not echoed back to the client that posted them unless the client explicitly requests this.
  
== Lock Messages ==
+
=== List Messages ===
  
'''Request Template'''
 
  
 +
'''Template'''
  
 
<pre><nowiki>
 
<pre><nowiki>
POST {base_url}/messages/locks{?tags,limit}
+
GET /v1/queues/{queue_name}/messages{?marker,limit,echo,include_claimed}
 
</nowiki></pre>
 
</nowiki></pre>
  
Line 636: Line 682:
 
'''Request'''
 
'''Request'''
  
 +
<pre><nowiki>
 +
GET /v1/queues/fizbit/messages?marker=1355-237242-783&limit=10 HTTP/1.1
 +
Host: marconi.example.com
  
<pre><nowiki>
+
...
POST /v1/480924/messages/locks?tags=686897696a7c876b7e&limit=5
 
  
[...]
 
 
 
{
 
  "ttl": 30 
 
}
 
 
</nowiki></pre>
 
</nowiki></pre>
  
Line 650: Line 693:
 
'''Response'''
 
'''Response'''
  
The client receives a lock ID and a list of locked messages, if any:
+
HTTP 200 if the query matched 1 or more messages, HTTP 204 otherwise (with no body).
  
 +
<pre><nowiki>
 +
HTTP/1.1 200 OK
 +
Content-Location: /v1/queues/fizbit/messages?marker=1355-237242-783&limit=10
  
<pre><nowiki>
+
...
@todo
 
  
 
{
 
{
   "lock": {
+
   "links": [
    "id": "9206ebcc0939d18c351e994555de97b9",
+
    {
    "ttl": 30,
+
      "rel": "next",
     "age": 0
+
      "href": "/v1/queues/fizbit/messages?marker=6244-244224-783&limit=10"
   },
+
     }
 +
   ],
 
   "messages": [
 
   "messages": [
     ...
+
     {
 +
      "href": "/v1/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b01",
 +
      "ttl": 800,
 +
      "age": 790,
 +
      "body": {
 +
        "event": "ActivateAccount",
 +
        "mode": "active"
 +
      }
 +
    }
 
   ]
 
   ]
 
}
 
}
Line 671: Line 725:
 
'''Discussion'''
 
'''Discussion'''
  
Locks a set of messages, up to limit, from oldest to newest, skipping any that are already locked.
+
Message IDs and markers are opaque strings; clients should make no assumptions about their format or length. Furthermore, clients should assume there is no relationship between markers and message IDs (i.e., one cannot be derived from the other). This allows for a wide variety of storage driver implementations.
 +
 
 +
Results are ordered by age, oldest message first.
 +
 
 +
<code><nowiki>limit</nowiki></code> specifies up to 20 messages (configurable) to return. If not specified, limit defaults to 10. When more messages are available than can be returned in a single request, the client can pick up the next batch by simply using the URI template parameters returned from the previous call in the "next" field (TBD).
 +
 
 +
<code><nowiki>marker</nowiki></code> is an opaque string that the client can use to request the next batch of messages. It communicates to the server which messages the client has already received.
  
Lock TTL should be <= message TTLs. If it is not, the messages will expire and be garbage collected before the lock expires, which is not what clients will expect. (Should this constraint be enforced server-side?)
+
Note: If <code><nowiki>marker</nowiki></code> is not specified, the API will return all messages at the head of the queue (up to limit).
  
<code><nowiki>limit</nowiki></code> specifies up to x messages to lock. Note that x is configurable, but must be at least 10. If not specified, limit defaults to x.  
+
<code><nowiki>echo</nowiki></code> is a boolean (i.e., "true" or "false") that determines whether or not the API will return a client's own messages, as determined by the <code><nowiki>uuid</nowiki></code> portion of the Client-ID header. If not specified, echo defaults to "false".
  
== Check the Status of a Lock ==
+
<code><nowiki>include_claimed</nowiki></code> is a boolean (i.e., "true" or "false") that determines whether or not the API will return claimed messages as well as unclaimed ones. If not specified, defaults to "false" (only unclaimed messages are returned).
  
'''Request Template'''
+
=== Get a Specific Message ===
  
 +
 +
'''Template'''
  
 
<pre><nowiki>
 
<pre><nowiki>
GET {base_url}/messages/locks/{lock_id}
+
GET /v1/queues/{queue_name}/messages/{message_id}
 
</nowiki></pre>
 
</nowiki></pre>
  
  
 +
'''Request'''
  
 
<pre><nowiki>
 
<pre><nowiki>
GET /v1/480924/messages/locks/9206ebcc0939d18c351e994555de97b9
+
GET /v1/queues/fizbat/messages/50b68a50d6f5b8c8a7c62b01 HTTP/1.1
 +
Host: marconi.example.com
  
[...]
+
...
  
 
</nowiki></pre>
 
</nowiki></pre>
Line 698: Line 762:
 
'''Response'''
 
'''Response'''
  
 +
<pre><nowiki>
 +
HTTP/1.1 200 OK
 +
Content-Location: /v1/queues/fizbat/messages/50b68a50d6f5b8c8a7c62b01
 +
 +
...
  
<pre><nowiki>
 
 
{
 
{
   "id": "9206ebcc0939d18c351e994555de97b9",
+
   "href": "/v1/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b01",
   "ttl": 30,
+
   "ttl": 800,
   "age": 17
+
   "age": 790,
 +
  "body": {
 +
    "event": "ActivateAccount",
 +
    "mode": "active"
 +
  }
 
}
 
}
 
</nowiki></pre>
 
</nowiki></pre>
Line 710: Line 782:
 
'''Discussion'''
 
'''Discussion'''
  
This query may be used to check the age of a given lock (relative to the server's clock).
+
Message fields are defined as follows:
  
== Renew a Lock ==
+
<code><nowiki>href</nowiki></code> is an opaque relative URI that the client can use to uniquely identify a message resource, and interact with it.
  
'''Request Template'''
+
<code><nowiki>ttl</nowiki></code> is the ttl set on the message when it was posted. The message will expire after (ttl - age) seconds.
  
 +
<code><nowiki>age</nowiki></code> number of seconds since ts, relative to the server's clock.
 +
 +
<code><nowiki>body</nowiki></code> Arbitrary document submitted along with the original request to post the message.
 +
 +
If either the message ID or the claim ID is malformed or nonexistent, no message is returned.
 +
 +
=== Get a Set of Messages by ID ===
 +
 +
 +
'''Template'''
  
 
<pre><nowiki>
 
<pre><nowiki>
PATCH {base_url}/messages/locks/9206ebcc0939d18c351e994555de97b9
+
GET /v1/queues/{queue_name}/messages{?ids}
 
</nowiki></pre>
 
</nowiki></pre>
  
Line 724: Line 806:
 
'''Request'''
 
'''Request'''
  
 +
<pre><nowiki>
 +
GET /v1/queues/fizbat/messages?ids=50b68a50d6f5b8c8a7c62b01,f5b8c8a7c62b0150b68a50d6 HTTP/1.1
 +
Host: marconi.example.com
  
<pre><nowiki>
+
...
PATCH /v1/480924/messages/locks/9206ebcc0939d18c351e994555de97b9
 
  
[...]
 
 
 
[
 
  {
 
    "replace": "/age",
 
    "value": 0
 
  }
 
]
 
 
</nowiki></pre>
 
</nowiki></pre>
  
Line 741: Line 817:
 
'''Response'''
 
'''Response'''
  
 +
<pre><nowiki>
 +
HTTP/1.1 200 OK
 +
Content-Location: /v1/queues/fizbat/messages?ids=50b68a50d6f5b8c8a7c62b01,f5b8c8a7c62b0150b68a50d6
  
<pre><nowiki>
+
...
HTTP/1.1 204 No Content
+
 
 +
[
 +
    {
 +
      "href": "/v1/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b01",
 +
      "ttl": 800,
 +
      "age": 32,
 +
      "body": {
 +
        "cmd": "EncodeVideo",
 +
        "jobid": 58229
 +
      }
 +
    },
 +
    {
 +
      "href": "/v1/queues/fizbit/messages/f5b8c8a7c62b0150b68a50d6",
 +
      "ttl": 800,
 +
      "age": 790,
 +
      "body": {
 +
        "cmd": "EncodeAudio",
 +
        "jobid": 58201
 +
      }
 +
    }
 +
]
 
</nowiki></pre>
 
</nowiki></pre>
  
Line 749: Line 848:
 
'''Discussion'''
 
'''Discussion'''
  
Clients should periodically renew locks during long-running batches of work to avoid loosing a lock in the middle of processing a message. The server only accepts "0" for the age value.
+
Unlike message listing, a client's own messages are always returned in this approach. Note that the list of ids may not exceed 20 (configurable, shared setting with number of messages that can be posted at once).
  
== Count Messages ==
+
This approach returns all the messages with matching IDs. If a malformed ID or a nonexistent message ID is provided, it is ignored and the remaining messages are returned.
  
'''Request Template'''
+
=== Post Message(s) ===
  
 +
 +
'''Template'''
  
 
<pre><nowiki>
 
<pre><nowiki>
HEAD /{version}/{tenant}/messages{?tags,ts,meta,echo}
+
POST /v1/queues/{queue_name}/messages
 
</nowiki></pre>
 
</nowiki></pre>
  
  
 
'''Request'''
 
'''Request'''
 
  
 
<pre><nowiki>
 
<pre><nowiki>
HEAD /v1/480924/messages?tags=foo,bar,bang&ts=1355237242783 HTTP/1.1
+
POST /v1/queues/fizbit/messages HTTP/1.1
 
Host: marconi.example.com
 
Host: marconi.example.com
  
[...]
+
...
  
 +
[
 +
  {
 +
    "ttl": 300,
 +
    "body": {
 +
      "event": "BackupStarted",
 +
      "backup_id": "c378813c-3f0b-11e2-ad92-7823d2b0f3ce"
 +
    }
 +
  },
 +
  {
 +
    "ttl": 60,
 +
    "body": {
 +
      "event": "BackupProgress",
 +
      "current_bytes": "0",
 +
      "total_bytes": "99614720"
 +
    }
 +
  }
 +
]
 
</nowiki></pre>
 
</nowiki></pre>
  
  
'''Response'''
+
'''Responses'''
HTTP/1.1 200 OK
 
  
 +
When a single message is submitted:
  
 
<pre><nowiki>
 
<pre><nowiki>
 +
HTTP/1.1 201 Created
 +
Location: /v1/queues/fizbit/messages?ids=50b68a50d6f5b8c8a7c62b01
 +
Content-Type: application/json
 +
 
{
 
{
  "messageCount": 46
+
    "resources": [
 +
        "/v1/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b01"
 +
    ],
 +
    "partial": false
 
}
 
}
 
</nowiki></pre>
 
</nowiki></pre>
  
 +
When multiple messages are submitted:
 +
 +
<pre><nowiki>
 +
HTTP/1.1 201 Created
 +
Location: /v1/queues/fizbit/messages?ids=50b68a50d6f5b8c8a7c62b01,50b68a50d6f5b8c8a7c62b05
 +
Content-Type: application/json
 +
 +
{
 +
    "resources": [
 +
        "/v1/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b01",
 +
        "/v1/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b05"
 +
    ],
 +
    "partial": false
 +
}
 +
</nowiki></pre>
  
 
'''Discussion'''
 
'''Discussion'''
  
See ''Get Messages'' for definitions of the query parameters.
+
Up to 20 messages (default, but configurable) may be submitted in a single request, but must always be encapsulated in a collection container (an array in JSON). The resulting value of the Location header or response body may be used to retrieve the created messages for further processing if needed.
 +
 
 +
The client only specifies the body and ttl for the message; the server will insert metadata such as id and age.
 +
 
 +
The response body contains a list of resource paths corresponding to each message submitted in the request, in the same order. In the case of a server-side error part-way through the processing of the submitted messages, a partial list will be returned, <code><nowiki>partial</nowiki></code> will be '''true''', and the client will need to retry posting the remaining messages. If no messages could be enqueued, the server will return 503 Service Unavailable.
  
== Check Health ==
+
NOTE: The ''partial'' field has been deprecated since the icehouse release, and will always be "false"; drivers are not longer allowed to return a "partial" success and so must either succeed or fail the entire batch of messages.
  
'''Request Template'''
+
The size of the request document, in characters including whitespace, is 256 KiB by default (configurable). The document MUST be valid JSON (Marconi will validate).
  
 +
<code><nowiki>body</nowiki></code> specifies an arbitrary document which constitutes the body of the message being sent.
  
<pre><nowiki>
+
<code><nowiki>ttl</nowiki></code> is how long the server should wait before expiring and removing the message from the queue. Value MUST be between 60 and 1209600 seconds (14 days, configurable), inclusive. Note that the server may not actually delete the message until it's age has reached up to (ttl + 60) seconds, to allow for flexibility in storage implementations.
GET {base_url}/health
 
</nowiki></pre>
 
  
 +
=== Delete Message ===
  
or
 
  
 +
'''Template'''
  
 
<pre><nowiki>
 
<pre><nowiki>
HEAD {base_url}/health
+
DELETE /v1/queues/{queue_name}/messages/{message_id}{?claim_id}
 
</nowiki></pre>
 
</nowiki></pre>
  
  
 
'''Request'''
 
'''Request'''
 
  
 
<pre><nowiki>
 
<pre><nowiki>
GET /v1/480924/health
+
DELETE /v1/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b01 HTTP/1.1
Host: example.marconi.com
+
Host: marconi.example.com
  
[...]
+
...
  
 
</nowiki></pre>
 
</nowiki></pre>
Line 820: Line 962:
 
'''Response'''
 
'''Response'''
  
 +
<pre><nowiki>
 +
HTTP/1.1 204 No Content
 +
</nowiki></pre>
 +
 +
 +
'''Discussion'''
 +
 +
<code><nowiki>message_id</nowiki></code> specifies the message to delete.
 +
 +
<code><nowiki>claim_id</nowiki></code> specifies that the message should only be deleted if it has the specified claim ID, and that claim has not expired. This is useful for ensuring only one agent processes any given message; whenever a worker client's claim expires before it has a chance to delete a message it has processed, the worker must roll back any actions it took based on that message, since another worker will now be able to claim and process the same message.
 +
 +
Note that if claim_id is not specified, but the message is claimed, the operation will fail. In other words, claimed messages can only ever be deleted by providing an appropriate claim_id.
 +
 +
=== Delete a Set of Messages by ID ===
 +
 +
'''Template'''
  
 
<pre><nowiki>
 
<pre><nowiki>
HTTP/1.1 200 OK
+
DELETE /v1/queues/{queue_name}/messages{?ids}
 +
</nowiki></pre>
 +
 
 +
 
 +
'''Request'''
 +
 
 +
<pre><nowiki>
 +
DELETE /v1/queues/fizbit/messages?ids=50b68a50d6f5b8c8a7c62b01,50b68a50d6f5b8c8a7c62b02 HTTP/1.1
 +
Host: marconi.example.com
  
[...]
+
...
  
{
 
  "code": "green",
 
  "link": {
 
    "rel": "status",
 
    "href": "http://marconi.example.com/status",
 
    "text": "Service status page"
 
  }
 
}
 
 
</nowiki></pre>
 
</nowiki></pre>
  
  
'''Discussion'''
+
'''Response'''
 +
 
 +
<pre><nowiki>
 +
HTTP/1.1 204 No Content
 +
</nowiki></pre>
  
Use this request to check on the Marconi service status as a whole. The following status values are defined:
 
  
{| border="1" cellpadding="2" cellspacing="0"
+
'''Discussion'''
|  Code
 
|-
 
|  green 
 
|-
 
|  yellow
 
|-
 
|  red
 
|  Degraded availability
 
|}
 
  
Note that if a client performs HEAD instead of GET, the server will return one of the following:
+
Bulk delete for messages.
  
{| border="1" cellpadding="2" cellspacing="0"
+
<code><nowiki>ids</nowiki></code> specifies the messages to delete, up to a maximum of 20 (configurable, shares the same setting as posting and listing messages)
|  Status
 
|-
 
|  HTTP/1.1 204 No Content
 
|-
 
|  HTTP/1.1 503 Service Unavailable
 
|}
 
  
== Poll Stats ==
+
If any of the message IDs are malformed or non-existent, they are ignored. The remaining valid message IDs are deleted.
  
@todo - also how authenticate?
+
== Claims ==
  
== Set Message Defaults ==
+
=== Claim Messages ===
  
'''Request Template'''
 
  
 +
'''Template'''
  
 
<pre><nowiki>
 
<pre><nowiki>
PUT {base_url}/config
+
POST /v1/queues/{queue_name}/claims{?limit}
 +
Content-Type: application/json
  
{
+
...
  "message": {
+
 
     "ttl": {ttl},
+
{  
     "durability": {durability},
+
     "ttl": {claim_ttl},
    "lock": {
+
     "grace": {message_grace}
      "ttl": {lock_ttl}
 
    }
 
  }
 
 
}
 
}
 
</nowiki></pre>
 
</nowiki></pre>
Line 889: Line 1,033:
  
 
<pre><nowiki>
 
<pre><nowiki>
PUT /v1/480924/config HTTP/1.1
+
POST /v1/queues/fizbit/claims?limit=5 HTTP/1.1
Host: marconi.example.com
+
Content-Type: application/json
 +
Accept: application/json
  
[...]
+
...
  
{
+
{  
  "message": {
+
     "ttl": 300,
     "ttl": 120,
+
     "grace": 300
    "durability": 1,
 
     "lock": {
 
      "ttl": 60
 
    }
 
  }
 
 
}
 
}
 
</nowiki></pre>
 
</nowiki></pre>
Line 908: Line 1,048:
 
'''Response'''
 
'''Response'''
  
 +
The client receives a claim URI and a list of claimed messages, if any:
  
 
<pre><nowiki>
 
<pre><nowiki>
HTTP/1.1 204 No Content
+
HTTP/1.1 201 Created
 +
Content-Type: application/json
 +
Location: /v1/queues/foo-bar/claims/a28ee94e-6cb4-11e2-b4d5-7703267a7926
 +
 
 +
...
 +
 
 +
[
 +
  {
 +
    "href": "/v1/queues/foo-bar/messages/50b68a50d6f5b8c8a7c62b01?claim_id=a28ee94e-6cb4-11e2-b4d5-7703267a7926",
 +
    "ttl": 800,
 +
    "age": 100,
 +
    "body": {
 +
      "object_id": "8a50d6",
 +
      "target": "h.264"
 +
    }
 +
  },
 +
  {
 +
    "href": "/v1/queues/foo-bar/messages/50b68a50d6f5b8c8a7c62b02?claim_id=a28ee94e-6cb4-11e2-b4d5-7703267a7926",
 +
    "ttl": 800,
 +
    "age": 790,
 +
    "body": {
 +
      "object_id": "fb8c8a",
 +
      "target": "h.264"
 +
    }
 +
  }
 +
]
 
</nowiki></pre>
 
</nowiki></pre>
  
Line 916: Line 1,082:
 
'''Discussion'''
 
'''Discussion'''
  
Set per-tenant default message options, such as ttl and durability. Lock TTL must be <= message TTL.
+
Claims a set of messages, up to limit, from oldest to newest, skipping any that are already claimed. If no unclaimed messages are available, returns 204 No Content.
  
== Get Message Defaults ==
+
The client should delete the message when it has finished processing it, before the claim expires, to ensure the message is processed only once. As part of the delete operation, all worker clients should specify the claim ID (this is best done by simply using the provided href). That way, the server can return an error if the claim just expired (notifying the client of the race condition), giving the worker a chance to roll back its own processing of the given message, since another worker will likely claim the message and process it.
  
'''Request Template'''
+
Just as with a message's age, the age given for the claim is relative to the server's clock, and is useful for determining how quickly messages are getting processed, and whether a given message's claim is about to expire.
  
 +
When a claim expires, it is removed, allowing another client worker to claim the message in the case that the original worker fails to process it.
 +
 +
<code><nowiki>limit</nowiki></code> specifies up to 20 messages (configurable) to claim. If not specified, limit defaults to 10. Note that claim creation is best-effort, meaning the server may claim and return less than the requested number of messages.
 +
 +
<code><nowiki>ttl</nowiki></code> is how long the server should wait before releasing the claim. Value MUST be between 60 and 43200 seconds (12 hours, configurable).
 +
 +
<code><nowiki>grace</nowiki></code> is the message grace period in seconds. Value MUST be between 60 and 43200 seconds (12 hours, configurable). The server will extend the lifetime of claimed messages to be at least as long as the lifetime of the claim itself, plus a specified grace period to deal with crashed workers (up to 1209600 or 14 days including claim lifetime). If a claimed message would normally live longer than the grace period, it's expiration will not be adjusted.
 +
 +
=== Query Claim ===
 +
 +
 +
'''Template'''
  
 
<pre><nowiki>
 
<pre><nowiki>
GET {base_url}/config
+
GET /v1/queues/{queue_name}/claims/{claim_id}
 
</nowiki></pre>
 
</nowiki></pre>
  
Line 931: Line 1,109:
  
 
<pre><nowiki>
 
<pre><nowiki>
GET /v1/480924/config HTTP/1.1
+
GET /v1/queues/foo-bar/claims/a28ee94e-6cb4-11e2-b4d5-7703267a7926
 
Host: marconi.example.com
 
Host: marconi.example.com
  
[...]
+
...
  
 
</nowiki></pre>
 
</nowiki></pre>
Line 940: Line 1,118:
  
 
'''Response'''
 
'''Response'''
 
  
 
<pre><nowiki>
 
<pre><nowiki>
 
HTTP/1.1 200 OK
 
HTTP/1.1 200 OK
 +
Content-Location: /v1/queues/foo-bar/claims/a28ee94e-6cb4-11e2-b4d5-7703267a7926
  
[...]
+
...
  
 
{
 
{
   "message": {
+
   "age": 19,
    "ttl": 120,
+
  "ttl": 30,
    "durability": 1,
+
  "messages": [
    "lock": {
+
     ...
      "ttl": 60
+
   ]
     }
 
   }
 
 
}
 
}
 
</nowiki></pre>
 
</nowiki></pre>
Line 961: Line 1,137:
 
'''Discussion'''
 
'''Discussion'''
  
Get per-tenant default message options, such as ttl and durability. If not set explicitly by a client, a default configuration is used, and can vary between API deployments.
+
=== Update Claim ===
  
== TBD ==
 
  
'''Request Template'''
+
'''Template'''
  
 +
<pre><nowiki>
 +
PATCH /v1/queues/{queue_name}/claims/{claim_id}
 +
Content-Type: application/json
 +
</nowiki></pre>
 +
 +
 +
'''Request'''
  
 
<pre><nowiki>
 
<pre><nowiki>
GET /{version}/{tenant}
+
PATCH /v1/queues/fizbit/claims/a28ee94e-6cb4-11e2-b4d5-7703267a7926 HTTP/1.1
 +
Content-Type: application/json
 +
 
 +
...
 +
 
 +
{ "ttl": 300 }
 
</nowiki></pre>
 
</nowiki></pre>
  
Line 975: Line 1,162:
 
'''Response'''
 
'''Response'''
  
 +
<pre><nowiki>
 +
HTTP/1.1 204 No Content
 +
</nowiki></pre>
 +
 +
 +
'''Discussion'''
 +
 +
Clients should periodically renew claims during long-running batches of work to avoid loosing a claim in the middle of processing a message. This is done by issuing a PATCH to a specific claim resource and including a new TTL for the claim (which may be different from the original TTL). The server will reset the age of the claim and apply the new TTL.
 +
 +
=== Release Claim ===
 +
 +
 +
'''Template'''
 +
 +
<pre><nowiki>
 +
DELETE /v1/queues/{queue_name}/claims/{claim_id}
 +
</nowiki></pre>
 +
 +
 +
'''Request'''
  
 
<pre><nowiki>
 
<pre><nowiki>
@todo
+
DELETE /v1/queues/foo-bar/claims/a28ee94e-6cb4-11e2-b4d5-7703267a7926 HTTP/1.1
 +
Host: marconi.example.com
 +
</nowiki></pre>
 +
 
 +
 
 +
'''Response'''
 +
 
 +
<pre><nowiki>
 +
HTTP/1.1 204 No Content
 
</nowiki></pre>
 
</nowiki></pre>
  
Line 983: Line 1,198:
 
'''Discussion'''
 
'''Discussion'''
  
@todo
+
Use this operation to immediately release a claim, making any (remaining, non-deleted) messages associated with the claim available to other workers.
 +
 
 +
This operation is useful when a worker is performing a graceful shutdown, fails to process one or more messages, or is taking longer than expected to process messages, and wishes to make the remainder available to other workers.

Latest revision as of 19:06, 15 September 2014

Marconi API: v1 Blueprint

The Marconi v1 API is FROZEN

We have started collecting feedback with an eye toward extensions and v2 of the API here: Marconi/specs/api/next


Overview

Marconi provides an HTTP-based API in the spirit of the REST architecture style.

This guide assumes the reader is familiar with REST, HTTP/1.1 and JSON. URI templates use the same syntax as RFC 6570.

Note: Error responses are enumerated on a separate page: Marconi/specs/api/v1/errors


Common API Elements

Clients

  • Clients should follow HTTP redirects
  • Clients should advertise gzip support
  • Clients must identify themselves in the UserAgent request header; e.g., "User-Agent: python/2.7 cloudthing/1.2"
  • Clients must not hard-code URI paths or templates since they may change over time. Instead, clients should cache links and link templates provided by the API.


API Versioning

The Marconi API uses a URI versioning scheme. The first element of the path contains the target version identifier, e.g.:

  https://marconi.example.com/v1

The URI version will only be incremented to accommodate major new features or API redesigns that can not be made backwards-compatible. When new API versions are released, older versions are deprecated. Marconi maintainers will work with developers and partners to ensure there is adequate time to migrate to new versions before deprecated ones are discontinued.

Since the version is only incremented to accommodate major API revisions, sub-versioning of the API will be rare, but may be used occasionally to release a "polished" major version.

Resource Media Types

The API supports `application/json` (XML is on the road map), encoded as UTF-8. Gzip'd requests are optionally supported by the server.

Unrecognized protocol elements should be ignored. This includes XML elements and attributes, JSON object properties, link relation types, media types, etc.


Authentication

May be optionally implemented by middleware or reverse proxy. If implemented, the following applies:

All requests to the API may only be performed by an authenticated agent.

To authenticate, an agent issues an authentication request to a Keystone Identity Service endpoint.

In response to valid credentials, Keystone responds with an auth token and a service catalog that contains a list of all services and endpoints available for the given token. Multiple endpoints may be returned for Marconi according to physical locations and performance/availability characteristics of different deployments.

Normally, Keystone middleware provides the X-Project-Id header based on the auth token submitted by the Marconi client. For this to work, clients MUST specify a valid auth token in the `X-Auth-Token` header for each request to the Marconi API. The API validates auth tokens against Keystone before servicing each request

If auth is not enabled, clients must provide the X-Project-Id header themselves.

Authorization

May be optionally implemented by middleware or reverse proxy. If implemented, the following applies:

The API needs to verify read/write access to all endpoints according to the provided auth token (RBAC). The ACL should be cached with the token.


Endpoints

Clients may choose to interact with any of the service endpoints retrieved from Keystone. Agents use the given endpoint as a home document href from which to discover links to all other API requests. Endpoints may be organized by region

An example endpoint:

https://marconi.example.com/v1

Errors

If any request results in an error, the server will return an appropriate 4xx or 5xx HTTP status code, and optionally the following information in the body:

  • Title
  • Description
  • Internal code
  • Link to more information


Example:

HTTP/1.1 400 Bad Request
Content-Type: application/json

{
  "title": "Unsupported limit",
  "description": "The given limit cannot be negative, and cannot be greater than 50.",
  "code": 1092,
  "link": {
    "rel": "help",
    "href": "http://docs.example.com/messages#limit",
    "text": "API documentation for the limit parameter"
  }
}

Common Headers

Each request to the API must include certain standard and extended HTTP headers. These headers provide host, agent, authentication and other pertinent information to the server.

Header Description
Host The host name of the API, as referenced by the Keystone service catalog
Date The current date and time, using the standard RFC 1123 HTTP date format
Accept Media type desired; only application/json is supported at this time.
Accept-Encoding Specifies that the agent accepts gzip-encoded response bodies
Content-Length For POST or PUT requests, the length in bytes of the message document being submitted
X-Auth-Token Keystone auth token
X-Project-Id An ID for a project to which the value of X-Auth-Token grants access. Queues will be created under this project.
Client-ID A UUID, used to distinguish each client using the service. The UUID must be submitted in its canonical form (e.g., 3381af92-2b9e-11e3-b191-71861300734c). In Marconi, this is used to avoid echoing a sender's messages back to the same instance, and may be logged by the server for future use. Should be generated once and persisted between restarts of the client.

HTTP Response Codes

See Response Codes

Sample API Request

GET /v1/queues/fizbat/messages?marker=1355-237242-783&limit=10 HTTP/1.1
Host: marconi.example.com
User-Agent: python/2.7 killer-rabbit/1.2
Date: Wed, 28 Nov 2012 21:14:19 GMT
Accept: application/json
Accept-Encoding: gzip
X-Auth-Token: 7d2f63fd-4dcc-4752-8e9b-1d08f989cc00
X-Project-Id: 518b51ea133c4facadae42c328d6b77b
Client-ID: 30387f00-39a0-11e2-be4d-a8d15f34bae2

Endpoints Synopsis

# Base endpoints
GET /v1
GET /v1/health

# Queues
GET /v1/queues{?marker,limit,detailed}
GET /v1/queues/{queue_name}  # existence check
HEAD /v1/queues/{queue_name}  # existence check
PUT /v1/queues/{name}
DELETE /v1/queues/{name}

# Queue metadata
PUT /v1/queues/{queue_name}/metadata
GET /v1/queues/{queue_name}/metadata
GET /v1/queues/{queue_name}/stats

# Messages
GET /v1/queues/{queue_name}/messages{?marker,limit,echo,include_claimed}
GET /v1/queues/{queue_name}/messages/{message_id}
GET /v1/queues/{queue_name}/messages{?ids}
POST /v1/queues/{queue_name}/messages
DELETE /v1/queues/{queue_name}/messages/{message_id}{?claim_id}
DELETE /v1/queues/{queue_name}/messages{?ids}

# Claims
POST /v1/queues/{queue_name}/claims{?limit}
GET /v1/queues/{queue_name}/claims/{claim_id}
PATCH /v1/queues/{queue_name}/claims/{claim_id}
DELETE /v1/queues/{queue_name}/claims/{claim_id}

Base Endpoints

Get Home Document

Template

GET /v1


Request

GET /v1 HTTP/1.1
Host: marconi.example.com

...


Response

HTTP/1.0 200 OK

...

{
    "resources": {
        "rel/queue-stats": {
            "href-template": "/v1/queues/{queue_name}/stats", 
            "href-vars": {
                "queue_name": "param/queue_name"
            }, 
            "hints": {
                "allow": [
                    "GET"
                ], 
                "formats": {
                    "application/json": {}
                }
            }
        }, 
        "rel/post-messages": {
            "href-template": "/v1/queues/{queue_name}/messages", 
            "href-vars": {
                "queue_name": "param/queue_name"
            }, 
            "hints": {
                "accept-post": [
                    "application/json"
                ], 
                "allow": [
                    "POST"
                ], 
                "formats": {
                    "application/json": {}
                }
            }
        }, 
        "rel/queue": {
            "href-template": "/v1/queues/{queue_name}", 
            "href-vars": {
                "queue_name": "param/queue_name"
            }, 
            "hints": {
                "allow": [
                    "GET", 
                    "HEAD", 
                    "PUT", 
                    "DELETE"
                ], 
                "formats": {
                    "application/json": {}
                }
            }
        }, 
        "rel/queue-metadata": {
            "href-template": "/v1/queues/{queue_name}/metadata", 
            "href-vars": {
                "queue_name": "param/queue_name"
            }, 
            "hints": {
                "allow": [
                    "GET", 
                    "PUT"
                ], 
                "formats": {
                    "application/json": {}
                }
            }
        }, 
        "rel/queues": {
            "href-template": "/v1/queues{?marker,limit,detailed}", 
            "href-vars": {
                "marker": "param/marker", 
                "detailed": "param/detailed", 
                "limit": "param/queue_limit"
            }, 
            "hints": {
                "allow": [
                    "GET"
                ], 
                "formats": {
                    "application/json": {}
                }
            }
        }, 
        "rel/messages": {
            "href-template": "/v1/queues/{queue_name}/messages{?marker,limit,echo,include_claimed}", 
            "href-vars": {
                "marker": "param/marker", 
                "include_claimed": "param/include_claimed", 
                "queue_name": "param/queue_name", 
                "limit": "param/messages_limit", 
                "echo": "param/echo"
            }, 
            "hints": {
                "allow": [
                    "GET"
                ], 
                "formats": {
                    "application/json": {}
                }
            }
        }, 
        "rel/claim": {
            "href-template": "/v1/queues/{queue_name}/claims{?limit}", 
            "href-vars": {
                "queue_name": "param/queue_name", 
                "limit": "param/claim_limit"
            }, 
            "hints": {
                "accept-post": [
                    "application/json"
                ], 
                "allow": [
                    "POST"
                ], 
                "formats": {
                    "application/json": {}
                }
            }
        }
    }
}


Discussion

The entire Marconi API is discoverable from a single starting point; agents do not need to know any more than this one URI in order to explore the entire API.

This document is cache-able (and should be cached by proxy and client).

See also: http://tools.ietf.org/html/draft-nottingham-json-home-02

Check Node Health

Template

GET /v1/health

or

HEAD /v1/health


Request

GET /v1/health HTTP/1.1
Host: example.marconi.com

...


Response

HTTP/1.1 204 No Content

or

HTTP/1.1 503 Service Unavailable


Discussion

Use this request to check on the Marconi node status. If the node is down, this endpoint will not be reachable and will act as a signal to the load balancer to take the node out of rotation. Alternatively, the endpoint may be reachable but the service is temporarily unavailable due to storage driver failure or some other error.

Queues

Create Queue

Template

PUT /v1/queues/{queue_name}

Request

PUT /v1/queues/fizbat HTTP/1.1
Host: marconi.example.com


Response

HTTP/1.1 201 Created
Location: /v1/queues/fizbit


Discussion

Creates a queue.

The body of the PUT is empty.

queue_name is the name to give the queue. The name MUST NOT exceed 64 bytes in length, and is limited to US-ASCII letters, digits, underscores and hyphens.

List Queues

Template

GET /v1/queues{?marker,limit,detailed}


Request

GET /v1/queues?marker=baz&detailed=true HTTP/1.1
Host: marconi.example.com

...


Response

HTTP/1.1 200 OK

{
  "links": [
    {
      "rel": "next",
      "href": "/v1/queues?marker=kooleo&limit=10&detailed=true"
    }
  ],
  "queues": [
    { "name": "boomerang", "href": "/v1/queues/boomerang", "metadata": {} },
    { "name": "fizbit", "href": "/v1/queues/fizbit", "metadata": { "handle": "@kgriffs" } },

    ...

    { "name": "kooleo",  "href": "/v1/queues/kooleo", "metadata": { "something": "something_else" } }
  ]
}

Checking Queue Existence

Template

GET /v1/queues/{queue_name}


Request

GET /v1/queues/{queue_name}
Host: marconi.example.com


Response

HTTP/1.1 204 OK


Discussion

Returns 204 if the queue exists, else 404.

HEAD also works.

Delete Queue

Template

DELETE /v1/queues/{queue_name}


Request

DELETE /v1/queues/fizbat HTTP/1.1
Host: marconi.example.com


Response

HTTP/1.1 204 No Content


Discussion

Use this operation to immediately delete a queue along with all its messages (if any).

Queue Metadata

Set Queue Metadata

Template

PUT /v1/queues/{queue_name}/metadata

{
   ...
}


Request

PUT /v1/queues/fizbat/metadata HTTP/1.1
Host: marconi.example.com

...

{
   "key": {
       "key2": "value",
       "key3": [1, 2, 3, 4, 5]
    }
}


Response

HTTP/1.1 204 No Content


Discussion

This operation allows for setting structured metadata. If PUT is used, this document will replace the existing metadata document in it's entirety, so clients need to be careful they do not accidentally delete fields.

The body of the request is an arbitrary document which allows storing contextual information about the way a particular application should interact with the queue. The size of this body, in characters and including whitespace must be <= 64 KiB (configurable). The document MUST be valid JSON (Marconi will validate).

Get Queue Metadata

Template

GET /v1/queues/{queue_name}/metadata


Request

GET /v1/queues/fizbit/metadata HTTP/1.1
Host: marconi.example.com

...


Response HTTP/1.1 200 OK

{
   "key": {
       "key2": "value",
       "key3": [1, 2, 3, 4, 5]
    }
}


Discussion

Returns queue metadata.

Get Queue Stats

Template

GET /v1/queues/{queue_name}/stats


Request

GET /v1/queues/fizbit/stats HTTP/1.1
Host: marconi.example.com

...


Response HTTP/1.1 200 OK

{
  "messages": {
    "free": 146929,
    "claimed": 2409,
    "total": 149338,
    "oldest": {
        "href": "/v1/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b01",
        "age": 63,
        "created": "2013-08-12T20:44:55Z"
    },
    "newest": {
        "href": "/v1/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b01",
        "age": 12,
        "created": "2013-08-12T20:45:46Z"
    }
}


Discussion

Returns queue statistics, including how many messages are in the queue, broken out by status.

NOTE: If total is 0, then "oldest" and "newest" message stats are not included.

Messages

All message-related operations require Client-Id to be included in the headers. This is to ensure that messages are not echoed back to the client that posted them unless the client explicitly requests this.

List Messages

Template

GET /v1/queues/{queue_name}/messages{?marker,limit,echo,include_claimed}


Request

GET /v1/queues/fizbit/messages?marker=1355-237242-783&limit=10 HTTP/1.1
Host: marconi.example.com

...


Response

HTTP 200 if the query matched 1 or more messages, HTTP 204 otherwise (with no body).

HTTP/1.1 200 OK
Content-Location: /v1/queues/fizbit/messages?marker=1355-237242-783&limit=10

...

{
  "links": [
    {
      "rel": "next",
      "href": "/v1/queues/fizbit/messages?marker=6244-244224-783&limit=10"
    }
  ],
  "messages": [
    {
      "href": "/v1/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b01",
      "ttl": 800,
      "age": 790,
      "body": {
        "event": "ActivateAccount",
        "mode": "active"
      }
    }
  ]
}


Discussion

Message IDs and markers are opaque strings; clients should make no assumptions about their format or length. Furthermore, clients should assume there is no relationship between markers and message IDs (i.e., one cannot be derived from the other). This allows for a wide variety of storage driver implementations.

Results are ordered by age, oldest message first.

limit specifies up to 20 messages (configurable) to return. If not specified, limit defaults to 10. When more messages are available than can be returned in a single request, the client can pick up the next batch by simply using the URI template parameters returned from the previous call in the "next" field (TBD).

marker is an opaque string that the client can use to request the next batch of messages. It communicates to the server which messages the client has already received.

Note: If marker is not specified, the API will return all messages at the head of the queue (up to limit).

echo is a boolean (i.e., "true" or "false") that determines whether or not the API will return a client's own messages, as determined by the uuid portion of the Client-ID header. If not specified, echo defaults to "false".

include_claimed is a boolean (i.e., "true" or "false") that determines whether or not the API will return claimed messages as well as unclaimed ones. If not specified, defaults to "false" (only unclaimed messages are returned).

Get a Specific Message

Template

GET /v1/queues/{queue_name}/messages/{message_id}


Request

GET /v1/queues/fizbat/messages/50b68a50d6f5b8c8a7c62b01 HTTP/1.1
Host: marconi.example.com

...


Response

HTTP/1.1 200 OK
Content-Location: /v1/queues/fizbat/messages/50b68a50d6f5b8c8a7c62b01

...

{
  "href": "/v1/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b01",
  "ttl": 800,
  "age": 790,
  "body": {
    "event": "ActivateAccount",
    "mode": "active"
  }
}


Discussion

Message fields are defined as follows:

href is an opaque relative URI that the client can use to uniquely identify a message resource, and interact with it.

ttl is the ttl set on the message when it was posted. The message will expire after (ttl - age) seconds.

age number of seconds since ts, relative to the server's clock.

body Arbitrary document submitted along with the original request to post the message.

If either the message ID or the claim ID is malformed or nonexistent, no message is returned.

Get a Set of Messages by ID

Template

GET /v1/queues/{queue_name}/messages{?ids}


Request

GET /v1/queues/fizbat/messages?ids=50b68a50d6f5b8c8a7c62b01,f5b8c8a7c62b0150b68a50d6 HTTP/1.1
Host: marconi.example.com

...


Response

HTTP/1.1 200 OK
Content-Location: /v1/queues/fizbat/messages?ids=50b68a50d6f5b8c8a7c62b01,f5b8c8a7c62b0150b68a50d6

...

[
    {
      "href": "/v1/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b01",
      "ttl": 800,
      "age": 32,
      "body": {
        "cmd": "EncodeVideo",
        "jobid": 58229 
      }
    },
    {
      "href": "/v1/queues/fizbit/messages/f5b8c8a7c62b0150b68a50d6",
      "ttl": 800,
      "age": 790,
      "body": {
        "cmd": "EncodeAudio",
        "jobid": 58201 
      }
    }
]


Discussion

Unlike message listing, a client's own messages are always returned in this approach. Note that the list of ids may not exceed 20 (configurable, shared setting with number of messages that can be posted at once).

This approach returns all the messages with matching IDs. If a malformed ID or a nonexistent message ID is provided, it is ignored and the remaining messages are returned.

Post Message(s)

Template

POST /v1/queues/{queue_name}/messages


Request

POST /v1/queues/fizbit/messages HTTP/1.1
Host: marconi.example.com

...

[
  {
    "ttl": 300,
    "body": {
      "event": "BackupStarted",
      "backup_id": "c378813c-3f0b-11e2-ad92-7823d2b0f3ce"
    }
  },
  {
    "ttl": 60,
    "body": {
      "event": "BackupProgress",
      "current_bytes": "0",
      "total_bytes": "99614720"
    }
  }
]


Responses

When a single message is submitted:

HTTP/1.1 201 Created
Location: /v1/queues/fizbit/messages?ids=50b68a50d6f5b8c8a7c62b01
Content-Type: application/json

{
    "resources": [
        "/v1/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b01"
    ],
    "partial": false
}

When multiple messages are submitted:

HTTP/1.1 201 Created
Location: /v1/queues/fizbit/messages?ids=50b68a50d6f5b8c8a7c62b01,50b68a50d6f5b8c8a7c62b05
Content-Type: application/json

{
    "resources": [
        "/v1/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b01",
        "/v1/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b05"
    ],
    "partial": false
}

Discussion

Up to 20 messages (default, but configurable) may be submitted in a single request, but must always be encapsulated in a collection container (an array in JSON). The resulting value of the Location header or response body may be used to retrieve the created messages for further processing if needed.

The client only specifies the body and ttl for the message; the server will insert metadata such as id and age.

The response body contains a list of resource paths corresponding to each message submitted in the request, in the same order. In the case of a server-side error part-way through the processing of the submitted messages, a partial list will be returned, partial will be true, and the client will need to retry posting the remaining messages. If no messages could be enqueued, the server will return 503 Service Unavailable.

NOTE: The partial field has been deprecated since the icehouse release, and will always be "false"; drivers are not longer allowed to return a "partial" success and so must either succeed or fail the entire batch of messages.

The size of the request document, in characters including whitespace, is 256 KiB by default (configurable). The document MUST be valid JSON (Marconi will validate).

body specifies an arbitrary document which constitutes the body of the message being sent.

ttl is how long the server should wait before expiring and removing the message from the queue. Value MUST be between 60 and 1209600 seconds (14 days, configurable), inclusive. Note that the server may not actually delete the message until it's age has reached up to (ttl + 60) seconds, to allow for flexibility in storage implementations.

Delete Message

Template

DELETE /v1/queues/{queue_name}/messages/{message_id}{?claim_id}


Request

DELETE /v1/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b01 HTTP/1.1
Host: marconi.example.com

...


Response

HTTP/1.1 204 No Content


Discussion

message_id specifies the message to delete.

claim_id specifies that the message should only be deleted if it has the specified claim ID, and that claim has not expired. This is useful for ensuring only one agent processes any given message; whenever a worker client's claim expires before it has a chance to delete a message it has processed, the worker must roll back any actions it took based on that message, since another worker will now be able to claim and process the same message.

Note that if claim_id is not specified, but the message is claimed, the operation will fail. In other words, claimed messages can only ever be deleted by providing an appropriate claim_id.

Delete a Set of Messages by ID

Template

DELETE /v1/queues/{queue_name}/messages{?ids}


Request

DELETE /v1/queues/fizbit/messages?ids=50b68a50d6f5b8c8a7c62b01,50b68a50d6f5b8c8a7c62b02 HTTP/1.1
Host: marconi.example.com

...


Response

HTTP/1.1 204 No Content


Discussion

Bulk delete for messages.

ids specifies the messages to delete, up to a maximum of 20 (configurable, shares the same setting as posting and listing messages)

If any of the message IDs are malformed or non-existent, they are ignored. The remaining valid message IDs are deleted.

Claims

Claim Messages

Template

POST /v1/queues/{queue_name}/claims{?limit}
Content-Type: application/json

...

{ 
    "ttl": {claim_ttl},
    "grace": {message_grace}
}


Request

POST /v1/queues/fizbit/claims?limit=5 HTTP/1.1
Content-Type: application/json
Accept: application/json

...

{ 
    "ttl": 300,
    "grace": 300
}


Response

The client receives a claim URI and a list of claimed messages, if any:

HTTP/1.1 201 Created
Content-Type: application/json
Location: /v1/queues/foo-bar/claims/a28ee94e-6cb4-11e2-b4d5-7703267a7926

...

[
  {
    "href": "/v1/queues/foo-bar/messages/50b68a50d6f5b8c8a7c62b01?claim_id=a28ee94e-6cb4-11e2-b4d5-7703267a7926",
    "ttl": 800,
    "age": 100,
    "body": {
      "object_id": "8a50d6",
      "target": "h.264"
    }
  },
  {
    "href": "/v1/queues/foo-bar/messages/50b68a50d6f5b8c8a7c62b02?claim_id=a28ee94e-6cb4-11e2-b4d5-7703267a7926",
    "ttl": 800,
    "age": 790,
    "body": {
      "object_id": "fb8c8a",
      "target": "h.264"
    }
  }
]


Discussion

Claims a set of messages, up to limit, from oldest to newest, skipping any that are already claimed. If no unclaimed messages are available, returns 204 No Content.

The client should delete the message when it has finished processing it, before the claim expires, to ensure the message is processed only once. As part of the delete operation, all worker clients should specify the claim ID (this is best done by simply using the provided href). That way, the server can return an error if the claim just expired (notifying the client of the race condition), giving the worker a chance to roll back its own processing of the given message, since another worker will likely claim the message and process it.

Just as with a message's age, the age given for the claim is relative to the server's clock, and is useful for determining how quickly messages are getting processed, and whether a given message's claim is about to expire.

When a claim expires, it is removed, allowing another client worker to claim the message in the case that the original worker fails to process it.

limit specifies up to 20 messages (configurable) to claim. If not specified, limit defaults to 10. Note that claim creation is best-effort, meaning the server may claim and return less than the requested number of messages.

ttl is how long the server should wait before releasing the claim. Value MUST be between 60 and 43200 seconds (12 hours, configurable).

grace is the message grace period in seconds. Value MUST be between 60 and 43200 seconds (12 hours, configurable). The server will extend the lifetime of claimed messages to be at least as long as the lifetime of the claim itself, plus a specified grace period to deal with crashed workers (up to 1209600 or 14 days including claim lifetime). If a claimed message would normally live longer than the grace period, it's expiration will not be adjusted.

Query Claim

Template

GET /v1/queues/{queue_name}/claims/{claim_id}


Request

GET /v1/queues/foo-bar/claims/a28ee94e-6cb4-11e2-b4d5-7703267a7926
Host: marconi.example.com

...


Response

HTTP/1.1 200 OK
Content-Location: /v1/queues/foo-bar/claims/a28ee94e-6cb4-11e2-b4d5-7703267a7926

...

{
  "age": 19,
  "ttl": 30,
  "messages": [
    ...
  ]
}


Discussion

Update Claim

Template

PATCH /v1/queues/{queue_name}/claims/{claim_id}
Content-Type: application/json


Request

PATCH /v1/queues/fizbit/claims/a28ee94e-6cb4-11e2-b4d5-7703267a7926 HTTP/1.1
Content-Type: application/json

...

{ "ttl": 300 }


Response

HTTP/1.1 204 No Content


Discussion

Clients should periodically renew claims during long-running batches of work to avoid loosing a claim in the middle of processing a message. This is done by issuing a PATCH to a specific claim resource and including a new TTL for the claim (which may be different from the original TTL). The server will reset the age of the claim and apply the new TTL.

Release Claim

Template

DELETE /v1/queues/{queue_name}/claims/{claim_id}


Request

DELETE /v1/queues/foo-bar/claims/a28ee94e-6cb4-11e2-b4d5-7703267a7926 HTTP/1.1
Host: marconi.example.com


Response

HTTP/1.1 204 No Content


Discussion

Use this operation to immediately release a claim, making any (remaining, non-deleted) messages associated with the claim available to other workers.

This operation is useful when a worker is performing a graceful shutdown, fails to process one or more messages, or is taking longer than expected to process messages, and wishes to make the remainder available to other workers.