Jump to: navigation, search

Difference between revisions of "Zaqar/specs/api/v1"

m (Claim Messages)
(Post Message(s): Note about "partial" deprecation)
 
(122 intermediate revisions by 7 users not shown)
Line 1: Line 1:
 
= Marconi API: v1 Blueprint =
 
= Marconi API: v1 Blueprint =
  
NOTE: For the latest changes under discussion (but not yet approved) see the etherpad working draft:
 
  
https://etherpad.openstack.org/queuing-api
+
<big>'''The Marconi v1 API is FROZEN'''</big>
  
General requirements: [[marconi/specs/grizzly]]
+
We have started collecting feedback with an eye toward extensions and v2 of the API here: [[Marconi/specs/api/next]]
 
 
''' To Do '''
 
 
 
* Specify all possible error responses for each type of request
 
* Define and register message media type/ messages media type - http://www.iana.org/assignments/media-types/index.html
 
* Flesh out examples, incomplete sections (obviously)
 
* Clean up this document, add a FAQ section
 
  
  
Line 22: Line 14:
  
 
Note: Error responses are enumerated on a separate page: [[Marconi/specs/api/v1/errors]]
 
Note: Error responses are enumerated on a separate page: [[Marconi/specs/api/v1/errors]]
 +
  
 
== Common API Elements ==
 
== Common API Elements ==
  
  
''' HTTPS '''
+
=== Clients ===
 
 
All requests to authenticate and operate against the API in production environments use SSL/TLS over HTTP (HTTPS) on TCP port 443. Web servers should only accept high-quality cipher suites, such as AES256_SHA and ECDHE_RSA_AES256_SHA. If hardware acceleration (e.g., AES_NI) is not available, RC4_SHA and ECDHE_RSA_RC4_SHA may be used with some discretion.
 
 
 
 
 
''' Clients '''
 
  
 
* Clients should follow HTTP redirects
 
* Clients should follow HTTP redirects
Line 39: Line 27:
  
  
''' API Versioning '''
+
=== API Versioning ===
  
 
The Marconi API uses a URI versioning scheme. The first element of the path contains the target version identifier, e.g.:
 
The Marconi API uses a URI versioning scheme. The first element of the path contains the target version identifier, e.g.:
Line 49: Line 37:
 
The URI version will only be incremented to accommodate major new features or API redesigns that can not be made backwards-compatible. When new API versions are released, older versions are deprecated. Marconi maintainers will work with developers and partners to ensure there is adequate time to migrate to new versions before deprecated ones are discontinued.
 
The URI version will only be incremented to accommodate major new features or API redesigns that can not be made backwards-compatible. When new API versions are released, older versions are deprecated. Marconi maintainers will work with developers and partners to ensure there is adequate time to migrate to new versions before deprecated ones are discontinued.
  
Since the base URI is only incremented to accommodate major API revisions, sub-versioning of the API is meaningless and is therefore not used. For example, the next version after "v1" would be "v2", not "v1.1" or some variant thereof. ([HATEOS and media types|http://www.informit.com/articles/article.aspx?p=1566460] are used in lieu of minor, URI-based versioning.)
+
Since the version is only incremented to accommodate major API revisions, sub-versioning of the API will be rare, but may be used occasionally to release a "polished" major version.
  
 
+
=== Resource Media Types ===
''' Resource Media Types '''
 
  
 
The API supports `application/json` (XML is on the road map), encoded as UTF-8. Gzip'd requests are optionally supported by the server.
 
The API supports `application/json` (XML is on the road map), encoded as UTF-8. Gzip'd requests are optionally supported by the server.
Line 59: Line 46:
  
  
''' Authentication '''
+
=== Authentication ===
 +
 
 +
''May be optionally implemented by middleware or reverse proxy. If implemented, the following applies:''
  
 
All requests to the API may only be performed by an authenticated agent.
 
All requests to the API may only be performed by an authenticated agent.
Line 67: Line 56:
 
In response to valid credentials, Keystone responds with an auth token and a service catalog that contains a list of all services and endpoints available for the given token. Multiple endpoints may be returned for Marconi according to physical locations and performance/availability characteristics of different deployments.
 
In response to valid credentials, Keystone responds with an auth token and a service catalog that contains a list of all services and endpoints available for the given token. Multiple endpoints may be returned for Marconi according to physical locations and performance/availability characteristics of different deployments.
  
Marconi clients must specify a valid auth token in the `X-Auth-Token` header for each request to the Marconi API. The API validates auth tokens against Keystone before servicing each request.
+
Normally, Keystone middleware provides the X-Project-Id header based on the auth token submitted by the Marconi client. For this to work, clients MUST specify a valid auth token in the `X-Auth-Token` header for each request to the Marconi API. The API validates auth tokens against Keystone before servicing each request
  
 +
If auth is not enabled, clients must provide the X-Project-Id header themselves.
  
''' Authorization '''
+
=== Authorization ===
 +
 
 +
''May be optionally implemented by middleware or reverse proxy. If implemented, the following applies:''
  
 
The API needs to verify read/write access to all endpoints according to the provided auth token (RBAC). The ACL should be cached with the token.
 
The API needs to verify read/write access to all endpoints according to the provided auth token (RBAC). The ACL should be cached with the token.
  
  
''' Tenant ID '''
+
=== Endpoints ===
  
Auth tokens are only valid for a particular tenant ID, which should be reflected in the service endpoints retrieved from Keystone. Agents use the given endpoint as a home document href from which to discover links to all other API requests.
+
Clients may choose to interact with any of the service endpoints retrieved from Keystone. Agents use the given endpoint as a home document href from which to discover links to all other API requests. Endpoints may be organized by region
  
 
An example endpoint:
 
An example endpoint:
  
 
<pre><nowiki>
 
<pre><nowiki>
https://marconi.example.com/v1/480924
+
https://marconi.example.com/v1
 
</nowiki></pre>
 
</nowiki></pre>
  
The client chooses one of the presented endpoints and uses it as the base URL for all subsequent requests.
+
=== Errors ===
  
 
+
If any request results in an error, the server will return an appropriate 4xx or 5xx HTTP status code, and optionally the following information in the body:
''' Errors '''
 
 
 
If any request results in an error, the server will return an appropriate 4xx or 5xx HTTP status code, as well as the following information in the body:
 
  
 
* Title
 
* Title
Line 116: Line 105:
 
</nowiki></pre>
 
</nowiki></pre>
  
 
+
=== Common Headers ===
''' Common Headers '''
 
  
 
Each request to the API must include certain standard and extended HTTP headers. These headers provide host, agent, authentication and other pertinent information to the server.
 
Each request to the API must include certain standard and extended HTTP headers. These headers provide host, agent, authentication and other pertinent information to the server.
Line 127: Line 115:
 
|  Host
 
|  Host
 
|  The host name of the API, as referenced by the Keystone service catalog
 
|  The host name of the API, as referenced by the Keystone service catalog
|-
 
|  User-Agent
 
|  The name and version of the Marconi client, as well as a UUID for that client. Marconi uses the UUID to distinguish publishers from subscribers, i.e., to avoid echoing an agent's own messages back to it.
 
 
|-
 
|-
 
|  Date
 
|  Date
Line 135: Line 120:
 
|-
 
|-
 
|  Accept
 
|  Accept
|  Media type desired; initially, only `application/json` will be supported
+
|  Media type desired; only <code><nowiki>application/json</nowiki></code> is supported at this time.
 
|-
 
|-
 
|  Accept-Encoding
 
|  Accept-Encoding
Line 145: Line 130:
 
|  X-Auth-Token
 
|  X-Auth-Token
 
|  Keystone auth token
 
|  Keystone auth token
 +
|-
 +
|  X-Project-Id
 +
|  An ID for a project to which the value of X-Auth-Token grants access. Queues will be created under this project.
 +
|-
 +
|  Client-ID
 +
|  A UUID, used to distinguish each client using the service. The UUID must be submitted in its canonical form (e.g., <code><nowiki>3381af92-2b9e-11e3-b191-71861300734c</nowiki></code>). In Marconi, this is used to avoid echoing a sender's messages back to the same instance, and may be logged by the server for future use. Should be generated once and persisted between restarts of the client.
 
|}
 
|}
  
 +
=== HTTP Response Codes ===
 +
 +
See [[Marconi/specs/api/v1/responsecodes| Response Codes]]
  
 
== Sample API Request ==
 
== Sample API Request ==
  
 
<pre><nowiki>
 
<pre><nowiki>
GET /v1/480924/queues/fizbat/messages?marker=1355-237242-783&limit=10 HTTP/1.1
+
GET /v1/queues/fizbat/messages?marker=1355-237242-783&limit=10 HTTP/1.1
 
Host: marconi.example.com
 
Host: marconi.example.com
User-Agent: python/2.7 killer-rabbit/1.2 uuid/30387f00-39a0-11e2-be4d-a8d15f34bae2
+
User-Agent: python/2.7 killer-rabbit/1.2
 
Date: Wed, 28 Nov 2012 21:14:19 GMT
 
Date: Wed, 28 Nov 2012 21:14:19 GMT
 
Accept: application/json
 
Accept: application/json
 
Accept-Encoding: gzip
 
Accept-Encoding: gzip
 
X-Auth-Token: 7d2f63fd-4dcc-4752-8e9b-1d08f989cc00
 
X-Auth-Token: 7d2f63fd-4dcc-4752-8e9b-1d08f989cc00
 +
X-Project-Id: 518b51ea133c4facadae42c328d6b77b
 +
Client-ID: 30387f00-39a0-11e2-be4d-a8d15f34bae2
 
</nowiki></pre>
 
</nowiki></pre>
  
 +
== Endpoints Synopsis ==
  
== Get Home Document ==
+
<pre><nowiki>
 +
# Base endpoints
 +
GET /v1
 +
GET /v1/health
  
 +
# Queues
 +
GET /v1/queues{?marker,limit,detailed}
 +
GET /v1/queues/{queue_name}  # existence check
 +
HEAD /v1/queues/{queue_name}  # existence check
 +
PUT /v1/queues/{name}
 +
DELETE /v1/queues/{name}
 +
 +
# Queue metadata
 +
PUT /v1/queues/{queue_name}/metadata
 +
GET /v1/queues/{queue_name}/metadata
 +
GET /v1/queues/{queue_name}/stats
 +
 +
# Messages
 +
GET /v1/queues/{queue_name}/messages{?marker,limit,echo,include_claimed}
 +
GET /v1/queues/{queue_name}/messages/{message_id}
 +
GET /v1/queues/{queue_name}/messages{?ids}
 +
POST /v1/queues/{queue_name}/messages
 +
DELETE /v1/queues/{queue_name}/messages/{message_id}{?claim_id}
 +
DELETE /v1/queues/{queue_name}/messages{?ids}
 +
 +
# Claims
 +
POST /v1/queues/{queue_name}/claims{?limit}
 +
GET /v1/queues/{queue_name}/claims/{claim_id}
 +
PATCH /v1/queues/{queue_name}/claims/{claim_id}
 +
DELETE /v1/queues/{queue_name}/claims/{claim_id}
 +
</nowiki></pre>
 +
 +
== Base Endpoints ==
 +
 +
=== Get Home Document ===
  
 
'''Template'''
 
'''Template'''
Line 177: Line 207:
 
Host: marconi.example.com
 
Host: marconi.example.com
  
[...]
+
...
  
 
</nowiki></pre>
 
</nowiki></pre>
Line 185: Line 215:
  
 
<pre><nowiki>
 
<pre><nowiki>
@todo
+
HTTP/1.0 200 OK
 +
 
 +
...
 +
 
 +
{
 +
    "resources": {
 +
        "rel/queue-stats": {
 +
            "href-template": "/v1/queues/{queue_name}/stats",
 +
            "href-vars": {
 +
                "queue_name": "param/queue_name"
 +
            },
 +
            "hints": {
 +
                "allow": [
 +
                    "GET"
 +
                ],
 +
                "formats": {
 +
                    "application/json": {}
 +
                }
 +
            }
 +
        },
 +
        "rel/post-messages": {
 +
            "href-template": "/v1/queues/{queue_name}/messages",
 +
            "href-vars": {
 +
                "queue_name": "param/queue_name"
 +
            },
 +
            "hints": {
 +
                "accept-post": [
 +
                    "application/json"
 +
                ],
 +
                "allow": [
 +
                    "POST"
 +
                ],
 +
                "formats": {
 +
                    "application/json": {}
 +
                }
 +
            }
 +
        },
 +
        "rel/queue": {
 +
            "href-template": "/v1/queues/{queue_name}",
 +
            "href-vars": {
 +
                "queue_name": "param/queue_name"
 +
            },
 +
            "hints": {
 +
                "allow": [
 +
                    "GET",
 +
                    "HEAD",
 +
                    "PUT",
 +
                    "DELETE"
 +
                ],
 +
                "formats": {
 +
                    "application/json": {}
 +
                }
 +
            }
 +
        },
 +
        "rel/queue-metadata": {
 +
            "href-template": "/v1/queues/{queue_name}/metadata",
 +
            "href-vars": {
 +
                "queue_name": "param/queue_name"
 +
            },
 +
            "hints": {
 +
                "allow": [
 +
                    "GET",
 +
                    "PUT"
 +
                ],
 +
                "formats": {
 +
                    "application/json": {}
 +
                }
 +
            }
 +
        },
 +
        "rel/queues": {
 +
            "href-template": "/v1/queues{?marker,limit,detailed}",
 +
            "href-vars": {
 +
                "marker": "param/marker",
 +
                "detailed": "param/detailed",
 +
                "limit": "param/queue_limit"
 +
            },
 +
            "hints": {
 +
                "allow": [
 +
                    "GET"
 +
                ],
 +
                "formats": {
 +
                    "application/json": {}
 +
                }
 +
            }
 +
        },
 +
        "rel/messages": {
 +
            "href-template": "/v1/queues/{queue_name}/messages{?marker,limit,echo,include_claimed}",
 +
            "href-vars": {
 +
                "marker": "param/marker",
 +
                "include_claimed": "param/include_claimed",
 +
                "queue_name": "param/queue_name",
 +
                "limit": "param/messages_limit",
 +
                "echo": "param/echo"
 +
            },
 +
            "hints": {
 +
                "allow": [
 +
                    "GET"
 +
                ],
 +
                "formats": {
 +
                    "application/json": {}
 +
                }
 +
            }
 +
        },
 +
        "rel/claim": {
 +
            "href-template": "/v1/queues/{queue_name}/claims{?limit}",
 +
            "href-vars": {
 +
                "queue_name": "param/queue_name",
 +
                "limit": "param/claim_limit"
 +
            },
 +
            "hints": {
 +
                "accept-post": [
 +
                    "application/json"
 +
                ],
 +
                "allow": [
 +
                    "POST"
 +
                ],
 +
                "formats": {
 +
                    "application/json": {}
 +
                }
 +
            }
 +
        }
 +
    }
 +
}
 
</nowiki></pre>
 
</nowiki></pre>
  
Line 197: Line 349:
 
See also: http://tools.ietf.org/html/draft-nottingham-json-home-02
 
See also: http://tools.ietf.org/html/draft-nottingham-json-home-02
  
 
+
=== Check Node Health ===
== Create Queue ==
 
  
  
Line 204: Line 355:
  
 
<pre><nowiki>
 
<pre><nowiki>
PUT {base_url}/queues/{queue_name}
+
GET /v1/health
 +
</nowiki></pre>
  
{
+
or
    ...
+
 
}
+
<pre><nowiki>
 +
HEAD /v1/health
 
</nowiki></pre>
 
</nowiki></pre>
 +
  
 
'''Request'''
 
'''Request'''
  
 
<pre><nowiki>
 
<pre><nowiki>
PUT /v1/480924/queues/fizbat HTTP/1.1
+
GET /v1/health HTTP/1.1
Host: marconi.example.com
+
Host: example.marconi.com
 +
 
 +
...
 +
 
 +
</nowiki></pre>
 +
 
 +
 
 +
'''Response'''
 +
 
 +
<pre><nowiki>
 +
HTTP/1.1 204 No Content
 +
</nowiki></pre>
 +
 
 +
or
 +
 
 +
<pre><nowiki>
 +
HTTP/1.1 503 Service Unavailable
 +
</nowiki></pre>
 +
 
 +
 
 +
 
 +
'''Discussion'''
 +
 
 +
Use this request to check on the Marconi node status. If the node is down, this endpoint will not be reachable and will act as a signal to the load balancer to take the node out of rotation. Alternatively, the endpoint may be reachable but the service is temporarily unavailable due to storage driver failure or some other error.
 +
 
 +
== Queues ==
 +
 
 +
=== Create Queue ===
 +
 
 +
 
 +
'''Template'''
  
[...]
+
<pre><nowiki>
 +
PUT /v1/queues/{queue_name}
 +
</nowiki></pre>
  
{
+
'''Request'''
    "game_id": "a34904d6-8dbc-11e2-864b-5d361e90beb5"
 
}
 
  
 +
<pre><nowiki>
 +
PUT /v1/queues/fizbat HTTP/1.1
 +
Host: marconi.example.com
 
</nowiki></pre>
 
</nowiki></pre>
  
Line 230: Line 417:
 
<pre><nowiki>
 
<pre><nowiki>
 
HTTP/1.1 201 Created
 
HTTP/1.1 201 Created
Location: /v1/480924/queues/fizbit
+
Location: /v1/queues/fizbit
 
</nowiki></pre>
 
</nowiki></pre>
  
Line 236: Line 423:
 
'''Discussion'''
 
'''Discussion'''
  
Creates a queue and sets its metadata.
+
Creates a queue.
  
The body of the PUT is a document that can contain arbitrary metadata to be associated with the queue, up to 4 KiB when serialized as JSON, including whitespace. Top-level field names MUST NOT start with an underscore; such names are reserved for future use.
+
The body of the PUT is empty.
  
<code><nowiki>queue_name</nowiki></code> is the name to give the queue. The name MUST NOT exceed 512 bytes in length. If the specified queue already exists, the existing queue's metadata will be overwritten with the provided document.
+
<code><nowiki>queue_name</nowiki></code> is the name to give the queue. The name MUST NOT exceed 64 bytes in length, and is limited to US-ASCII letters, digits, underscores and hyphens.
  
== List Queues ==
+
=== List Queues ===
  
  
Line 248: Line 435:
  
 
<pre><nowiki>
 
<pre><nowiki>
GET {base_url}/queues{?marker,limit,metadata}
+
GET /v1/queues{?marker,limit,detailed}
  
 
</nowiki></pre>
 
</nowiki></pre>
Line 256: Line 443:
  
 
<pre><nowiki>
 
<pre><nowiki>
GET {base_url}/queues?marker=baz&metadata=true HTTP/1.1
+
GET /v1/queues?marker=baz&detailed=true HTTP/1.1
 
Host: marconi.example.com
 
Host: marconi.example.com
  
[...]
+
...
 
</nowiki></pre>
 
</nowiki></pre>
  
Line 270: Line 457:
 
{
 
{
 
   "links": [
 
   "links": [
    {
 
      "rel": "prev",
 
      "href": "/v1/480924/queues/fizbit/queues?marker=axon&limit=10"
 
    },
 
 
     {
 
     {
 
       "rel": "next",
 
       "rel": "next",
       "href": "/v1/480924/queues/fizbit/queues?marker=kooleo&limit=10"
+
       "href": "/v1/queues?marker=kooleo&limit=10&detailed=true"
 
     }
 
     }
 
   ],
 
   ],
 
   "queues": [
 
   "queues": [
     { "name": "boomerang", "href": "/v1/480924/queues/boomerang", "metadata": {} },
+
     { "name": "boomerang", "href": "/v1/queues/boomerang", "metadata": {} },
     { "name": "fizbit", href: "/v1/480924/queues/fizbit", "metadata": { "handle": "@kgriffs" } },
+
     { "name": "fizbit", "href": "/v1/queues/fizbit", "metadata": { "handle": "@kgriffs" } },
  
     [...]
+
     ...
  
     { "name": "kooleo",  "href": "/v1/480924/queues/kooleo", "metadata": { "something": "something_else" } }
+
     { "name": "kooleo",  "href": "/v1/queues/kooleo", "metadata": { "something": "something_else" } }
 
   ]
 
   ]
 
}
 
}
  
 +
</nowiki></pre>
 +
 +
=== Checking Queue Existence ===
 +
 +
'''Template'''
 +
 +
<pre><nowiki>
 +
GET /v1/queues/{queue_name}
 
</nowiki></pre>
 
</nowiki></pre>
  
  
'''Discussion'''
+
'''Request'''
  
Lists queues belonging to the specified tenant, sorted alphabetically by name.
+
<pre><nowiki>
 +
GET /v1/queues/{queue_name}
 +
Host: marconi.example.com
 +
</nowiki></pre>
  
<code><nowiki>limit</nowiki></code> specifies up to 100 to return. If not specified, limit defaults to 10. Paging is supported via "prev" and "next" links.
 
  
<code><nowiki>marker</nowiki></code> is the name of the last queue recieved in a previous request, or none to get the first page of results.
+
'''Response'''
  
<code><nowiki>metadata</nowiki></code> is a boolean ("true" or "false") that determines whether queue metadata is included in the response. Defaults to "false".
+
<pre><nowiki>
 +
HTTP/1.1 204 OK
 +
</nowiki></pre>
  
  
== Set Queue Metadata ==
+
'''Discussion'''
  
 +
Returns 204 if the queue exists, else 404.
 +
 +
<code><nowiki>HEAD</nowiki></code> also works.
 +
 +
=== Delete Queue ===
  
 
'''Template'''
 
'''Template'''
  
 
<pre><nowiki>
 
<pre><nowiki>
PUT {base_url}/queues/{queue_name}
+
DELETE /v1/queues/{queue_name}
 
 
{
 
  ...
 
}
 
 
</nowiki></pre>
 
</nowiki></pre>
  
Line 320: Line 516:
  
 
<pre><nowiki>
 
<pre><nowiki>
PUT /v1/480924/queues/fizbat HTTP/1.1
+
DELETE /v1/queues/fizbat HTTP/1.1
 
Host: marconi.example.com
 
Host: marconi.example.com
 
[...]
 
 
{
 
  "key": {
 
      "key2": "value",
 
      "key3": [1, 2, 3, 4, 5]
 
    }
 
}
 
 
</nowiki></pre>
 
</nowiki></pre>
  
Line 343: Line 530:
 
'''Discussion'''
 
'''Discussion'''
  
Set queue metadata. If PUT is used, this document will replace the existing metadata document in it's entirety, so clients need to be careful they do not accidentally delete fields.
+
Use this operation to immediately delete a queue along with all its messages (if any).
 
 
PATCH support is a "future" or "if we have time" feature.
 
  
 +
== Queue Metadata ==
  
== Get Queue Metadata ==
+
=== Set Queue Metadata ===
  
  
Line 354: Line 540:
  
 
<pre><nowiki>
 
<pre><nowiki>
GET {base_url}/queues/{queue_name}
+
PUT /v1/queues/{queue_name}/metadata
 +
 
 +
{
 +
  ...
 +
}
 
</nowiki></pre>
 
</nowiki></pre>
  
Line 361: Line 551:
  
 
<pre><nowiki>
 
<pre><nowiki>
GET /v1/480924/queues/fizbit HTTP/1.1
+
PUT /v1/queues/fizbat/metadata HTTP/1.1
 
Host: marconi.example.com
 
Host: marconi.example.com
  
[...]
+
...
  
 +
{
 +
  "key": {
 +
      "key2": "value",
 +
      "key3": [1, 2, 3, 4, 5]
 +
    }
 +
}
 
</nowiki></pre>
 
</nowiki></pre>
  
  
 
'''Response'''
 
'''Response'''
HTTP/1.1 200 OK
 
  
 
<pre><nowiki>
 
<pre><nowiki>
{
+
HTTP/1.1 204 No Content
  "ttl": 86400,
 
}
 
 
</nowiki></pre>
 
</nowiki></pre>
  
Line 381: Line 574:
 
'''Discussion'''
 
'''Discussion'''
  
Returns queue metadata, such as message TTL.
+
This operation allows for setting structured metadata. If PUT is used, this document will replace the existing metadata document in it's entirety, so clients need to be careful they do not accidentally delete fields.
  
 +
The body of the request is an arbitrary document which allows storing contextual information about the way a particular application should interact with the queue. The size of this body, in characters and including whitespace must be <= 64 KiB (configurable). The document MUST be valid JSON (Marconi will validate).
  
== Get Queue Stats ==
+
=== Get Queue Metadata ===
  
  
Line 390: Line 584:
  
 
<pre><nowiki>
 
<pre><nowiki>
GET {base_url}/queues/{queue_name}/stats
+
GET /v1/queues/{queue_name}/metadata
 
</nowiki></pre>
 
</nowiki></pre>
  
Line 397: Line 591:
  
 
<pre><nowiki>
 
<pre><nowiki>
GET /v1/480924/queues/fizbit/stats HTTP/1.1
+
GET /v1/queues/fizbit/metadata HTTP/1.1
 
Host: marconi.example.com
 
Host: marconi.example.com
  
[...]
+
...
  
 
</nowiki></pre>
 
</nowiki></pre>
Line 410: Line 604:
 
<pre><nowiki>
 
<pre><nowiki>
 
{
 
{
  "messages": 46,
+
  "key": {
  "actions": 122
+
      "key2": "value",
 +
      "key3": [1, 2, 3, 4, 5]
 +
    }
 
}
 
}
 
</nowiki></pre>
 
</nowiki></pre>
Line 418: Line 614:
 
'''Discussion'''
 
'''Discussion'''
  
Returns queue statistics, including how many messages are in the queue, and how many actions have been recorded (actions can be retrieved by performing a GET on {queue_name}/actions).
+
Returns queue metadata.
  
 
+
=== Get Queue Stats ===
== Delete Queue ==
 
  
  
Line 427: Line 622:
  
 
<pre><nowiki>
 
<pre><nowiki>
DELETE {base_url}/queues/{queue_name}
+
GET /v1/queues/{queue_name}/stats
 
</nowiki></pre>
 
</nowiki></pre>
  
Line 434: Line 629:
  
 
<pre><nowiki>
 
<pre><nowiki>
DELETE /v1/480924/queues/fizbat HTTP/1.1
+
GET /v1/queues/fizbit/stats HTTP/1.1
 
Host: marconi.example.com
 
Host: marconi.example.com
 +
 +
...
 +
 
</nowiki></pre>
 
</nowiki></pre>
  
  
 
'''Response'''
 
'''Response'''
 +
HTTP/1.1 200 OK
  
 
<pre><nowiki>
 
<pre><nowiki>
HTTP/1.1 204 No Content
+
{
 +
  "messages": {
 +
    "free": 146929,
 +
    "claimed": 2409,
 +
    "total": 149338,
 +
    "oldest": {
 +
        "href": "/v1/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b01",
 +
        "age": 63,
 +
        "created": "2013-08-12T20:44:55Z"
 +
    },
 +
    "newest": {
 +
        "href": "/v1/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b01",
 +
        "age": 12,
 +
        "created": "2013-08-12T20:45:46Z"
 +
    }
 +
}
 
</nowiki></pre>
 
</nowiki></pre>
  
Line 448: Line 662:
 
'''Discussion'''
 
'''Discussion'''
  
Use this operation to immediately delete a queue along with all its messages (if any).
+
Returns queue statistics, including how many messages are in the queue, broken out by status.
 +
 
 +
NOTE: If total is 0, then "oldest" and "newest" message stats are not included.
 +
 
 +
== Messages ==
  
 +
All message-related operations require <code>Client-Id</code> to be included in the headers. This is to ensure that messages are not echoed back to the client that posted them unless the client explicitly requests this.
  
== Get Messages ==
+
=== List Messages ===
  
  
Line 457: Line 676:
  
 
<pre><nowiki>
 
<pre><nowiki>
GET {base_url}/queues/{queue_name}/messages/{?marker,limit,echo}
+
GET /v1/queues/{queue_name}/messages{?marker,limit,echo,include_claimed}
 
</nowiki></pre>
 
</nowiki></pre>
  
Line 464: Line 683:
  
 
<pre><nowiki>
 
<pre><nowiki>
GET /v1/480924/queues/fizbit/messages&marker=1355-237242-783&limit=10 HTTP/1.1
+
GET /v1/queues/fizbit/messages?marker=1355-237242-783&limit=10 HTTP/1.1
 
Host: marconi.example.com
 
Host: marconi.example.com
  
[...]
+
...
  
 
</nowiki></pre>
 
</nowiki></pre>
Line 478: Line 697:
 
<pre><nowiki>
 
<pre><nowiki>
 
HTTP/1.1 200 OK
 
HTTP/1.1 200 OK
Content-Location: /v1/480924/queues/fizbit/messages?marker=1355-237242-783&limit=10
+
Content-Location: /v1/queues/fizbit/messages?marker=1355-237242-783&limit=10
  
[...]
+
...
  
 
{
 
{
Line 486: Line 705:
 
     {
 
     {
 
       "rel": "next",
 
       "rel": "next",
       "href": "/v1/480924/queues/fizbit/messages?marker=6244-244224-783&limit=10"
+
       "href": "/v1/queues/fizbit/messages?marker=6244-244224-783&limit=10"
 
     }
 
     }
 
   ],
 
   ],
 
   "messages": [
 
   "messages": [
 
     {
 
     {
       "id": "50b68a50d6f5b8c8a7c62b01",
+
       "href": "/v1/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b01",
 
       "ttl": 800,
 
       "ttl": 800,
 
       "age": 790,
 
       "age": 790,
Line 510: Line 729:
 
Results are ordered by age, oldest message first.
 
Results are ordered by age, oldest message first.
  
<code><nowiki>limit</nowiki></code> specifies up to 100 messages to return. If not specified, limit defaults to 10. When more messages are available than can be returned in a single request, the client can pick up the next batch by simply using the URI template parameters returned from the previous call in the "next" field (TBD).
+
<code><nowiki>limit</nowiki></code> specifies up to 20 messages (configurable) to return. If not specified, limit defaults to 10. When more messages are available than can be returned in a single request, the client can pick up the next batch by simply using the URI template parameters returned from the previous call in the "next" field (TBD).
  
 
<code><nowiki>marker</nowiki></code> is an opaque string that the client can use to request the next batch of messages. It communicates to the server which messages the client has already received.
 
<code><nowiki>marker</nowiki></code> is an opaque string that the client can use to request the next batch of messages. It communicates to the server which messages the client has already received.
Line 516: Line 735:
 
Note: If <code><nowiki>marker</nowiki></code> is not specified, the API will return all messages at the head of the queue (up to limit).
 
Note: If <code><nowiki>marker</nowiki></code> is not specified, the API will return all messages at the head of the queue (up to limit).
  
<code><nowiki>echo</nowiki></code> is a boolean (i.e., "true" or "false") that determines whether or not the API will return a client's own messages, as determined by the <code><nowiki>uuid</nowiki></code> portion of the User-Agent header. If not specified, echo defaults to "false".
+
<code><nowiki>echo</nowiki></code> is a boolean (i.e., "true" or "false") that determines whether or not the API will return a client's own messages, as determined by the <code><nowiki>uuid</nowiki></code> portion of the Client-ID header. If not specified, echo defaults to "false".
  
 +
<code><nowiki>include_claimed</nowiki></code> is a boolean (i.e., "true" or "false") that determines whether or not the API will return claimed messages as well as unclaimed ones. If not specified, defaults to "false" (only unclaimed messages are returned).
  
== Get Actions ==
+
=== Get a Specific Message ===
  
  
Line 525: Line 745:
  
 
<pre><nowiki>
 
<pre><nowiki>
GET {base_url}/queues/{queue_name}/actions/{?marker,limit}
+
GET /v1/queues/{queue_name}/messages/{message_id}
 
</nowiki></pre>
 
</nowiki></pre>
  
Line 532: Line 752:
  
 
<pre><nowiki>
 
<pre><nowiki>
GET /v1/480924/queues/fizbit/actions&marker=1355-237242-783&limit=10 HTTP/1.1
+
GET /v1/queues/fizbat/messages/50b68a50d6f5b8c8a7c62b01 HTTP/1.1
 
Host: marconi.example.com
 
Host: marconi.example.com
  
[...]
+
...
  
 
</nowiki></pre>
 
</nowiki></pre>
Line 541: Line 761:
  
 
'''Response'''
 
'''Response'''
 
HTTP 200 if the query matched 1 or more actions, HTTP 204 otherwise (with no body).
 
  
 
<pre><nowiki>
 
<pre><nowiki>
 
HTTP/1.1 200 OK
 
HTTP/1.1 200 OK
Content-Location: /v1/480924/queues/fizbit/actions?marker=1355-237242-783&limit=10
+
Content-Location: /v1/queues/fizbat/messages/50b68a50d6f5b8c8a7c62b01
  
[...]
+
...
  
 
{
 
{
   "links": [
+
   "href": "/v1/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b01",
    {
+
   "ttl": 800,
      "rel": "next",
+
  "age": 790,
      "href": "/v1/480924/queues/fizbit/actions?marker=6244-244224-783&limit=10"
+
  "body": {
    }
+
    "event": "ActivateAccount",
  ],
+
     "mode": "active"
   "messages": [
+
  }
    {
 
      "id": 10b00a50d6f5b8c8a7c62ccb,
 
      "ttl": 140,
 
      "age": 120,
 
      "body": {
 
        "event": "LockExpired",
 
        "details": {
 
          "message_id": "50b68a50d6f5b8c8a7c62a87",
 
          "user_agent": "python/2.7 killer-rabbit/1.2 uuid/79ed56f8-2519-11e2-b835-acf6018e45a1",
 
        }
 
      }
 
    },
 
     {
 
      "id": 10b00a50d6f5b8c8a7c62ccc,
 
      "ttl": 140,
 
      "age": 121,
 
      "body": {
 
        "event": "MessageDeleted",
 
        "details": {
 
          "message_id": "50b68a50d6f5b8c8a7c62b01",
 
          "user_agent": "python/2.7 killer-rabbit/1.2 uuid/79ed56f8-2519-11e2-b835-acf6018e45a1",
 
        }
 
      }
 
    }
 
  ]
 
 
}
 
}
 
</nowiki></pre>
 
</nowiki></pre>
Line 589: Line 782:
 
'''Discussion'''
 
'''Discussion'''
  
The interactions of various agents/workers with a cloud queuing services can be difficult to audit and debug. Marconi emits action messages to a special queue, from which auditors can retrieve a list of recent actions involving a specific queue's messages. These actions can then be archived by the auditor for future analysis and diagnostics.
+
Message fields are defined as follows:
  
Action messages expire after 5 minutes, so auditors should regularly poll for actions and persist them for posterity.
+
<code><nowiki>href</nowiki></code> is an opaque relative URI that the client can use to uniquely identify a message resource, and interact with it.
  
The following actions are recorded, since they cannot be otherwise observed:
+
<code><nowiki>ttl</nowiki></code> is the ttl set on the message when it was posted. The message will expire after (ttl - age) seconds.
  
* Delete message
+
<code><nowiki>age</nowiki></code> number of seconds since ts, relative to the server's clock.
* Claim messages
 
* Update claim
 
* Release claim
 
* Claim expired
 
  
Message IDs and markers are opaque strings; clients should make no assumptions on their format or length. Furthermore, clients should assume there is no relationship between markers and message IDs (i.e., one cannot be derived from the other). This ensures maximum flexibility when implementing storage drivers.
+
<code><nowiki>body</nowiki></code> Arbitrary document submitted along with the original request to post the message.
  
Results are always ordered by age, oldest message first.
+
If either the message ID or the claim ID is malformed or nonexistent, no message is returned.
  
<code><nowiki>limit</nowiki></code> specifies up to 100 messages to return. If not specified, limit defaults to 10. When more messages are available than can be returned in a single request, the client can pick up the next batch by simply using the URI template parameters returned from the previous call in the "next" field.
+
=== Get a Set of Messages by ID ===
 
 
<code><nowiki>marker</nowiki></code> is an opaque string that the client can use to request the next batch of messages. It communicates to the server which messages the client has already received.
 
 
 
Note: If <code><nowiki>marker</nowiki></code> is not specified, the API will return all messages at the head of the queue (up to limit).
 
 
 
 
 
== Get a Specific Message ==
 
  
  
Line 618: Line 800:
  
 
<pre><nowiki>
 
<pre><nowiki>
GET {base_url}/queues/{queue_name}/messages/{message_id}
+
GET /v1/queues/{queue_name}/messages{?ids}
 
</nowiki></pre>
 
</nowiki></pre>
  
Line 625: Line 807:
  
 
<pre><nowiki>
 
<pre><nowiki>
GET /v1/480924/queues/fizbat/messages/50b68a50d6f5b8c8a7c62b01 HTTP/1.1
+
GET /v1/queues/fizbat/messages?ids=50b68a50d6f5b8c8a7c62b01,f5b8c8a7c62b0150b68a50d6 HTTP/1.1
 
Host: marconi.example.com
 
Host: marconi.example.com
  
[...]
+
...
  
 
</nowiki></pre>
 
</nowiki></pre>
Line 637: Line 819:
 
<pre><nowiki>
 
<pre><nowiki>
 
HTTP/1.1 200 OK
 
HTTP/1.1 200 OK
Content-Location: /v1/480924/queues/fizbat/messages/50b68a50d6f5b8c8a7c62b01
+
Content-Location: /v1/queues/fizbat/messages?ids=50b68a50d6f5b8c8a7c62b01,f5b8c8a7c62b0150b68a50d6
  
[...]
+
...
  
{
+
[
  "id": "50b68a50d6f5b8c8a7c62b01",
+
    {
  "ttl": 800,
+
      "href": "/v1/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b01",
  "age": 790,
+
      "ttl": 800,
  "body": {
+
      "age": 32,
     "event": "ActivateAccount",
+
      "body": {
    "mode": "active"
+
        "cmd": "EncodeVideo",
  }
+
        "jobid": 58229
}
+
      }
 +
    },
 +
     {
 +
      "href": "/v1/queues/fizbit/messages/f5b8c8a7c62b0150b68a50d6",
 +
      "ttl": 800,
 +
      "age": 790,
 +
      "body": {
 +
        "cmd": "EncodeAudio",
 +
        "jobid": 58201
 +
      }
 +
    }
 +
]
 
</nowiki></pre>
 
</nowiki></pre>
  
Line 655: Line 848:
 
'''Discussion'''
 
'''Discussion'''
  
Message fields are defined as follows:
+
Unlike message listing, a client's own messages are always returned in this approach. Note that the list of ids may not exceed 20 (configurable, shared setting with number of messages that can be posted at once).
 
 
<code><nowiki>id</nowiki></code> is an opaque string that the client can use to uniquely identify the message.
 
 
 
<code><nowiki>ttl</nowiki></code> is the ttl set on the message when it was posted. The message will expire after (ttl - age) seconds.
 
 
 
<code><nowiki>age</nowiki></code> number of seconds since ts, relative to the server's clock.
 
 
 
<code><nowiki>body</nowiki></code> Arbitrary document submitted along with the original request to post the message.
 
  
 +
This approach returns all the messages with matching IDs. If a malformed ID or a nonexistent message ID is provided, it is ignored and the remaining messages are returned.
  
== Post Message(s) ==
+
=== Post Message(s) ===
  
  
Line 672: Line 858:
  
 
<pre><nowiki>
 
<pre><nowiki>
POST {base_url}/queues/{queue_name}/messages
+
POST /v1/queues/{queue_name}/messages
 
</nowiki></pre>
 
</nowiki></pre>
  
Line 679: Line 865:
  
 
<pre><nowiki>
 
<pre><nowiki>
POST /v1/480924/queues/fizbit/messages HTTP/1.1
+
POST /v1/queues/fizbit/messages HTTP/1.1
 
Host: marconi.example.com
 
Host: marconi.example.com
  
[...]
+
...
  
 
[
 
[
Line 710: Line 896:
 
<pre><nowiki>
 
<pre><nowiki>
 
HTTP/1.1 201 Created
 
HTTP/1.1 201 Created
Location: /v1/480924/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b01
+
Location: /v1/queues/fizbit/messages?ids=50b68a50d6f5b8c8a7c62b01
 +
Content-Type: application/json
 +
 
 +
{
 +
    "resources": [
 +
        "/v1/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b01"
 +
    ],
 +
    "partial": false
 +
}
 
</nowiki></pre>
 
</nowiki></pre>
  
Line 717: Line 911:
 
<pre><nowiki>
 
<pre><nowiki>
 
HTTP/1.1 201 Created
 
HTTP/1.1 201 Created
Location: /v1/480924/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b01,2938472984,234092834,1230487
+
Location: /v1/queues/fizbit/messages?ids=50b68a50d6f5b8c8a7c62b01,50b68a50d6f5b8c8a7c62b05
 +
Content-Type: application/json
 +
 
 +
{
 +
    "resources": [
 +
        "/v1/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b01",
 +
        "/v1/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b05"
 +
    ],
 +
    "partial": false
 +
}
 
</nowiki></pre>
 
</nowiki></pre>
  
TODO(kgriffs): Define a URI template that allows for multiple IDs, also update Get Message(s) spec.
+
'''Discussion'''
  
 +
Up to 20 messages (default, but configurable) may be submitted in a single request, but must always be encapsulated in a collection container (an array in JSON). The resulting value of the Location header or response body may be used to retrieve the created messages for further processing if needed.
  
'''Discussion'''
+
The client only specifies the body and ttl for the message; the server will insert metadata such as id and age.
  
Up to 100 messages may be submitted in a single request, but must always be encapsulated in a collection container (an array in JSON).
+
The response body contains a list of resource paths corresponding to each message submitted in the request, in the same order. In the case of a server-side error part-way through the processing of the submitted messages, a partial list will be returned, <code><nowiki>partial</nowiki></code> will be '''true''', and the client will need to retry posting the remaining messages. If no messages could be enqueued, the server will return 503 Service Unavailable.
  
The client only specifies the body and ttl for the message; the server will insert metadata such as id and age.
+
NOTE: The ''partial'' field has been deprecated since the icehouse release, and will always be "false"; drivers are not longer allowed to return a "partial" success and so must either succeed or fail the entire batch of messages.
  
<code><nowiki>body</nowiki></code> specifies an arbitrary document which constitutes the body of the message being sent. The size of this body, in characters and including whitespace, is 4 KiB. The document MUST be valid JSON (Marconi will validate).
+
The size of the request document, in characters including whitespace, is 256 KiB by default (configurable). The document MUST be valid JSON (Marconi will validate).  
  
<code><nowiki>ttl</nowiki></code> is how long the server should wait before expiring and removing the message from the queue. Value MUST between 60 and 1209600 seconds (14 days).
+
<code><nowiki>body</nowiki></code> specifies an arbitrary document which constitutes the body of the message being sent.  
  
 +
<code><nowiki>ttl</nowiki></code> is how long the server should wait before expiring and removing the message from the queue. Value MUST be between 60 and 1209600 seconds (14 days, configurable), inclusive. Note that the server may not actually delete the message until it's age has reached up to (ttl + 60) seconds, to allow for flexibility in storage implementations.
  
== Delete Message ==
+
=== Delete Message ===
  
  
Line 740: Line 945:
  
 
<pre><nowiki>
 
<pre><nowiki>
DELETE {base_url}/queues/{queue_name}/messages/{message_id}{?claim_id}
+
DELETE /v1/queues/{queue_name}/messages/{message_id}{?claim_id}
 
</nowiki></pre>
 
</nowiki></pre>
  
Line 747: Line 952:
  
 
<pre><nowiki>
 
<pre><nowiki>
DELETE /v1/480924/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b01 HTTP/1.1
+
DELETE /v1/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b01 HTTP/1.1
 
Host: marconi.example.com
 
Host: marconi.example.com
  
[...]
+
...
  
 
</nowiki></pre>
 
</nowiki></pre>
Line 768: Line 973:
 
<code><nowiki>claim_id</nowiki></code> specifies that the message should only be deleted if it has the specified claim ID, and that claim has not expired. This is useful for ensuring only one agent processes any given message; whenever a worker client's claim expires before it has a chance to delete a message it has processed, the worker must roll back any actions it took based on that message, since another worker will now be able to claim and process the same message.
 
<code><nowiki>claim_id</nowiki></code> specifies that the message should only be deleted if it has the specified claim ID, and that claim has not expired. This is useful for ensuring only one agent processes any given message; whenever a worker client's claim expires before it has a chance to delete a message it has processed, the worker must roll back any actions it took based on that message, since another worker will now be able to claim and process the same message.
  
 +
Note that if claim_id is not specified, but the message is claimed, the operation will fail. In other words, claimed messages can only ever be deleted by providing an appropriate claim_id.
  
== Claim Messages ==
+
=== Delete a Set of Messages by ID ===
 +
 
 +
'''Template'''
 +
 
 +
<pre><nowiki>
 +
DELETE /v1/queues/{queue_name}/messages{?ids}
 +
</nowiki></pre>
 +
 
 +
 
 +
'''Request'''
 +
 
 +
<pre><nowiki>
 +
DELETE /v1/queues/fizbit/messages?ids=50b68a50d6f5b8c8a7c62b01,50b68a50d6f5b8c8a7c62b02 HTTP/1.1
 +
Host: marconi.example.com
 +
 
 +
...
 +
 
 +
</nowiki></pre>
 +
 
 +
 
 +
'''Response'''
 +
 
 +
<pre><nowiki>
 +
HTTP/1.1 204 No Content
 +
</nowiki></pre>
 +
 
 +
 
 +
'''Discussion'''
 +
 
 +
Bulk delete for messages.
 +
 
 +
<code><nowiki>ids</nowiki></code> specifies the messages to delete, up to a maximum of 20 (configurable, shares the same setting as posting and listing messages)
 +
 
 +
If any of the message IDs are malformed or non-existent, they are ignored. The remaining valid message IDs are deleted.
 +
 
 +
== Claims ==
 +
 
 +
=== Claim Messages ===
  
  
Line 775: Line 1,018:
  
 
<pre><nowiki>
 
<pre><nowiki>
POST {base_url}/queues/{queue_name}/claims{?limit}
+
POST /v1/queues/{queue_name}/claims{?limit}
 
Content-Type: application/json
 
Content-Type: application/json
  
[...]
+
...
  
 
{  
 
{  
 
     "ttl": {claim_ttl},
 
     "ttl": {claim_ttl},
     "grace": {message_grace},
+
     "grace": {message_grace}
 
}
 
}
 
</nowiki></pre>
 
</nowiki></pre>
Line 790: Line 1,033:
  
 
<pre><nowiki>
 
<pre><nowiki>
POST /v1/480924/queues/fizbit/claims?limit=5 HTTP/1.1
+
POST /v1/queues/fizbit/claims?limit=5 HTTP/1.1
 
Content-Type: application/json
 
Content-Type: application/json
 
Accept: application/json
 
Accept: application/json
  
[...]
+
...
  
 
{  
 
{  
Line 805: Line 1,048:
 
'''Response'''
 
'''Response'''
  
The client receives a claim ID and a list of claimed messages, if any:
+
The client receives a claim URI and a list of claimed messages, if any:
  
 
<pre><nowiki>
 
<pre><nowiki>
HTTP/1.1 200 OK
+
HTTP/1.1 201 Created
 
Content-Type: application/json
 
Content-Type: application/json
Location: /v1/12345/queues/foo-bar/claims/a28ee94e-6cb4-11e2-b4d5-7703267a7926
+
Location: /v1/queues/foo-bar/claims/a28ee94e-6cb4-11e2-b4d5-7703267a7926
  
[...]
+
...
  
{
+
[
  "id": "a28ee94e-6cb4-11e2-b4d5-7703267a7926",
+
  {
  "age": 0,
+
    "href": "/v1/queues/foo-bar/messages/50b68a50d6f5b8c8a7c62b01?claim_id=a28ee94e-6cb4-11e2-b4d5-7703267a7926",
   "ttl": 600,
+
    "ttl": 800,
  "messages": [
+
    "age": 100,
     ...
+
    "body": {
   ]
+
      "object_id": "8a50d6",
}
+
      "target": "h.264"
 +
    }
 +
  },
 +
   {
 +
    "href": "/v1/queues/foo-bar/messages/50b68a50d6f5b8c8a7c62b02?claim_id=a28ee94e-6cb4-11e2-b4d5-7703267a7926",
 +
    "ttl": 800,
 +
    "age": 790,
 +
     "body": {
 +
      "object_id": "fb8c8a",
 +
      "target": "h.264"
 +
    }
 +
   }
 +
]
 
</nowiki></pre>
 
</nowiki></pre>
  
Line 827: Line 1,082:
 
'''Discussion'''
 
'''Discussion'''
  
Claims a set of messages, up to limit, from oldest to newest, skipping any that are already claimed.
+
Claims a set of messages, up to limit, from oldest to newest, skipping any that are already claimed. If no unclaimed messages are available, returns 204 No Content.
  
The client should delete the message when it has finished processing it, before the claim expires, to ensure the message is processed only once. As part of the delete operation, all worker clients should specify the claim ID. That way, the server can return an error if the claim just expired (notifying the client of the race condition), giving the worker a chance to roll back its own processing of the given message, since another worker will likely claim the message and process it.
+
The client should delete the message when it has finished processing it, before the claim expires, to ensure the message is processed only once. As part of the delete operation, all worker clients should specify the claim ID (this is best done by simply using the provided href). That way, the server can return an error if the claim just expired (notifying the client of the race condition), giving the worker a chance to roll back its own processing of the given message, since another worker will likely claim the message and process it.
  
 
Just as with a message's age, the age given for the claim is relative to the server's clock, and is useful for determining how quickly messages are getting processed, and whether a given message's claim is about to expire.
 
Just as with a message's age, the age given for the claim is relative to the server's clock, and is useful for determining how quickly messages are getting processed, and whether a given message's claim is about to expire.
Line 835: Line 1,090:
 
When a claim expires, it is removed, allowing another client worker to claim the message in the case that the original worker fails to process it.
 
When a claim expires, it is removed, allowing another client worker to claim the message in the case that the original worker fails to process it.
  
<code><nowiki>limit</nowiki></code> specifies up to 100 messages to claim. If not specified, limit defaults to 10.
+
<code><nowiki>limit</nowiki></code> specifies up to 20 messages (configurable) to claim. If not specified, limit defaults to 10. Note that claim creation is best-effort, meaning the server may claim and return less than the requested number of messages.
  
<code><nowiki>ttl</nowiki></code> is how long the server should wait before releasing the claim. Value MUST be between 60 and 43200 seconds (12 hours).
+
<code><nowiki>ttl</nowiki></code> is how long the server should wait before releasing the claim. Value MUST be between 60 and 43200 seconds (12 hours, configurable).
  
<code><nowiki>grace</nowiki></code> is the message grace period in seconds. Value MUST be between 60 and 43200 seconds (12 hours). The server will extend the lifetime of claimed messages to be at least as long as the lifetime of the claim itself, plus a specified grace period to deal with crashed workers (up to 1209600 or 14 days including claim lifetime). If a claimed message would normally live longer than the grace period, it's expiration will not be adjusted.
+
<code><nowiki>grace</nowiki></code> is the message grace period in seconds. Value MUST be between 60 and 43200 seconds (12 hours, configurable). The server will extend the lifetime of claimed messages to be at least as long as the lifetime of the claim itself, plus a specified grace period to deal with crashed workers (up to 1209600 or 14 days including claim lifetime). If a claimed message would normally live longer than the grace period, it's expiration will not be adjusted.
  
== Query Claim ==
+
=== Query Claim ===
  
  
Line 847: Line 1,102:
  
 
<pre><nowiki>
 
<pre><nowiki>
GET {base_url}/queues/{queue_name}/claims/{claim_id}
+
GET /v1/queues/{queue_name}/claims/{claim_id}
 
</nowiki></pre>
 
</nowiki></pre>
  
Line 854: Line 1,109:
  
 
<pre><nowiki>
 
<pre><nowiki>
GET /v1/12345/queues/foo-bar/claims/a28ee94e-6cb4-11e2-b4d5-7703267a7926
+
GET /v1/queues/foo-bar/claims/a28ee94e-6cb4-11e2-b4d5-7703267a7926
 
Host: marconi.example.com
 
Host: marconi.example.com
  
[...]
+
...
  
 
</nowiki></pre>
 
</nowiki></pre>
Line 866: Line 1,121:
 
<pre><nowiki>
 
<pre><nowiki>
 
HTTP/1.1 200 OK
 
HTTP/1.1 200 OK
Content-Location: /v1/12345/queues/foo-bar/claims/a28ee94e-6cb4-11e2-b4d5-7703267a7926
+
Content-Location: /v1/queues/foo-bar/claims/a28ee94e-6cb4-11e2-b4d5-7703267a7926
  
[...]
+
...
  
 
{
 
{
  "id": "a28ee94e-6cb4-11e2-b4d5-7703267a7926",
 
 
   "age": 19,
 
   "age": 19,
 
   "ttl": 30,
 
   "ttl": 30,
Line 883: Line 1,137:
 
'''Discussion'''
 
'''Discussion'''
  
 
+
=== Update Claim ===
== Update Claim ==
 
  
  
Line 890: Line 1,143:
  
 
<pre><nowiki>
 
<pre><nowiki>
PATCH {base_url}/queues/{queue_name}/claims/{claim_id}
+
PATCH /v1/queues/{queue_name}/claims/{claim_id}
 
Content-Type: application/json
 
Content-Type: application/json
 
</nowiki></pre>
 
</nowiki></pre>
Line 898: Line 1,151:
  
 
<pre><nowiki>
 
<pre><nowiki>
PATCH /v1/480924/queues/fizbit/claims/a28ee94e-6cb4-11e2-b4d5-7703267a7926 HTTP/1.1
+
PATCH /v1/queues/fizbit/claims/a28ee94e-6cb4-11e2-b4d5-7703267a7926 HTTP/1.1
 
Content-Type: application/json
 
Content-Type: application/json
[...]
+
 
 +
...
  
 
{ "ttl": 300 }
 
{ "ttl": 300 }
Line 917: Line 1,171:
 
Clients should periodically renew claims during long-running batches of work to avoid loosing a claim in the middle of processing a message. This is done by issuing a PATCH to a specific claim resource and including a new TTL for the claim (which may be different from the original TTL). The server will reset the age of the claim and apply the new TTL.
 
Clients should periodically renew claims during long-running batches of work to avoid loosing a claim in the middle of processing a message. This is done by issuing a PATCH to a specific claim resource and including a new TTL for the claim (which may be different from the original TTL). The server will reset the age of the claim and apply the new TTL.
  
== Release Claim ==
+
=== Release Claim ===
  
  
Line 923: Line 1,177:
  
 
<pre><nowiki>
 
<pre><nowiki>
DELETE {base_url}/queues/{queue_name}/claims/{claim_id}
+
DELETE /v1/queues/{queue_name}/claims/{claim_id}
 
</nowiki></pre>
 
</nowiki></pre>
  
Line 930: Line 1,184:
  
 
<pre><nowiki>
 
<pre><nowiki>
DELETE /v1/12345/queues/foo-bar/claims/a28ee94e-6cb4-11e2-b4d5-7703267a7926 HTTP/1.1
+
DELETE /v1/queues/foo-bar/claims/a28ee94e-6cb4-11e2-b4d5-7703267a7926 HTTP/1.1
 
Host: marconi.example.com
 
Host: marconi.example.com
 
</nowiki></pre>
 
</nowiki></pre>
Line 947: Line 1,201:
  
 
This operation is useful when a worker is performing a graceful shutdown, fails to process one or more messages, or is taking longer than expected to process messages, and wishes to make the remainder available to other workers.
 
This operation is useful when a worker is performing a graceful shutdown, fails to process one or more messages, or is taking longer than expected to process messages, and wishes to make the remainder available to other workers.
 
 
== Check Node Health ==
 
 
 
'''Template'''
 
 
<pre><nowiki>
 
GET {base_url}/health
 
</nowiki></pre>
 
 
or
 
 
<pre><nowiki>
 
HEAD {base_url}/health
 
</nowiki></pre>
 
 
 
'''Request'''
 
 
<pre><nowiki>
 
GET /v1/480924/health HTTP/1.1
 
Host: example.marconi.com
 
 
[...]
 
 
</nowiki></pre>
 
 
 
'''Response'''
 
 
<pre><nowiki>
 
HTTP/1.1 204 No Content
 
</nowiki></pre>
 
 
or
 
 
<pre><nowiki>
 
HTTP/1.1 503 Service Unavailable
 
</nowiki></pre>
 
 
 
 
'''Discussion'''
 
 
Use this request to check on the Marconi node status. If the node is down, this endpoint will not be reachable and will act as a signal to the load balancer to take the node out of rotation. Alternatively, the endpoint may be reachable but the service is temporarily unavailable due to storage driver failure or some other error.
 
 
 
== TBD ==
 
 
 
'''Template'''
 
 
<pre><nowiki>
 
GET /{version}/{tenant}
 
</nowiki></pre>
 
 
 
'''Request'''
 
 
<pre><nowiki>
 
GET /{version}/{tenant} HTTP/1.1
 
</nowiki></pre>
 
 
 
'''Response'''
 
 
<pre><nowiki>
 
@todo
 
</nowiki></pre>
 
 
 
'''Discussion'''
 
 
@todo
 

Latest revision as of 19:06, 15 September 2014

Marconi API: v1 Blueprint

The Marconi v1 API is FROZEN

We have started collecting feedback with an eye toward extensions and v2 of the API here: Marconi/specs/api/next


Overview

Marconi provides an HTTP-based API in the spirit of the REST architecture style.

This guide assumes the reader is familiar with REST, HTTP/1.1 and JSON. URI templates use the same syntax as RFC 6570.

Note: Error responses are enumerated on a separate page: Marconi/specs/api/v1/errors


Common API Elements

Clients

  • Clients should follow HTTP redirects
  • Clients should advertise gzip support
  • Clients must identify themselves in the UserAgent request header; e.g., "User-Agent: python/2.7 cloudthing/1.2"
  • Clients must not hard-code URI paths or templates since they may change over time. Instead, clients should cache links and link templates provided by the API.


API Versioning

The Marconi API uses a URI versioning scheme. The first element of the path contains the target version identifier, e.g.:

  https://marconi.example.com/v1

The URI version will only be incremented to accommodate major new features or API redesigns that can not be made backwards-compatible. When new API versions are released, older versions are deprecated. Marconi maintainers will work with developers and partners to ensure there is adequate time to migrate to new versions before deprecated ones are discontinued.

Since the version is only incremented to accommodate major API revisions, sub-versioning of the API will be rare, but may be used occasionally to release a "polished" major version.

Resource Media Types

The API supports `application/json` (XML is on the road map), encoded as UTF-8. Gzip'd requests are optionally supported by the server.

Unrecognized protocol elements should be ignored. This includes XML elements and attributes, JSON object properties, link relation types, media types, etc.


Authentication

May be optionally implemented by middleware or reverse proxy. If implemented, the following applies:

All requests to the API may only be performed by an authenticated agent.

To authenticate, an agent issues an authentication request to a Keystone Identity Service endpoint.

In response to valid credentials, Keystone responds with an auth token and a service catalog that contains a list of all services and endpoints available for the given token. Multiple endpoints may be returned for Marconi according to physical locations and performance/availability characteristics of different deployments.

Normally, Keystone middleware provides the X-Project-Id header based on the auth token submitted by the Marconi client. For this to work, clients MUST specify a valid auth token in the `X-Auth-Token` header for each request to the Marconi API. The API validates auth tokens against Keystone before servicing each request

If auth is not enabled, clients must provide the X-Project-Id header themselves.

Authorization

May be optionally implemented by middleware or reverse proxy. If implemented, the following applies:

The API needs to verify read/write access to all endpoints according to the provided auth token (RBAC). The ACL should be cached with the token.


Endpoints

Clients may choose to interact with any of the service endpoints retrieved from Keystone. Agents use the given endpoint as a home document href from which to discover links to all other API requests. Endpoints may be organized by region

An example endpoint:

https://marconi.example.com/v1

Errors

If any request results in an error, the server will return an appropriate 4xx or 5xx HTTP status code, and optionally the following information in the body:

  • Title
  • Description
  • Internal code
  • Link to more information


Example:

HTTP/1.1 400 Bad Request
Content-Type: application/json

{
  "title": "Unsupported limit",
  "description": "The given limit cannot be negative, and cannot be greater than 50.",
  "code": 1092,
  "link": {
    "rel": "help",
    "href": "http://docs.example.com/messages#limit",
    "text": "API documentation for the limit parameter"
  }
}

Common Headers

Each request to the API must include certain standard and extended HTTP headers. These headers provide host, agent, authentication and other pertinent information to the server.

Header Description
Host The host name of the API, as referenced by the Keystone service catalog
Date The current date and time, using the standard RFC 1123 HTTP date format
Accept Media type desired; only application/json is supported at this time.
Accept-Encoding Specifies that the agent accepts gzip-encoded response bodies
Content-Length For POST or PUT requests, the length in bytes of the message document being submitted
X-Auth-Token Keystone auth token
X-Project-Id An ID for a project to which the value of X-Auth-Token grants access. Queues will be created under this project.
Client-ID A UUID, used to distinguish each client using the service. The UUID must be submitted in its canonical form (e.g., 3381af92-2b9e-11e3-b191-71861300734c). In Marconi, this is used to avoid echoing a sender's messages back to the same instance, and may be logged by the server for future use. Should be generated once and persisted between restarts of the client.

HTTP Response Codes

See Response Codes

Sample API Request

GET /v1/queues/fizbat/messages?marker=1355-237242-783&limit=10 HTTP/1.1
Host: marconi.example.com
User-Agent: python/2.7 killer-rabbit/1.2
Date: Wed, 28 Nov 2012 21:14:19 GMT
Accept: application/json
Accept-Encoding: gzip
X-Auth-Token: 7d2f63fd-4dcc-4752-8e9b-1d08f989cc00
X-Project-Id: 518b51ea133c4facadae42c328d6b77b
Client-ID: 30387f00-39a0-11e2-be4d-a8d15f34bae2

Endpoints Synopsis

# Base endpoints
GET /v1
GET /v1/health

# Queues
GET /v1/queues{?marker,limit,detailed}
GET /v1/queues/{queue_name}  # existence check
HEAD /v1/queues/{queue_name}  # existence check
PUT /v1/queues/{name}
DELETE /v1/queues/{name}

# Queue metadata
PUT /v1/queues/{queue_name}/metadata
GET /v1/queues/{queue_name}/metadata
GET /v1/queues/{queue_name}/stats

# Messages
GET /v1/queues/{queue_name}/messages{?marker,limit,echo,include_claimed}
GET /v1/queues/{queue_name}/messages/{message_id}
GET /v1/queues/{queue_name}/messages{?ids}
POST /v1/queues/{queue_name}/messages
DELETE /v1/queues/{queue_name}/messages/{message_id}{?claim_id}
DELETE /v1/queues/{queue_name}/messages{?ids}

# Claims
POST /v1/queues/{queue_name}/claims{?limit}
GET /v1/queues/{queue_name}/claims/{claim_id}
PATCH /v1/queues/{queue_name}/claims/{claim_id}
DELETE /v1/queues/{queue_name}/claims/{claim_id}

Base Endpoints

Get Home Document

Template

GET /v1


Request

GET /v1 HTTP/1.1
Host: marconi.example.com

...


Response

HTTP/1.0 200 OK

...

{
    "resources": {
        "rel/queue-stats": {
            "href-template": "/v1/queues/{queue_name}/stats", 
            "href-vars": {
                "queue_name": "param/queue_name"
            }, 
            "hints": {
                "allow": [
                    "GET"
                ], 
                "formats": {
                    "application/json": {}
                }
            }
        }, 
        "rel/post-messages": {
            "href-template": "/v1/queues/{queue_name}/messages", 
            "href-vars": {
                "queue_name": "param/queue_name"
            }, 
            "hints": {
                "accept-post": [
                    "application/json"
                ], 
                "allow": [
                    "POST"
                ], 
                "formats": {
                    "application/json": {}
                }
            }
        }, 
        "rel/queue": {
            "href-template": "/v1/queues/{queue_name}", 
            "href-vars": {
                "queue_name": "param/queue_name"
            }, 
            "hints": {
                "allow": [
                    "GET", 
                    "HEAD", 
                    "PUT", 
                    "DELETE"
                ], 
                "formats": {
                    "application/json": {}
                }
            }
        }, 
        "rel/queue-metadata": {
            "href-template": "/v1/queues/{queue_name}/metadata", 
            "href-vars": {
                "queue_name": "param/queue_name"
            }, 
            "hints": {
                "allow": [
                    "GET", 
                    "PUT"
                ], 
                "formats": {
                    "application/json": {}
                }
            }
        }, 
        "rel/queues": {
            "href-template": "/v1/queues{?marker,limit,detailed}", 
            "href-vars": {
                "marker": "param/marker", 
                "detailed": "param/detailed", 
                "limit": "param/queue_limit"
            }, 
            "hints": {
                "allow": [
                    "GET"
                ], 
                "formats": {
                    "application/json": {}
                }
            }
        }, 
        "rel/messages": {
            "href-template": "/v1/queues/{queue_name}/messages{?marker,limit,echo,include_claimed}", 
            "href-vars": {
                "marker": "param/marker", 
                "include_claimed": "param/include_claimed", 
                "queue_name": "param/queue_name", 
                "limit": "param/messages_limit", 
                "echo": "param/echo"
            }, 
            "hints": {
                "allow": [
                    "GET"
                ], 
                "formats": {
                    "application/json": {}
                }
            }
        }, 
        "rel/claim": {
            "href-template": "/v1/queues/{queue_name}/claims{?limit}", 
            "href-vars": {
                "queue_name": "param/queue_name", 
                "limit": "param/claim_limit"
            }, 
            "hints": {
                "accept-post": [
                    "application/json"
                ], 
                "allow": [
                    "POST"
                ], 
                "formats": {
                    "application/json": {}
                }
            }
        }
    }
}


Discussion

The entire Marconi API is discoverable from a single starting point; agents do not need to know any more than this one URI in order to explore the entire API.

This document is cache-able (and should be cached by proxy and client).

See also: http://tools.ietf.org/html/draft-nottingham-json-home-02

Check Node Health

Template

GET /v1/health

or

HEAD /v1/health


Request

GET /v1/health HTTP/1.1
Host: example.marconi.com

...


Response

HTTP/1.1 204 No Content

or

HTTP/1.1 503 Service Unavailable


Discussion

Use this request to check on the Marconi node status. If the node is down, this endpoint will not be reachable and will act as a signal to the load balancer to take the node out of rotation. Alternatively, the endpoint may be reachable but the service is temporarily unavailable due to storage driver failure or some other error.

Queues

Create Queue

Template

PUT /v1/queues/{queue_name}

Request

PUT /v1/queues/fizbat HTTP/1.1
Host: marconi.example.com


Response

HTTP/1.1 201 Created
Location: /v1/queues/fizbit


Discussion

Creates a queue.

The body of the PUT is empty.

queue_name is the name to give the queue. The name MUST NOT exceed 64 bytes in length, and is limited to US-ASCII letters, digits, underscores and hyphens.

List Queues

Template

GET /v1/queues{?marker,limit,detailed}


Request

GET /v1/queues?marker=baz&detailed=true HTTP/1.1
Host: marconi.example.com

...


Response

HTTP/1.1 200 OK

{
  "links": [
    {
      "rel": "next",
      "href": "/v1/queues?marker=kooleo&limit=10&detailed=true"
    }
  ],
  "queues": [
    { "name": "boomerang", "href": "/v1/queues/boomerang", "metadata": {} },
    { "name": "fizbit", "href": "/v1/queues/fizbit", "metadata": { "handle": "@kgriffs" } },

    ...

    { "name": "kooleo",  "href": "/v1/queues/kooleo", "metadata": { "something": "something_else" } }
  ]
}

Checking Queue Existence

Template

GET /v1/queues/{queue_name}


Request

GET /v1/queues/{queue_name}
Host: marconi.example.com


Response

HTTP/1.1 204 OK


Discussion

Returns 204 if the queue exists, else 404.

HEAD also works.

Delete Queue

Template

DELETE /v1/queues/{queue_name}


Request

DELETE /v1/queues/fizbat HTTP/1.1
Host: marconi.example.com


Response

HTTP/1.1 204 No Content


Discussion

Use this operation to immediately delete a queue along with all its messages (if any).

Queue Metadata

Set Queue Metadata

Template

PUT /v1/queues/{queue_name}/metadata

{
   ...
}


Request

PUT /v1/queues/fizbat/metadata HTTP/1.1
Host: marconi.example.com

...

{
   "key": {
       "key2": "value",
       "key3": [1, 2, 3, 4, 5]
    }
}


Response

HTTP/1.1 204 No Content


Discussion

This operation allows for setting structured metadata. If PUT is used, this document will replace the existing metadata document in it's entirety, so clients need to be careful they do not accidentally delete fields.

The body of the request is an arbitrary document which allows storing contextual information about the way a particular application should interact with the queue. The size of this body, in characters and including whitespace must be <= 64 KiB (configurable). The document MUST be valid JSON (Marconi will validate).

Get Queue Metadata

Template

GET /v1/queues/{queue_name}/metadata


Request

GET /v1/queues/fizbit/metadata HTTP/1.1
Host: marconi.example.com

...


Response HTTP/1.1 200 OK

{
   "key": {
       "key2": "value",
       "key3": [1, 2, 3, 4, 5]
    }
}


Discussion

Returns queue metadata.

Get Queue Stats

Template

GET /v1/queues/{queue_name}/stats


Request

GET /v1/queues/fizbit/stats HTTP/1.1
Host: marconi.example.com

...


Response HTTP/1.1 200 OK

{
  "messages": {
    "free": 146929,
    "claimed": 2409,
    "total": 149338,
    "oldest": {
        "href": "/v1/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b01",
        "age": 63,
        "created": "2013-08-12T20:44:55Z"
    },
    "newest": {
        "href": "/v1/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b01",
        "age": 12,
        "created": "2013-08-12T20:45:46Z"
    }
}


Discussion

Returns queue statistics, including how many messages are in the queue, broken out by status.

NOTE: If total is 0, then "oldest" and "newest" message stats are not included.

Messages

All message-related operations require Client-Id to be included in the headers. This is to ensure that messages are not echoed back to the client that posted them unless the client explicitly requests this.

List Messages

Template

GET /v1/queues/{queue_name}/messages{?marker,limit,echo,include_claimed}


Request

GET /v1/queues/fizbit/messages?marker=1355-237242-783&limit=10 HTTP/1.1
Host: marconi.example.com

...


Response

HTTP 200 if the query matched 1 or more messages, HTTP 204 otherwise (with no body).

HTTP/1.1 200 OK
Content-Location: /v1/queues/fizbit/messages?marker=1355-237242-783&limit=10

...

{
  "links": [
    {
      "rel": "next",
      "href": "/v1/queues/fizbit/messages?marker=6244-244224-783&limit=10"
    }
  ],
  "messages": [
    {
      "href": "/v1/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b01",
      "ttl": 800,
      "age": 790,
      "body": {
        "event": "ActivateAccount",
        "mode": "active"
      }
    }
  ]
}


Discussion

Message IDs and markers are opaque strings; clients should make no assumptions about their format or length. Furthermore, clients should assume there is no relationship between markers and message IDs (i.e., one cannot be derived from the other). This allows for a wide variety of storage driver implementations.

Results are ordered by age, oldest message first.

limit specifies up to 20 messages (configurable) to return. If not specified, limit defaults to 10. When more messages are available than can be returned in a single request, the client can pick up the next batch by simply using the URI template parameters returned from the previous call in the "next" field (TBD).

marker is an opaque string that the client can use to request the next batch of messages. It communicates to the server which messages the client has already received.

Note: If marker is not specified, the API will return all messages at the head of the queue (up to limit).

echo is a boolean (i.e., "true" or "false") that determines whether or not the API will return a client's own messages, as determined by the uuid portion of the Client-ID header. If not specified, echo defaults to "false".

include_claimed is a boolean (i.e., "true" or "false") that determines whether or not the API will return claimed messages as well as unclaimed ones. If not specified, defaults to "false" (only unclaimed messages are returned).

Get a Specific Message

Template

GET /v1/queues/{queue_name}/messages/{message_id}


Request

GET /v1/queues/fizbat/messages/50b68a50d6f5b8c8a7c62b01 HTTP/1.1
Host: marconi.example.com

...


Response

HTTP/1.1 200 OK
Content-Location: /v1/queues/fizbat/messages/50b68a50d6f5b8c8a7c62b01

...

{
  "href": "/v1/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b01",
  "ttl": 800,
  "age": 790,
  "body": {
    "event": "ActivateAccount",
    "mode": "active"
  }
}


Discussion

Message fields are defined as follows:

href is an opaque relative URI that the client can use to uniquely identify a message resource, and interact with it.

ttl is the ttl set on the message when it was posted. The message will expire after (ttl - age) seconds.

age number of seconds since ts, relative to the server's clock.

body Arbitrary document submitted along with the original request to post the message.

If either the message ID or the claim ID is malformed or nonexistent, no message is returned.

Get a Set of Messages by ID

Template

GET /v1/queues/{queue_name}/messages{?ids}


Request

GET /v1/queues/fizbat/messages?ids=50b68a50d6f5b8c8a7c62b01,f5b8c8a7c62b0150b68a50d6 HTTP/1.1
Host: marconi.example.com

...


Response

HTTP/1.1 200 OK
Content-Location: /v1/queues/fizbat/messages?ids=50b68a50d6f5b8c8a7c62b01,f5b8c8a7c62b0150b68a50d6

...

[
    {
      "href": "/v1/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b01",
      "ttl": 800,
      "age": 32,
      "body": {
        "cmd": "EncodeVideo",
        "jobid": 58229 
      }
    },
    {
      "href": "/v1/queues/fizbit/messages/f5b8c8a7c62b0150b68a50d6",
      "ttl": 800,
      "age": 790,
      "body": {
        "cmd": "EncodeAudio",
        "jobid": 58201 
      }
    }
]


Discussion

Unlike message listing, a client's own messages are always returned in this approach. Note that the list of ids may not exceed 20 (configurable, shared setting with number of messages that can be posted at once).

This approach returns all the messages with matching IDs. If a malformed ID or a nonexistent message ID is provided, it is ignored and the remaining messages are returned.

Post Message(s)

Template

POST /v1/queues/{queue_name}/messages


Request

POST /v1/queues/fizbit/messages HTTP/1.1
Host: marconi.example.com

...

[
  {
    "ttl": 300,
    "body": {
      "event": "BackupStarted",
      "backup_id": "c378813c-3f0b-11e2-ad92-7823d2b0f3ce"
    }
  },
  {
    "ttl": 60,
    "body": {
      "event": "BackupProgress",
      "current_bytes": "0",
      "total_bytes": "99614720"
    }
  }
]


Responses

When a single message is submitted:

HTTP/1.1 201 Created
Location: /v1/queues/fizbit/messages?ids=50b68a50d6f5b8c8a7c62b01
Content-Type: application/json

{
    "resources": [
        "/v1/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b01"
    ],
    "partial": false
}

When multiple messages are submitted:

HTTP/1.1 201 Created
Location: /v1/queues/fizbit/messages?ids=50b68a50d6f5b8c8a7c62b01,50b68a50d6f5b8c8a7c62b05
Content-Type: application/json

{
    "resources": [
        "/v1/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b01",
        "/v1/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b05"
    ],
    "partial": false
}

Discussion

Up to 20 messages (default, but configurable) may be submitted in a single request, but must always be encapsulated in a collection container (an array in JSON). The resulting value of the Location header or response body may be used to retrieve the created messages for further processing if needed.

The client only specifies the body and ttl for the message; the server will insert metadata such as id and age.

The response body contains a list of resource paths corresponding to each message submitted in the request, in the same order. In the case of a server-side error part-way through the processing of the submitted messages, a partial list will be returned, partial will be true, and the client will need to retry posting the remaining messages. If no messages could be enqueued, the server will return 503 Service Unavailable.

NOTE: The partial field has been deprecated since the icehouse release, and will always be "false"; drivers are not longer allowed to return a "partial" success and so must either succeed or fail the entire batch of messages.

The size of the request document, in characters including whitespace, is 256 KiB by default (configurable). The document MUST be valid JSON (Marconi will validate).

body specifies an arbitrary document which constitutes the body of the message being sent.

ttl is how long the server should wait before expiring and removing the message from the queue. Value MUST be between 60 and 1209600 seconds (14 days, configurable), inclusive. Note that the server may not actually delete the message until it's age has reached up to (ttl + 60) seconds, to allow for flexibility in storage implementations.

Delete Message

Template

DELETE /v1/queues/{queue_name}/messages/{message_id}{?claim_id}


Request

DELETE /v1/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b01 HTTP/1.1
Host: marconi.example.com

...


Response

HTTP/1.1 204 No Content


Discussion

message_id specifies the message to delete.

claim_id specifies that the message should only be deleted if it has the specified claim ID, and that claim has not expired. This is useful for ensuring only one agent processes any given message; whenever a worker client's claim expires before it has a chance to delete a message it has processed, the worker must roll back any actions it took based on that message, since another worker will now be able to claim and process the same message.

Note that if claim_id is not specified, but the message is claimed, the operation will fail. In other words, claimed messages can only ever be deleted by providing an appropriate claim_id.

Delete a Set of Messages by ID

Template

DELETE /v1/queues/{queue_name}/messages{?ids}


Request

DELETE /v1/queues/fizbit/messages?ids=50b68a50d6f5b8c8a7c62b01,50b68a50d6f5b8c8a7c62b02 HTTP/1.1
Host: marconi.example.com

...


Response

HTTP/1.1 204 No Content


Discussion

Bulk delete for messages.

ids specifies the messages to delete, up to a maximum of 20 (configurable, shares the same setting as posting and listing messages)

If any of the message IDs are malformed or non-existent, they are ignored. The remaining valid message IDs are deleted.

Claims

Claim Messages

Template

POST /v1/queues/{queue_name}/claims{?limit}
Content-Type: application/json

...

{ 
    "ttl": {claim_ttl},
    "grace": {message_grace}
}


Request

POST /v1/queues/fizbit/claims?limit=5 HTTP/1.1
Content-Type: application/json
Accept: application/json

...

{ 
    "ttl": 300,
    "grace": 300
}


Response

The client receives a claim URI and a list of claimed messages, if any:

HTTP/1.1 201 Created
Content-Type: application/json
Location: /v1/queues/foo-bar/claims/a28ee94e-6cb4-11e2-b4d5-7703267a7926

...

[
  {
    "href": "/v1/queues/foo-bar/messages/50b68a50d6f5b8c8a7c62b01?claim_id=a28ee94e-6cb4-11e2-b4d5-7703267a7926",
    "ttl": 800,
    "age": 100,
    "body": {
      "object_id": "8a50d6",
      "target": "h.264"
    }
  },
  {
    "href": "/v1/queues/foo-bar/messages/50b68a50d6f5b8c8a7c62b02?claim_id=a28ee94e-6cb4-11e2-b4d5-7703267a7926",
    "ttl": 800,
    "age": 790,
    "body": {
      "object_id": "fb8c8a",
      "target": "h.264"
    }
  }
]


Discussion

Claims a set of messages, up to limit, from oldest to newest, skipping any that are already claimed. If no unclaimed messages are available, returns 204 No Content.

The client should delete the message when it has finished processing it, before the claim expires, to ensure the message is processed only once. As part of the delete operation, all worker clients should specify the claim ID (this is best done by simply using the provided href). That way, the server can return an error if the claim just expired (notifying the client of the race condition), giving the worker a chance to roll back its own processing of the given message, since another worker will likely claim the message and process it.

Just as with a message's age, the age given for the claim is relative to the server's clock, and is useful for determining how quickly messages are getting processed, and whether a given message's claim is about to expire.

When a claim expires, it is removed, allowing another client worker to claim the message in the case that the original worker fails to process it.

limit specifies up to 20 messages (configurable) to claim. If not specified, limit defaults to 10. Note that claim creation is best-effort, meaning the server may claim and return less than the requested number of messages.

ttl is how long the server should wait before releasing the claim. Value MUST be between 60 and 43200 seconds (12 hours, configurable).

grace is the message grace period in seconds. Value MUST be between 60 and 43200 seconds (12 hours, configurable). The server will extend the lifetime of claimed messages to be at least as long as the lifetime of the claim itself, plus a specified grace period to deal with crashed workers (up to 1209600 or 14 days including claim lifetime). If a claimed message would normally live longer than the grace period, it's expiration will not be adjusted.

Query Claim

Template

GET /v1/queues/{queue_name}/claims/{claim_id}


Request

GET /v1/queues/foo-bar/claims/a28ee94e-6cb4-11e2-b4d5-7703267a7926
Host: marconi.example.com

...


Response

HTTP/1.1 200 OK
Content-Location: /v1/queues/foo-bar/claims/a28ee94e-6cb4-11e2-b4d5-7703267a7926

...

{
  "age": 19,
  "ttl": 30,
  "messages": [
    ...
  ]
}


Discussion

Update Claim

Template

PATCH /v1/queues/{queue_name}/claims/{claim_id}
Content-Type: application/json


Request

PATCH /v1/queues/fizbit/claims/a28ee94e-6cb4-11e2-b4d5-7703267a7926 HTTP/1.1
Content-Type: application/json

...

{ "ttl": 300 }


Response

HTTP/1.1 204 No Content


Discussion

Clients should periodically renew claims during long-running batches of work to avoid loosing a claim in the middle of processing a message. This is done by issuing a PATCH to a specific claim resource and including a new TTL for the claim (which may be different from the original TTL). The server will reset the age of the claim and apply the new TTL.

Release Claim

Template

DELETE /v1/queues/{queue_name}/claims/{claim_id}


Request

DELETE /v1/queues/foo-bar/claims/a28ee94e-6cb4-11e2-b4d5-7703267a7926 HTTP/1.1
Host: marconi.example.com


Response

HTTP/1.1 204 No Content


Discussion

Use this operation to immediately release a claim, making any (remaining, non-deleted) messages associated with the claim available to other workers.

This operation is useful when a worker is performing a graceful shutdown, fails to process one or more messages, or is taking longer than expected to process messages, and wishes to make the remainder available to other workers.