Jump to: navigation, search

Zaqar/specs/api/v1

< Zaqar
Revision as of 14:44, 21 March 2013 by Kgriffs (talk | contribs) (Added ttl to messages in get, list)

Marconi API: v1 Blueprint

NOTE: For the latest changes under discussion (but not yet approved) see the etherpad working draft:

https://etherpad.openstack.org/queuing-api

General requirements: marconi/specs/grizzly

To Do

  • Specify all possible error responses for each type of request
  • Define and register message media type/ messages media type - http://www.iana.org/assignments/media-types/index.html
  • Flesh out examples, incomplete sections (obviously)
  • Clean up this document, add a FAQ section


Overview

Marconi provides an HTTP-based API in the spirit of the REST architecture style.

This guide assumes the reader is familiar with REST, HTTP/1.1 and JSON. URI templates use the same syntax as RFC 6570.


Common API Elements

HTTPS

All requests to authenticate and operate against the API in production environments use SSL/TLS over HTTP (HTTPS) on TCP port 443. Web servers should only accept high-quality cipher suites, such as AES256_SHA and ECDHE_RSA_AES256_SHA. If hardware acceleration (e.g., AES_NI) is not available, RC4_SHA and ECDHE_RSA_RC4_SHA may be used with some discretion.


Clients

  • Clients should follow HTTP redirects
  • Clients should advertise gzip support
  • Clients must identify themselves in the UserAgent request header; e.g., "User-Agent: python/2.7 cloudthing/1.2"
  • Clients must not hard-code URI paths or templates since they may change over time. Instead, clients should cache links and link templates provided by the API.


API Versioning

The Marconi API uses a URI versioning scheme. The first element of the path contains the target version identifier, e.g.:

  https://marconi.example.com/v1

The URI version will only be incremented to accommodate major new features or API redesigns that can not be made backwards-compatible. When new API versions are released, older versions are deprecated. Marconi maintainers will work with developers and partners to ensure there is adequate time to migrate to new versions before deprecated ones are discontinued.

Since the base URI is only incremented to accommodate major API revisions, sub-versioning of the API is meaningless and is therefore not used. For example, the next version after "v1" would be "v2", not "v1.1" or some variant thereof. ([HATEOS and media types|http://www.informit.com/articles/article.aspx?p=1566460] are used in lieu of minor, URI-based versioning.)


Resource Media Types

The API supports `application/json` (XML is on the road map), encoded as UTF-8. Gzip'd requests are optionally supported by the server.

Unrecognized protocol elements should be ignored. This includes XML elements and attributes, JSON object properties, link relation types, media types, etc.


Authentication

All requests to the API may only be performed by an authenticated agent.

To authenticate, an agent issues an authentication request to a Keystone Identity Service endpoint.

In response to valid credentials, Keystone responds with an auth token and a service catalog that contains a list of all services and endpoints available for the given token. Multiple endpoints may be returned for Marconi according to physical locations and performance/availability characteristics of different deployments.

Marconi clients must specify a valid auth token in the `X-Auth-Token` header for each request to the Marconi API. The API validates auth tokens against Keystone before servicing each request.


Authorization

The API needs to verify read/write access to all endpoints according to the provided auth token (RBAC). The ACL should be cached with the token.


Tenant ID

Auth tokens are only valid for a particular tenant ID, which should be reflected in the service endpoints retrieved from Keystone. Agents use the given endpoint as a home document href from which to discover links to all other API requests.

An example endpoint:

https://marconi.example.com/v1/480924

The client chooses one of the presented endpoints and uses it as the base URL for all subsequent requests.


Errors

If any request results in an error, the server will return an appropriate 4xx or 5xx HTTP status code, as well as the following information in the body:

  • Title
  • Description
  • Internal code
  • Link to more information


Example:

HTTP/1.1 400 Bad Request
Content-Type: application/json

{
  "title": "Unsupported limit",
  "description": "The given limit cannot be negative, and cannot be greater than 50.",
  "code": 1092,
  "link": {
    "rel": "help",
    "href": "http://docs.example.com/messages#limit",
    "text": "API documentation for the limit parameter"
  }
}


Common Headers

Each request to the API must include certain standard and extended HTTP headers. These headers provide host, agent, authentication and other pertinent information to the server.

Header Description
Host The host name of the API, as referenced by the Keystone service catalog
User-Agent The name and version of the Marconi client, as well as a UUID for that client. Marconi uses the UUID to distinguish publishers from subscribers, i.e., to avoid echoing an agent's own messages back to it.
Date The current date and time, using the standard RFC 1123 HTTP date format
Accept Media type desired; initially, only `application/json` will be supported
Accept-Encoding Specifies that the agent accepts gzip-encoded response bodies
Content-Length For POST or PUT requests, the length in bytes of the message document being submitted
X-Auth-Token Keystone auth token


Sample API Request

GET /v1/480924/queues/fizbat/messages?marker=1355-237242-783&limit=10 HTTP/1.1
Host: marconi.example.com
User-Agent: python/2.7 killer-rabbit/1.2 uuid/30387f00-39a0-11e2-be4d-a8d15f34bae2
Date: Wed, 28 Nov 2012 21:14:19 GMT
Accept: application/json
Accept-Encoding: gzip
X-Auth-Token: 7d2f63fd-4dcc-4752-8e9b-1d08f989cc00


Get Home Document

Template

GET /v1


Request

GET /v1 HTTP/1.1
Host: marconi.example.com

[...]


Response

@todo


Discussion

The entire Marconi API is discoverable from a single starting point; agents do not need to know any more than this one URI in order to explore the entire API.

This document is cache-able (and should be cached by proxy and client).

See also: http://tools.ietf.org/html/draft-nottingham-json-home-02


Create Queue

Template

PUT {base_url}/queues/{queue_name}

{
    ...
}

Request

PUT /v1/480924/queues/fizbat HTTP/1.1
Host: marconi.example.com

[...]

{
    "game_id": "a34904d6-8dbc-11e2-864b-5d361e90beb5"
}


Response

HTTP/1.1 201 Created
Location: /v1/480924/queues/fizbit


Discussion

Creates a queue and sets its metadata.

The body of the PUT is a document that can contain arbitrary metadata to be associated with the queue, up to 4 KiB when serialized as JSON, including whitespace. Top-level field names MUST NOT start with an underscore; such names are reserved for future use.


List Queues

Template

GET {base_url}/queues{?marker,limit,metadata}


Request

GET {base_url}/queues?marker=baz&metadata=true HTTP/1.1
Host: marconi.example.com

[...]


Response

HTTP/1.1 200 OK

{
  "links": [
    {
      "rel": "prev",
      "href": "/v1/480924/queues/fizbit/queues?marker=axon&limit=10"
    },
    {
      "rel": "next",
      "href": "/v1/480924/queues/fizbit/queues?marker=kooleo&limit=10"
    }
  ],
  "queues": [
    { "name": "boomerang", "href": "/v1/480924/queues/boomerang", "metadata": {} },
    { "name": "fizbit", href: "/v1/480924/queues/fizbit", "metadata": { "handle": "@kgriffs" } },

    [...]

    { "name": "kooleo",  "href": "/v1/480924/queues/kooleo", "metadata": { "something": "something_else" } }
  ]
}


Discussion

Lists queues belonging to the specified tenant, sorted alphabetically by name.

limit specifies up to 100 to return. If not specified, limit defaults to 10. Paging is supported via "prev" and "next" links.

marker is the name of the last queue recieved in a previous request, or none to get the first page of results.

metadata is a boolean ("true" or "false") that determines whether queue metadata is included in the response. Defaults to "false".


Set Queue Metadata

Template

PUT {base_url}/queues/{queue_name}

{
   ...
}


Request

PUT /v1/480924/queues/fizbat HTTP/1.1
Host: marconi.example.com

[...]

{
   "key": {
       "key2": "value",
       "key3": [1, 2, 3, 4, 5]
    }
}


Response

HTTP/1.1 204 No Content


Discussion

Set queue metadata. If PUT is used, this document will replace the existing metadata document in it's entirety, so clients need to be careful they do not accidentally delete fields.

PATCH support is a "future" or "if we have time" feature.


Get Queue Metadata

Template

GET {base_url}/queues/{queue_name}


Request

GET /v1/480924/queues/fizbit HTTP/1.1
Host: marconi.example.com

[...]


Response HTTP/1.1 200 OK

{
  "ttl": 86400,
}


Discussion

Returns queue metadata, such as message TTL.


Get Queue Stats

Template

GET {base_url}/queues/{queue_name}/stats


Request

GET /v1/480924/queues/fizbit/stats HTTP/1.1
Host: marconi.example.com

[...]


Response HTTP/1.1 200 OK

{
  "messages": 46,
  "actions": 122
}


Discussion

Returns queue statistics, including how many messages are in the queue, and how many actions have been recorded (actions can be retrieved by performing a GET on {queue_name}/actions).


Delete Queue

Template

DELETE {base_url}/queues/{queue_name}


Request

DELETE /v1/480924/queues/fizbat HTTP/1.1
Host: marconi.example.com


Response

HTTP/1.1 204 No Content


Discussion

Use this operation to immediately delete a queue along with all its messages (if any).


Get Messages

Template

GET {base_url}/queues/{queue_name}/messages/{?marker,limit,echo}


Request

GET /v1/480924/queues/fizbit/messages&marker=1355-237242-783&limit=10 HTTP/1.1
Host: marconi.example.com

[...]


Response

HTTP 200 if the query matched 1 or more messages, HTTP 204 otherwise (with no body).

HTTP/1.1 200 OK
Content-Location: /v1/480924/queues/fizbit/messages?marker=1355-237242-783&limit=10

[...]

{
  "links": [
    {
      "rel": "next",
      "href": "/v1/480924/queues/fizbit/messages?marker=6244-244224-783&limit=10"
    }
  ],
  "messages": [
    {
      "id": "50b68a50d6f5b8c8a7c62b01",
      "ttl": 800,
      "age": 790,
      "body": {
        "event": "ActivateAccount",
        "mode": "active"
      }
    }
  ]
}


Discussion

Message IDs and markers are opaque strings; clients should make no assumptions about their format or length. Furthermore, clients should assume there is no relationship between markers and message IDs (i.e., one cannot be derived from the other). This allows for a wide variety of storage driver implementations.

Results are ordered by age, oldest message first.

limit specifies up to 100 messages to return. If not specified, limit defaults to 10. When more messages are available than can be returned in a single request, the client can pick up the next batch by simply using the URI template parameters returned from the previous call in the "next" field (TBD).

marker is an opaque string that the client can use to request the next batch of messages. It communicates to the server which messages the client has already received.

Note: If marker is not specified, the API will return all messages at the head of the queue (up to limit).

echo is a boolean (i.e., "true" or "false") that determines whether or not the API will return a client's own messages, as determined by the uuid portion of the User-Agent header. If not specified, echo defaults to "false".


Get Actions

Template

GET {base_url}/queues/{queue_name}/actions/{?marker,limit}


Request

GET /v1/480924/queues/fizbit/actions&marker=1355-237242-783&limit=10 HTTP/1.1
Host: marconi.example.com

[...]


Response

HTTP 200 if the query matched 1 or more actions, HTTP 204 otherwise (with no body).

HTTP/1.1 200 OK
Content-Location: /v1/480924/queues/fizbit/actions?marker=1355-237242-783&limit=10

[...]

{
  "links": [
    {
      "rel": "next",
      "href": "/v1/480924/queues/fizbit/actions?marker=6244-244224-783&limit=10"
    }
  ],
  "messages": [
    {
      "id": 10b00a50d6f5b8c8a7c62ccb,
      "ttl": 140,
      "age": 120,
      "body": {
        "event": "LockExpired",
        "details": {
          "message_id": "50b68a50d6f5b8c8a7c62a87",
          "user_agent": "python/2.7 killer-rabbit/1.2 uuid/79ed56f8-2519-11e2-b835-acf6018e45a1",
        }
      }
    },
    {
      "id": 10b00a50d6f5b8c8a7c62ccc,
      "ttl": 140,
      "age": 121,
      "body": {
        "event": "MessageDeleted",
        "details": {
          "message_id": "50b68a50d6f5b8c8a7c62b01",
          "user_agent": "python/2.7 killer-rabbit/1.2 uuid/79ed56f8-2519-11e2-b835-acf6018e45a1",
        }
      }
    }
  ]
}


Discussion

The interactions of various agents/workers with a cloud queuing services can be difficult to audit and debug. Marconi emits action messages to a special queue, from which auditors can retrieve a list of recent actions involving a specific queue's messages. These actions can then be archived by the auditor for future analysis and diagnostics.

Action messages expire after 5 minutes, so auditors should regularly poll for actions and persist them for posterity.

The following actions are recorded, since they cannot be otherwise observed:

  • Delete message
  • Claim messages
  • Update claim
  • Release claim
  • Claim expired

Message IDs and markers are opaque strings; clients should make no assumptions on their format or length. Furthermore, clients should assume there is no relationship between markers and message IDs (i.e., one cannot be derived from the other). This ensures maximum flexibility when implementing storage drivers.

Results are always ordered by age, oldest message first.

limit specifies up to 100 messages to return. If not specified, limit defaults to 10. When more messages are available than can be returned in a single request, the client can pick up the next batch by simply using the URI template parameters returned from the previous call in the "next" field.

marker is an opaque string that the client can use to request the next batch of messages. It communicates to the server which messages the client has already received.

Note: If marker is not specified, the API will return all messages at the head of the queue (up to limit).


Get a Specific Message

Template

GET {base_url}/queues/{queue_name}/messages/{message_id}


Request

GET /v1/480924/queues/fizbat/messages/50b68a50d6f5b8c8a7c62b01 HTTP/1.1
Host: marconi.example.com

[...]


Response

HTTP/1.1 200 OK
Content-Location: /v1/480924/queues/fizbat/messages/50b68a50d6f5b8c8a7c62b01

[...]

{
  "id": "50b68a50d6f5b8c8a7c62b01",
  "ttl": 800,
  "age": 790,
  "body": {
    "event": "ActivateAccount",
    "mode": "active"
  }
}


Discussion

Message fields are defined as follows:

id is an opaque string that the client can use to uniquely identify the message.

ttl is the ttl set on the message when it was posted. The message will expire after (ttl - age) seconds.

age number of seconds since ts, relative to the server's clock.

body Arbitrary document submitted along with the original request to post the message.


Post Message(s)

Template

POST {base_url}/queues/{queue_name}/messages


Request

POST /v1/480924/queues/fizbit/messages HTTP/1.1
Host: marconi.example.com

[...]

[
  {
    "ttl": 300,
    "body": {
      "event": "BackupStarted",
      "backup_id": "c378813c-3f0b-11e2-ad92-7823d2b0f3ce"
    }
  },
  {
    "ttl": 60,
    "body": {
      "event": "BackupProgress",
      "current_bytes": "0",
      "total_bytes": "99614720"
    }
  }
]


Responses

When a single message is submitted:

HTTP/1.1 201 Created
Location: /v1/480924/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b01

When multiple messages are submitted:

HTTP/1.1 201 Created
Location: /v1/480924/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b01,2938472984,234092834,1230487

TODO(kgriffs): Define a URI template that allows for multiple IDs, also update Get Message(s) spec.


Discussion

Up to 100 messages may be submitted in a single request, but must always be encapsulated in a collection container (an array in JSON).

The client only specifies the body and ttl for the message; the server will insert metadata such as id and age.

body specifies an arbitrary document which constitutes the body of the message being sent. The size of this body, in characters and including whitespace, is 4 KiB. The document MUST be valid JSON (Marconi will validate).

ttl is how long the server should wait before expiring and removing the message from the queue. Value MUST between 60 and 1209600 seconds (14 days).


Delete Message

Template

DELETE {base_url}/queues/{queue_name}/messages/{message_id}{?claim_id}


Request

DELETE /v1/480924/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b01 HTTP/1.1
Host: marconi.example.com

[...]


Response

HTTP/1.1 204 No Content


Discussion

message_id specifies the message to delete.

claim_id specifies that the message should only be deleted if it has the specified claim ID, and that claim has not expired. This is useful for ensuring only one agent processes any given message; whenever a worker client's claim expires before it has a chance to delete a message it has processed, the worker must roll back any actions it took based on that message, since another worker will now be able to claim and process the same message.


Claim Messages

Template

POST {base_url}/queues/{queue_name}/claims{?limit}
Content-Type: application/json

[...]

{ "ttl": {claim_ttl} }


Request

POST /v1/480924/queues/fizbit/claims?limit=5 HTTP/1.1
Content-Type: application/json
Accept: application/json

[...]

{ "ttl": 300 }


Response

The client receives a claim ID and a list of claimed messages, if any:

HTTP/1.1 200 OK
Content-Type: application/json
Location: /v1/12345/queues/foo-bar/claims/a28ee94e-6cb4-11e2-b4d5-7703267a7926

[...]

{
  "id": "a28ee94e-6cb4-11e2-b4d5-7703267a7926",
  "age": 0,
  "ttl": 30,
  "messages": [
    ...
  ]
}


Discussion

Claims a set of messages, up to limit, from oldest to newest, skipping any that are already claimed.

WARNING: The claim's ttl should be significantly less than the minimum expected ttl of any message in the queue, to avoid losing messages in the middle of processing a claim. This will not be enforced by the server. Use the actions resource to diagnose problems in this area.

The client should delete the message when it has finished processing it, before the claim expires, to ensure the message is processed only once. As part of the delete operation, all worker clients should specify the claim ID. That way, the server can return an error if the claim just expired (notifying the client of the race condition), giving the worker a chance to roll back its own processing of the given message, since another worker will likely claim the message and process it.

Just as with a message's age, the age given for the claim is relative to the server's clock, and is useful for determining how quickly messages are getting processed, and whether a given message's claim is about to expire.

When a claim expires, it is removed, allowing another client worker to claim the message in the case that the original worker fails to process it.

limit specifies up to 100 messages to claim. If not specified, limit defaults to 10.


Query Claim

Template

GET {base_url}/queues/{queue_name}/claims/{claim_id}


Request

GET /v1/12345/queues/foo-bar/claims/a28ee94e-6cb4-11e2-b4d5-7703267a7926
Host: marconi.example.com

[...]


Response

HTTP/1.1 200 OK
Content-Location: /v1/12345/queues/foo-bar/claims/a28ee94e-6cb4-11e2-b4d5-7703267a7926

[...]

{
  "id": "a28ee94e-6cb4-11e2-b4d5-7703267a7926",
  "age": 19,
  "ttl": 30,
  "messages": [
    ...
  ]
}


Discussion


Update Claim

Template

PATCH {base_url}/queues/{queue_name}/claims/{claim_id}
Content-Type: application/json-patch


Request

PATCH /v1/480924/queues/fizbit/claims/a28ee94e-6cb4-11e2-b4d5-7703267a7926 HTTP/1.1
Content-Type: application/json-patch

[...]

[
  {
    "op": "replace", "path": "/ttl", "value": 60,
    "op": "replace", "path": "/age", "value": 0
  }
]


Response

HTTP/1.1 204 No Content


Discussion

Clients should periodically renew claims during long-running batches of work to avoid loosing a claim in the middle of processing a message. Clients may optionally change the claim's ttl as well.


Release Claim

Template

DELETE {base_url}/queues/{queue_name}/claims/{claim_id}


Request

DELETE /v1/12345/queues/foo-bar/claims/a28ee94e-6cb4-11e2-b4d5-7703267a7926 HTTP/1.1
Host: marconi.example.com


Response

HTTP/1.1 204 No Content


Discussion

Use this operation to immediately release a claim, making any (remaining, non-deleted) messages associated with the claim available to other workers.

This operation is useful when a worker is performing a graceful shutdown, fails to process one or more messages, or is taking longer than expected to process messages, and wishes to make the remainder available to other workers.


Check Node Health

Template

GET {base_url}/health

or

HEAD {base_url}/health


Request

GET /v1/480924/health HTTP/1.1
Host: example.marconi.com

[...]


Response

HTTP/1.1 204 No Content

or

HTTP/1.1 503 Service Unavailable


Discussion

Use this request to check on the Marconi node status. If the node is down, this endpoint will not be reachable and will act as a signal to the load balancer to take the node out of rotation. Alternatively, the endpoint may be reachable but the service is temporarily unavailable due to storage driver failure or some other error.


TBD

Template

GET /{version}/{tenant}


Request

GET /{version}/{tenant} HTTP/1.1


Response

@todo


Discussion

@todo