Jump to: navigation, search

Zaqar/specs/api/v1

< Zaqar
Revision as of 23:23, 5 December 2012 by Kgriffs (talk) (First draft, with lots of rough edges)

Marconi API: v1 Blueprint

See also: marconi/specs/grizzly

TODO: Compare/align this API with Swift and Nova.

Overview

Marconi provides an HTTP-based API in the spirit of the REST architecture style.

REST is a natural fit for Marconi's design philosophy, shamelessly borrowed from Donald A. Norman's work regarding The Design of Everyday Things:

 The value of a well-designed object is when it has such a rich set of affordances that the people who use it can do things with it that the designer never imagined.

This guide assumes the reader is familiar with REST, HTTP/1.1, JSON and XML. URI templates use the form specified in RFC 6570.

Common API Elements

HTTPS

All requests to authenticate and operate against the API in production environments use SSL/TLS over HTTP (HTTPS) on TCP port 443. Web servers should only accept high-quality cipher suites, such as AES256_SHA and ECDHE_RSA_AES256_SHA. If hardware acceleration (e.g., AES_NI) is not available, RC4_SHA and ECDHE_RSA_RC4_SHA may be used with some discretion.

Clients

  • Clients should follow HTTP redirects
  • Clients should advertise gzip support
  • Clients must identify themselves in the UserAgent request header; e.g., "User-Agent: python/2.7 cloudthing/1.2"
  • Clients must not hard-code URI paths or templates since they may change over time. Instead, clients should cache links and link templates provided by the API.

API Versioning

The Marconi API uses a URI versioning scheme. The first element of the path contains the target version identifier, e.g.:


  https://marconi.example.com/v1


he URI version will only be incrmented to accomodate major new features or API redesigns that can not be made backwards-compatible. When new API versions are released, older versions are deprecated. Marconi maintainers will work with developers and partners to ensure there is adequate time to migrate to new versions before deprecated ones are discontinued.

Since the base URI is only incremented to accommodate major API revisions, sub-versioning of the API is meaningless and is therefore not used. For example, the next version after "v1" would be "v2", not "v1.1" or some variant thereof. (HATEOS and media types are used in lieu of minor, URI-based versioning.)

Resource Media Types

The API supports `application/json` and `application/xml`, encoded as UTF-8. Gzip'd requests are optionally supported by the server.

Unrecognised protocol elements should be ignored. This includes XML elements and attributes, JSON object properties, link relation types, media types, etc.

TODO: Define and register message media type/ messages media type - http://www.iana.org/assignments/media-types/index.html

Authentication

All requests to the API may only be performed by an authenticated agent.

To authenticate, an agent issues an authentication request to a Keystone Identity Service endpoint.

In response to valid credentials, Keystone responds with an auth token and a service catalog that contains a list of all services and endpoints available for the given token. Multiple endpoints may be returned for Marconi according to physical locations and performance/availability characteristics of different deployments.

Marconi clients must specify a valid auth token in the `X-Auth-Token` header for each request to the Marconi API. The API validates auth tokens against Keystone before servicing each request.

Authorization

TBD: The API needs to verify read/write access to all endpoints according to the provided auth token.

Request Signing

Messages may optionally contain an HMAC.

TODO: How to register and query public keys? Via Keystone? Shouldn't be part of Marconi, in any case.

How to calculate signature?

SecP256K1 or something using RSA-2048? (encrypt a SHA-512 value). Would be nice to allow different schemes... How is Ceilometer doing it?

See also: http://stackoverflow.com/questions/9897023/asymmetric-hmac#

Tenant ID

Auth tokens are only valid for a particular tenant ID, which should be reflected in the service endpoints retrieved from Keystone. Agents use the given endpoint as a home document href from which to discover links to all other API requests.

An example endpoint:


https://marconi.example.com/v1/480924


Errors

If any request results in an error, the server will return an appropriate 4xx or 5xx HTTP status code, as well as the following information in the body:

  • Title
  • Description
  • Internal code
  • Link to more information

Example:


HTTP/1.1 400 Bad Request
Content-Type: application/json

{
  "title": "Unsupported limit",
  "description": "The given limit cannot be negative, and cannot be greater than 50.",
  "code": 1092,
  "link": {
    "rel": "api-doc",
    "href": "http://example.com/marconi/docs/messages#limit",
    "text": "API documentation for the limit parameter"
  }
}


Common Headers

Each request to the API must include certain standard and extended HTTP headers. These headers provide host, agent, authentication and other pertinent information to the server.

Header Description
Host The host name of the API, as referenced by the Keystone service catalog
User-Agent The name and version of the Marconi client, as well as a UUID for that client. Marconi uses the UUID to distinguish publishers from subscribers, i.e., to avoid echoing an agent's own messages back to it.
Date The current date and time, using the standard RFC 1123 HTTP date format
Accept Media type desired; may be either `application/json` or `application/xml`
Accept-Encoding Specifies that the agent accepts gzip-encoded response bodies
Content-Length For POST or PUT requests, the length in bytes of the JSON or XML document being submitted
X-Auth-Token Keystone auth token

Sample API Request

GET /v1/480924/messages?tags=channel-foo,topic-bar&marker=50b68a50d6f5b8c8a7c62a22&limit=10 HTTP/1.1
Host: marconi.example.com
User-Agent: python/2.7 killer-rabbit/1.2 uuid/30387f00-39a0-11e2-be4d-a8d15f34bae2
Date: Wed, 28 Nov 2012 21:14:19 GMT
Accept: application/json
Accept-Encoding: gzip
X-Auth-Token: 7d2f63fd-4dcc-4752-8e9b-1d08f989cc00


Get Home Document

Request Template


GET /{version}/{tenant}


Sample Response


@todo


Discussion

The entire Marconi API is discoverable from a single starting point; agents do not need to know any more than this one URI in order to explore the entire API.

See also: http://tools.ietf.org/html/draft-nottingham-json-home-02

Get a Specific Message

Request Template


GET /{version}/{tenant}/messages/{messageId}


Sample Request


GET /v1/480924/messages/50b68a50d6f5b8c8a7c62b01 HTTP/1.1


Sample Response


HTTP/1.1 200 OK

{
  "id": 50b68a50d6f5b8c8a7c62b01,
  "userAgent": "python/2.7 killer-rabbit/1.2 uuid/79ed56f8-2519-11e2-b835-acf6018e45a1",
  "age": 790,
  "tags": [
    foo,
    snap,
    bar,
    bang
  ],
  "signature": {
    "scheme": "SecP256K1",
    "salt": "b8c850d6f5b8a7c6",
    "value": "T2gJfGQNgUe1XMDHyucwH27zn628it0fWrCFgE2mPWR+oMTOiW7jb1OPNuZtLus5O1IzTzy+5ALyLCyUq7JoLQ=="
  },
  "body": {
    "event": "ActivateAccount",
    "mode": "active"
  }
}


Discussion

@todo Define message fields.

Get Messages

Request Template


GET /{version}/{tenant}/messages{?tags,marker,limit,sort,meta,echo}


Sample Request


GET /v1/480924/messages?tags=foo,bar,bang&marker=50b68a50d6f5b8c8a7c62a22&limit=10&sort=desc HTTP/1.1


Sample Response

HTTP 200 if the query matched 1 or more messages, HTTP 204 otherwise (with no body).


HTTP/1.1 200 OK

{
  "totalMessages": 1,
  "next": {
    "tags": "foo,bar,bang",
    "marker": "50b68a50d6f5b8c8a7c62b01",
    "limit": 10,
    "sort": "desc"
  },
  "messages": [
    {
      "id": 50b68a50d6f5b8c8a7c62b01,
      "userAgent": "python/2.7 killer-rabbit/1.2 uuid/79ed56f8-2519-11e2-b835-acf6018e45a1",
      "age": 790,
      "tags": [
        foo,
        snap,
        bar,
        bang
      ],
      "signature": {
        "scheme": "SecP256K1",
        "salt": "b8c850d6f5b8a7c6",
        "value": "T2gJfGQNgUe1XMDHyucwH27zn628it0fWrCFgE2mPWR+oMTOiW7jb1OPNuZtLus5O1IzTzy+5ALyLCyUq7JoLQ=="
      },
      "body": {
        "event": "ActivateAccount",
        "mode": "active"
      }
    }
  ]
}


Example meta-messages response (if a client were to add meta=true to the sample query):


HTTP/1.1 200 OK

{
  "totalMessages": 2,
  "next": {
    "tags": "foo,bar",
    "marker": "10b00a50d6f5b8c8a7c62ccc",
    "limit": 10,
    "sort": "desc"
  },
  "messages": [
    {
      "id": 10b00a50d6f5b8c8a7c62ccc,
      "userAgent": "marconi/1 uuid/f2e4b36a-3f05-11e2-b71d-7823d2b0f3ce",
      "age": 790,
      "tags": [
        foo,
        snap,
        bar,
        bang
      ],
      "body": {
        "event": "MessageCreated",
        "timestamp": "2012-12-04 16:53:20Z",
        "details": {
          "messageId": "50b68a50d6f5b8c8a7c62b01",
          "userAgent": "python/2.7 killer-rabbit/1.2 uuid/79ed56f8-2519-11e2-b835-acf6018e45a1",
        }
      }
    },
    {
      "id": 10b00a50d6f5b8c8a7c62ccb,
      "userAgent": "marconi/1 uuid/f2e4b36a-3f05-11e2-b71d-7823d2b0f3ce",
      "age": 792,
      "tags": [
        foo,
        snap,
        bar,
        bang
      ],
      "body": {
        "event": "TransactionExpired",
        "timestamp": "2012-12-04 16:53:18Z",
        "details": {
          "messageId": "50b68a50d6f5b8c8a7c62a87",
          "userAgent": "python/2.7 killer-rabbit/1.2 uuid/79ed56f8-2519-11e2-b835-acf6018e45a1",
        }
      }
    }
  ]
}


Discussion

tags is a list of up to n message tags, where n >= 2. The maximum number of tags supported is configurable. The API will return only those messages containing ALL of the specified tags. If no tags are specified, all messages are returned.

limit specifies up to x messages to return, where x >= 10. Note that x is configurable. If not specified, limit defaults to x. When more messages are available than can be returned in a single request, the client can pick up the next batchby simply using the URI template parameters returned from the previous call in the "next" field (TBD).

marker is the id from the last message the client saw. Note that message IDs should be treated as opaque strings by clients; no assumptions can be made on their format or length, except that string lengths are <= 50). The API will return messages that were enqueued after the specified message, minus t milliseconds, where t is an implementation-defined number of milliseconds within which the server cannot guarantee message ordering (admittedly, a leaky abstraction). Note that the message having the given marker will always be part of the result set unless that particular message has expired. The client must cache messages for t milleseconds in order to check for duplicates returned in subsequent requests. If marker is not specified, the API will return all messages.

sort specifies how to chronologically order results. Use "asc" or "desc" for ascending or descending, respectively. If sort is not given, the API will return results in ascending order (oldest message first).

meta is a boolean value (i.e., "true" or "false") that determines whether the API will return actual messages or meta-messages. Meta-messages are "messages about messages", and are automatically generated by the server. Clients may query for meta-messages in order to audit a business processes or to diagnose data flow issues. If meta is not specified, the API defaults to "false".

echo is a boolean (i.e., "true" or "false") that determines whether or not the API will return a client's own messages, as determined by the uuid portion of the User-Agent header. If not specificed, echo defaults to "false".

Post Messages

Request Template


POST /{version}/{tenant}/messages


Sample Request


POST /v1/480924/messages HTTP/1.1

[
  {
    "ttl": 10,
    "durability": 3,
    "tags": [420D29D6-3F24-11E2-BC14-7823D2B0F3CE, checkpoint]
    "signature": {
      "scheme": "SecP256K1",
      "salt": "b8c850d6f5b8a7c6",
      "value": "T2gJfGQNgUe1XMDHyucwH27zn628it0fWrCFgE2mPWR+oMTOiW7jb1OPNuZtLus5O1IzTzy+5ALyLCyUq7JoLQ=="
    },
    "body": {
      "event": "BackupStarted",
      "backupId": "c378813c-3f0b-11e2-ad92-7823d2b0f3ce"
    }
  },
  {
    "tags": [420D29D6-3F24-11E2-BC14-7823D2B0F3CE, progress]
    "body": {
      "event": "BackupProgress",
      "currentBytes": "0",
      "totalBytes": "99614720"
    }
  }
]


Sample Responses

When a single message is submitted:


HTTP/1.1 201 Created
Location: https://marconi.example.com/v1/480924/messages/50b68a50d6f5b8c8a7c62b01


When multiple messages are submitted:


HTTP/1.1 201 Created
Location: https://marconi.example.com/v1/480924/messages&marker=50b68a50d6f5b8c8a7c62b01


Discussion

One or more messages may be submitted in a single request, but must always be encapsulated in a collection container (an array in JSON, a parent tag in XML). In the case of a batch POST, querying the returned Location may return messages posted concurrently by other agents.

ttl is the number of seconds the server will keep a message before automatically deleting it. Should be long enough to give all observers ample time to retrieve the message. If not specified, defaults to the maximum default set for any of the message's associated tags, or the default set for the tenant (TBD), or the one configured in the deployment (?).

tags is a list of up to n tags to associate with a given message, where n >= 2. The maximum number of tags supported is configurable (the default is 5). The maximum length of each tag is likewise configurable (with a default of 150 characters).

signature - TBD, probably something like sign(hash(salt + payload))

body specifies a custom document which constitutes the body of the message being sent. The size of this body, in characters and including whitespace, is configurable (the default is 64 KiB).

durability requests a certain durability guarantee from the server. The purpose of this parameter is to allow clients to dynamically make tradeoffs between durability and cost/performance depending on the type of message being sent. Note that the maximum durability level supported by the server is configurable; not all deployments will support all levels. If a level is unsupported, the server will return 400 Bad Request. If a level is supported but the server is unable to complete the request, and appropriate 5xx error will be returned to the client.

The following levels are defined:

Level
0
1
2
3
4

Note that higher durability levels assume the guarantees (if any) of all lower levels.

Delete a Single Message

Request Template


DELETE /{version}/{tenant}/messages/{messageId}{?transactionId}


Sample Request


DELETE /v1/480924/messages/50b68a50d6f5b8c8a7c62b01 HTTP/1.1


Sample Response


HTTP/1.1 204 No Content


Discussion

messageId specifies the message to delete.

transactionId specifies that the message should only be deleted if it has the specified transaction ID. This is useful for ensuring only one agent processes any given message; whenever a worker client's transaction expires before it has a chance to delete a message it has processed, the worker must roll back any actions it took based on that message, since another worker will likely process the same message as part of a different transaction.

Delete Several Messages

Request Template


DELETE /{version}/{tenant}/messages{?tags,all}


Sample Request


DELETE /v1/480924/messages?tags=420D29D6-3F24-11E2-BC14-7823D2B0F3CE HTTP/1.1


Sample Response


HTTP/1.1 200 OK

{
  "totalMessages": 43
}


Discussion

tags specifies that only messages having the given tags will be deleted. To avoid accidentally deleting all messages for a given tenant, if tags is not specified, all must equal "true"; otherwise, no messages will be deleted and the server will return 400 Bad Request.

all is a boolean value (i.e., "true" or "false") that determines whether all messages will be deleted in the case that no tags are specified. If tags are specified, however, the server simply ignores all.

Unsupported

marker is not allowed due to the best-effort ordering and quering of messages by their IDs.

transactionId is not allowed when deleting multiple messages due to the server-side complexity of tracking which messages were part of a previous transaction.

Open a Transaction

Request Template


GET /{version}/{tenant}


Sample Response


@todo


Discussion

@todo

Renew a Transaction

Request Template


GET /{version}/{tenant}


Sample Response


@todo


Discussion

@todo

Count Messages

Request Template


HEAD /{version}/{tenant}/messages{?tags,marker,meta,echo}


Sample Request


HEAD /v1/480924/messages?tags=foo,bar,bang&marker=50b68a50d6f5b8c8a7c62a22 HTTP/1.1


Sample Response HTTP/1.1 200 OK


{
  "totalMessages": 46
}


Discussion

See Get Messages for definitions of the query parameters.

Check Health

Request Template


GET /{version}


HEAD also allowed?

Sample Request


GET /v1


Sample Response


HTTP/1.1 200 OK

{
  "code": "green",
  "link": {
    "rel": "status",
    "href": "http://marconi.example.com/status",
    "text": "Service status page"
  }
}


Discussion

Use this request to check on the Marconi service status as a whole. The following status values are defined:

Code
green
yellow
red

Set Tag Defaults

TBD - should marconi support tenant defaults in addition to, or instead of, tag-based defaults? Maybe start with tenant defaults, add tag defaults later?

Request Template


GET /{version}/{tenant}


Sample Response


@todo


Discussion

TTL, Durability, ACLs?

@todo

Get Tag Defaults

Request Template


GET /{version}/{tenant}


Sample Response


@todo


Discussion

@todo

TBD

Request Template


GET /{version}/{tenant}


Sample Response


@todo


Discussion

@todo