Jump to: navigation, search

Difference between revisions of "Zaqar/specs/api/v1.1"

(Messages can no longer be retrieved by ID)
(Endpoints Synopsis)
Line 197: Line 197:
 
# Queues
 
# Queues
 
GET /v1.1/queues{?marker,limit,detailed}
 
GET /v1.1/queues{?marker,limit,detailed}
GET /v1.1/queues/{queue_name}  # existence check
 
HEAD /v1.1/queues/{queue_name}  # existence check
 
 
PUT /v1.1/queues/{name}
 
PUT /v1.1/queues/{name}
 
DELETE /v1.1/queues/{name}
 
DELETE /v1.1/queues/{name}

Revision as of 18:10, 10 June 2014

Marconi API: v1.1 Blueprint

Notes

The Marconi v1.1 API is a WIP and currently UNDER DISCUSSION

Feedback on the v1.1 spec is welcome (find us in #openstack-marconi, suggestions tracked on this etherpad). We have also started collecting feedback with an eye toward extensions and v2 of the API here: Marconi/specs/api/next

TODO

  • Document version discovery

Overview

Marconi provides an HTTP-based API in the spirit of the REST architecture style.

This guide assumes the reader is familiar with REST, HTTP/1.1 and JSON. URI templates use the same syntax as RFC 6570.

Note: Error responses are enumerated on a separate page: Marconi/specs/api/v1.1/errors

Changes from v1.0

BREAKING

  • Messages can no longer be retrieved by ID, although individual messages can still be deleted by ID.
  • The X-Project-Id header is now required for every request. This was done in anticipation of identity allowing a single user to manage multiple projects, in which case we will need to know which project this user is requesting (vs. assuming 1:1 mapping between auth token and project). It also affords rate limiting at the LB.
  • The Client-ID header is now required for every request.
  • The Content-Location header is no longer set in responses
  • The partial field is no longer included in responses to a message posting request. When using the official SQL and MongoDB drivers, Marconi will either succeed in enqueuing all messages in the submitted batch, or fail to enqueue any of them. Any backend driver that supports batch message posting will need to guarantee atomic inserts of the entire batch (v2.0 of the API will define some operations as optional, such as batch message posting).
  • When a collection resource is requested (i.e., queues, messages) and the collection is empty, an empty JSON array is now returned in the response body, rather than HTTP 204 No Content.
  • All top-level JSON arrays in response bodies are now encapsulated within an object.
  • Health is now an admin-only endpoint and returns operational stats for a given node. A new "ping" endpoint was added for load balancers to use.


NON-BREAKING

  • MessagePack is now supported in addition to JSON
  • Extraneous path elements from relative URIs were removed
  • Control plane for storage sharding was added
  • Version discovery is now available by accessing the root URI path
  • The Location header, when present, now specifies a full (i.e., non-relative) URI
  • A claim field has been added to messages. It is omitted if the message is not claimed.
  • An id field has been added to messages
  • Message href's now always include claim_id in the query string if the message is claimed.
  • When posting messages or creating a claim, the ttl field may be omitted. If it is not given by the client, Marconi will default to the max TTL setting configured by the cloud operator.
  • New sharding API and capability
  • Added new "pop" semantics for claiming and deleting messages in a single request. Note that this should only be used when an application is OK with the risk of missing a message when a worker crashes.
  • The server will automatically create a queue when posting a message, if the queue does not already exist.
  • Queries for messages and stats in non-existing queues now return empty lists, etc. rather than 404s.

Common API Elements

Clients

  • Clients should follow HTTP redirects
  • Clients should advertise gzip support
  • Clients should identify themselves in the UserAgent request header; e.g., "User-Agent: python/2.7 cloudthing/1.2"
  • Clients should not hard-code URI paths or templates since they may change over time. Instead, clients should cache links and link templates provided by the API.


API Versioning

The Marconi API uses a URI versioning scheme. The first element of the path contains the target version identifier, e.g.:

  https://marconi.example.com/v1.1

The URI version will only be incremented to accommodate major new features or API redesigns that can not be made backwards-compatible. When new API versions are released, older versions are deprecated. Marconi maintainers will work with developers and partners to ensure there is adequate time to migrate to new versions before deprecated ones are discontinued.

Since the version is only incremented to accommodate major API revisions, sub-versioning of the API will be rare.

Resource Media Types

The API supports `application/json`, encoded as UTF-8. MessagePack is also supported ('application/x-msgpack'), and is the recommended default for client libraries.

Unrecognized protocol elements received from the server should simply be ignored. This includes JSON object properties, link relation types, media types, etc.

Authentication

May be optionally implemented by middleware or reverse proxy. If implemented, the following applies:

All requests to the API may only be performed by an authenticated agent.

To authenticate, an agent issues an authentication request to a Keystone Identity Service endpoint.

In response to valid credentials, Keystone responds with an auth token and a service catalog that contains a list of all services and endpoints available for the given token. Multiple endpoints may be returned for Marconi according to physical locations and performance/availability characteristics of different deployments.

Normally, Keystone middleware provides the X-Project-Id header based on the auth token submitted by the Marconi client. For this to work, clients MUST specify a valid auth token in the `X-Auth-Token` header for each request to the Marconi API. The API validates auth tokens against Keystone before servicing each request

If auth is not enabled, clients must provide the X-Project-Id header themselves.

Authorization

May be optionally implemented by middleware or reverse proxy. If implemented, the following applies:

The API needs to verify read/write access to all endpoints according to the provided auth token (RBAC). The ACL should be cached with the token.


Endpoints

Clients may choose to interact with any of the service endpoints retrieved from Keystone. Agents use the given endpoint as a home document href from which to discover links to all other API requests. Endpoints may be organized by region

An example endpoint:

https://marconi.example.com/v1.1

Errors

If any request results in an error, the server will return an appropriate 4xx or 5xx HTTP status code, and optionally the following information in the body:

  • Title
  • Description
  • Internal code
  • Link to more information


Example:

HTTP/1.1 400 Bad Request
Content-Type: application/json

{
  "title": "Unsupported limit",
  "description": "The given limit cannot be negative, and cannot be greater than 50.",
  "code": 1092,
  "link": {
    "rel": "help",
    "href": "http://docs.example.com/messages#limit",
    "text": "API documentation for the limit parameter"
  }
}

Common Headers

Each request to the API must include certain standard and extended HTTP headers. These headers provide host, agent, authentication and other pertinent information to the server.

Header Description
Host The host name of the API, as referenced by the Keystone service catalog
Date The current date and time, using the standard RFC 1123 HTTP date format
Accept Media type desired; only application/json is supported at this time.
Accept-Encoding Specifies that the agent accepts gzip-encoded response bodies.
Content-Length For POST or PUT requests, the length in bytes of the message document being submitted
X-Auth-Token Keystone auth token, required in every request when auth is enabled
X-Project-Id An ID for a project to which the value of X-Auth-Token grants access. Queues will be created under this project. This header is required in every request for multi-tenant deployments.
Client-ID A UUID, used to distinguish each client using the service. The UUID must be submitted in its canonical form (e.g., 3381af92-2b9e-11e3-b191-71861300734c). In Marconi, this is used to avoid echoing a sender's messages back to the same instance, and may be logged by the server to provide diagnostics and statistics. Should be generated once and persisted between restarts of the client to avoid the race condition where a client sends a message, restarts with a different ID, and then receives its own message, echoed back.

HTTP Response Codes

See Response Codes

Sample API Request

GET /v1.1/queues/fizbat/messages?marker=1355-237242-783&limit=10 HTTP/1.1
Host: marconi.example.com
User-Agent: python/2.7 killer-rabbit/1.2
Date: Wed, 28 Nov 2012 21:14:19 GMT
Accept: application/json
Accept-Encoding: gzip
X-Auth-Token: 7d2f63fd-4dcc-4752-8e9b-1d08f989cc00
X-Project-Id: 518b51ea133c4facadae42c328d6b77b
Client-ID: 30387f00-39a0-11e2-be4d-a8d15f34bae2

Endpoints Synopsis

# Base endpoints
GET /v1.1
GET /v1.1/health
GET /v1.1/ping

# Queues
GET /v1.1/queues{?marker,limit,detailed}
PUT /v1.1/queues/{name}
DELETE /v1.1/queues/{name}
PUT /v1.1/queues/{queue_name}/metadata
GET /v1.1/queues/{queue_name}/metadata

# Queue stats
GET /v1.1/queues/{queue_name}/stats

# Messages
GET /v1.1/queues/{queue_name}/messages{?marker,limit,echo,include_claimed}
POST /v1.1/queues/{queue_name}/messages
DELETE /v1.1/queues/{queue_name}/messages/{message_id}{?claim_id}
DELETE /v1.1/queues/{queue_name}/messages{?ids}
DELETE /v1.1/queues/{queue_name}/messages{?pop}

# Claims
POST /v1.1/queues/{queue_name}/claims{?limit}
GET /v1.1/queues/{queue_name}/claims/{claim_id}
PATCH /v1.1/queues/{queue_name}/claims/{claim_id}
DELETE /v1.1/queues/{queue_name}/claims/{claim_id}

# Sharding
GET /v1.1/shards?detailed=False&marker=None&limit=10
GET /v1.1/shards/{shard}?detailed=False
PUT /v1.1/shards/{shard}
DELETE /v1.1/shards/{shard}
PATCH /v1.1/shards/{shard}

Base Endpoints

Get Home Document

Template

GET /v1.1


Request

GET /v1.1 HTTP/1.1
Host: marconi.example.com

...


Response

HTTP/1.0 200 OK
Cache-Control: max-age=86400
Content-Length: 4345
Content-Type: application/json-home
Date: Tue, 06 Aug 2013 16:31:48 GMT
Server: WSGIServer/0.1 Python/2.7.3

{
    "resources": {
        "rel/queue-stats": {
            "href-template": "/v1.1/queues/{queue_name}/stats",
            "href-vars": {
                "queue_name": "param/queue_name"
            },
            "hints": {
                "allow": [
                    "GET"
                ],
                "formats": {
                    "application/json": {}
                }
            }
        },
        "rel/post-messages": {
            "href-template": "/v1.1/queues/{queue_name}/messages",
            "href-vars": {
                "queue_name": "param/queue_name"
            },
            "hints": {
                "accept-post": [
                    "application/json"
                ],
                "allow": [
                    "POST"
                ],
                "formats": {
                    "application/json": {}
                }
            }
        },
        "rel/queue": {
            "href-template": "/v1.1/queues/{queue_name}",
            "href-vars": {
                "queue_name": "param/queue_name"
            },
            "hints": {
                "allow": [
                    "GET",
                    "HEAD",
                    "PUT",
                    "DELETE"
                ],
                "formats": {
                    "application/json": {}
                }
            }
        },
        "rel/queues": {
            "href-template": "/v1.1/queues{?marker,limit,detailed}",
            "href-vars": {
                "marker": "param/marker",
                "detailed": "param/detailed",
                "limit": "param/queue_limit"
            },
            "hints": {
                "allow": [
                    "GET"
                ],
                "formats": {
                    "application/json": {}
                }
            }
        },
        "rel/messages": {
            "href-template": "/v1.1/queues/{queue_name}/messages{?marker,limit,echo,include_claimed}",
            "href-vars": {
                "marker": "param/marker",
                "include_claimed": "param/include_claimed",
                "queue_name": "param/queue_name",
                "limit": "param/messages_limit",
                "echo": "param/echo"
            },
            "hints": {
                "allow": [
                    "GET"
                ],
                "formats": {
                    "application/json": {}
                }
            }
        },
        "rel/messages-delete": {
            "href-template": "/v1.1/queues/{queue_name}/messages{?ids,pop}",
            "href-vars": {
                "ids": "param/ids",
                "pop": "param/pop",
            },
            "hints": {
                "allow": [
                    "DELETE"
                ],
                "formats": {
                    "application/json": {}
                }
            }
        },
        "rel/claim": {
            "href-template": "/v1.1/queues/{queue_name}/claims{?limit}",
            "href-vars": {
                "queue_name": "param/queue_name",
                "limit": "param/claim_limit"
            },
            "hints": {
                "accept-post": [
                    "application/json"
                ],
                "allow": [
                    "POST"
                ],
                "formats": {
                    "application/json": {}
                }
            }
        }
    }
}


Discussion

The entire Marconi API is discoverable from a single starting point; agents do not need to know any more than this one URI in order to explore the entire API.

This document is cache-able (and should be cached by proxy and client).

See also: http://tools.ietf.org/html/draft-nottingham-json-home-03

Check Node Health

Template

GET /v1.1/health

or

HEAD /v1.1/health


Request

GET /v1.1/health HTTP/1.1
Host: example.marconi.com

...


Response

TBD


Discussion

This is an operator endpoint and should not be implemented in client libraries intended for end users. It should also not be included in the json-home document unless the requesting user is an administrator.


Use this request to get detailed operational stats for a specific Marconi node.


Ping a Node

Template

GET /v1.1/ping

or

HEAD /v1.1/ping


Request

GET /v1.1/ping HTTP/1.1
Host: example.marconi.com

...


Response

HTTP/1.1 204 No Content

or

HTTP/1.1 503 Service Unavailable


Discussion

This is an operator endpoint and should not be implemented in client libraries intended for end users. It should also not be included in the json-home document unless the requesting user is an administrator.

Use this resource to check whether a given Marconi node is online. If the node is down, this endpoint will not be reachable and will act as a signal to the load balancer to take the node out of rotation. Alternatively, the endpoint may be reachable but the service is temporarily unavailable due to storage driver failure or some other error.

The health endpoint may be accessed only if the X-Forwarded-For header is present. In that case, it will not require auth, allowing load balancers to query a given node's health.

For requests NOT sent via the LB, Marconi will return 404 Not Found, since this endpoint is not meant to be accessed by end users and there is no point in exposing another potential DDoS vector to the world.

Queues

Create Queue

Template

PUT /v1.1/queues/{queue_name}

Request

PUT /v1.1/queues/fizbat HTTP/1.1
Host: marconi.example.com


Response

HTTP/1.1 201 Created
Location: https://marconi.example.com/v1.1/queues/fizbit


Discussion

Creates a queue.

The body of the PUT is empty.

queue_name is the name to give the queue. The name MUST NOT exceed 64 bytes in length, and is limited to US-ASCII letters, digits, underscores and hyphens.

List Queues

Template

GET /v1.1/queues{?marker,limit,detailed}


Request

GET /v1.1/queues?marker=baz&detailed=true HTTP/1.1
Host: marconi.example.com

...


Response

HTTP/1.1 200 OK

{
  "links": [
    {
      "rel": "next",
      "href": "queues?marker=kooleo&detailed=true"
    }
  ],
  "queues": [
    { "name": "boomerang", "href": "queues/boomerang" },
    { "name": "fizbit", "href": "queues/fizbit" },

    ...

    { "name": "kooleo",  "href": "queues/kooleo" }
  ]
}

or, if no queues exist:

HTTP/1.1 200 OK

{
  "links": [
    {
      "rel": "next",
      "href": "queues?marker=baz&detailed=true"
    }
  ],
  "queues": []
}

Checking Queue Existence

Template

GET /v1.1/queues/{queue_name}


Request

GET /v1.1/queues/{queue_name}
Host: marconi.example.com


Response

HTTP/1.1 204 OK


Discussion

Returns 204 if the queue exists, else 404.

HEAD also works.

Delete Queue

Template

DELETE /v1.1/queues/{queue_name}


Request

DELETE /v1.1/queues/fizbat HTTP/1.1
Host: marconi.example.com


Response

HTTP/1.1 204 No Content


Discussion

Use this operation to immediately delete a queue along with all its messages (if any).

Queue Metadata

Set Queue Metadata

Template

PUT /v1.1/queues/{queue_name}/metadata

{
   ...
}


Request

PUT /v1.1/queues/fizbat/metadata HTTP/1.1
Host: marconi.example.com

...

{
   "key": {
       "key2": "value",
       "key3": [1, 2, 3, 4, 5]
    }
}


Response

HTTP/1.1 204 No Content


Discussion

This operation allows for setting structured metadata. If PUT is used, this document will replace the existing metadata document in it's entirety, so clients need to be careful they do not accidentally delete any document fields.

The body of the request is an arbitrary document which allows storing contextual information about the way a particular application should interact with the queue. The size of this body, in characters and including whitespace must be <= 64 KiB (configurable). The document MUST be valid JSON (Marconi will validate the document).

Get Queue Metadata

Template

GET /v1.1/queues/{queue_name}/metadata


Request

GET /v1.1/queues/fizbit/metadata HTTP/1.1
Host: marconi.example.com

...


Response HTTP/1.1 200 OK

{
   "key": {
       "key2": "value",
       "key3": [1, 2, 3, 4, 5]
    }
}


Discussion

Returns queue metadata.

Queue Stats

Template

GET /v1.1/queues/{queue_name}/stats


Request

GET /v1.1/queues/fizbit/stats HTTP/1.1
Host: marconi.example.com

...


Response HTTP/1.1 200 OK

{
  "messages": {
    "free": 146929,
    "claimed": 2409,
    "total": 149338,
    "oldest": {
        "href": "queues/fizbit/messages/50b68a50d6f5b8c8a7c62b01",
        "age": 63,
        "created": "2013-08-12T20:44:55Z"
    },
    "newest": {
        "href": "queues/fizbit/messages/50b68a50d6f5b8c8a7c62b01",
        "age": 12,
        "created": "2013-08-12T20:45:46Z"
    }
  }
}


Discussion

Returns queue statistics, including how many messages are in the queue, broken out by status.

NOTE: If total is 0, then "oldest" and "newest" message stats are not included.

Messages

All message-related operations require Client-Id to be included in the headers. This is to ensure that messages are not echoed back to the client that posted them unless the client explicitly requests this.

List Messages

Template

GET /v1.1/queues/{queue_name}/messages{?marker,limit,echo,include_claimed}


Request

GET /v1.1/queues/fizbit/messages?marker=1355-237242-783&limit=10 HTTP/1.1
Host: marconi.example.com

...


Response

HTTP/1.1 200 OK

...

{
  "links": [
    {
      "rel": "next",
      "href": "messages?marker=6244-244224-783&limit=10"
    }
  ],
  "messages": [
    {
      "href": "messages/50b68a50d6f5b8c8a7c62b01",
      "id": "50b68a50d6f5b8c8a7c62b01",
      "ttl": 120,
      "age": 53,
      "body": {
        "event": "ActivateAccount",
        "mode": "active"
      }
    },
    {
      "href": "messages/50b68a50d6f5b8c8a7c62b02?claim_id=06ef2372-6746-11e3-b311-b3adcbe406e9",
      "id": "50b68a50d6f5b8c8a7c62b01",
      "ttl": 800,
      "age": 790,
      "claim": {
        "id": "06ef2372-6746-11e3-b311-b3adcbe406e9",
        "client_id": "cc0eef38-6742-11e3-92c2-7e45c531c7ca"
      },
      "body": {
        "event": "CreateInvoice",
        "customer_id": "90fd8734-6746-11e3-be3c-7e45c531c7ca"
      }
    }
  ]
}

or, if no messages are available:

HTTP/1.1 200 OK

...

{
  "links": [
    {
      "rel": "next",
      "href": "messages?marker=6244-244224-783&limit=10"
    }
  ],
  "messages": []
}


Discussion

Message IDs and markers are opaque strings; clients should make no assumptions about their format or length. Furthermore, clients should assume there is no relationship between markers and message IDs (i.e., one cannot be derived from the other). This allows for a wide variety of storage driver implementations.

Results are ordered by age, oldest message first.

limit specifies up to 20 messages (configurable) to return. If not specified, limit defaults to 10. When more messages are available than can be returned in a single request, the client can pick up the next batch by simply using the URI template parameters returned from the previous call in the "next" field (TBD).

marker is an opaque string that the client can use to request the next batch of messages. It communicates to the server which messages the client has already received.

Note: If marker is not specified, the API will return all messages at the head of the queue (up to limit).

echo is a boolean (i.e., "true" or "false") that determines whether or not the API will return a client's own messages, as determined by the uuid portion of the Client-ID header. If not specified, echo defaults to "false".

include_claimed is a boolean (i.e., "true" or "false") that determines whether or not the API will return claimed messages as well as unclaimed ones. If not specified, defaults to "false" (only unclaimed messages are returned).

Post Message(s)

Template

POST /v1.1/queues/{queue_name}/messages


Request

POST /v1.1/queues/fizbit/messages HTTP/1.1
Host: marconi.example.com

...


{
  "messages" = [
    {
      "ttl": 300,
      "body": {
        "event": "BackupStarted",
        "backup_id": "c378813c-3f0b-11e2-ad92-7823d2b0f3ce"
      }
    },
    {
      "body": {
        "event": "BackupProgress",
        "current_bytes": "0",
        "total_bytes": "99614720"
      }
    }
  ]
}


Responses

When a single message is submitted:

HTTP/1.1 201 Created
Location: https://marconi.example.com/v1.1/queues/fizbit/messages?ids=50b68a50d6f5b8c8a7c62b01
Content-Type: application/json

{
    "links": [
        {
            "rel": "rel/message",
            "href": "messages/50b68a50d6f5b8c8a7c62b01"
        }
    ]
}

When multiple messages are submitted:

HTTP/1.1 201 Created
Location: https://marconi.example.com/v1.1/queues/fizbit/messages?ids=50b68a50d6f5b8c8a7c62b01,50b68a50d6f5b8c8a7c62b05
Content-Type: application/json

{
    "links": [
        {
            "rel": "rel/message",
            "href": "messages/50b68a50d6f5b8c8a7c62b01"
        },
        {
            "rel": "rel/message",
            "href": "messages/50b68a50d6f5b8c8a7c62b05"
        }
    ]
}

Discussion

Up to 20 messages (default, but configurable) may be submitted in a single request, but must always be encapsulated in a collection container (e.g., an array in JSON). The resulting value of the Location header or response body may be used to retrieve the created messages for further processing if needed.

The client only specifies the body and ttl for the message; the server will insert metadata such as id and age.

Note: If the specified queue does not already exist, the server will create it. Applications do not need to explicitly manage the lifetimes of their queues, but are encouraged to delete queues that are no longer in use.

The response body contains a list of resource paths corresponding to each message submitted in the request, in the same order. In the case of a server-side error part-way through the processing of the submitted messages, the entire batch will be abandoned (all-or-nothing), and the client will have to retry its request.

The size of the request document, in characters including whitespace, is 256 KiB by default (configurable). The document MUST be a well-formed JSON or MessagePack document (Marconi will validate).

body specifies an arbitrary document which constitutes the body of the message being sent.

ttl is how long the server should wait before expiring and removing the message from the queue. Value MUST be between 60 and 1209600 seconds (14 days, configurable), inclusive. If it is ommitted in the request, the TTL defaults to the max TTL that was configured by the operater (14 days by default). Note that the server may not actually delete the message until it's age has reached up to (ttl + 60) seconds, to allow for flexibility in storage implementations.

Delete Message

Template

DELETE /v1.1/queues/{queue_name}/messages/{message_id}{?claim_id}


Request

DELETE /v1.1/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b01 HTTP/1.1
Host: marconi.example.com

...


Response

HTTP/1.1 204 No Content


Discussion

message_id specifies the message to delete.

claim_id specifies that the message should only be deleted if it has the specified claim ID, and that claim has not expired. This is useful for ensuring only one agent processes any given message; whenever a worker client's claim expires before it has a chance to delete a message it has processed, the worker must roll back any actions it took based on that message, since another worker will now be able to claim and process the same message.

Note that if claim_id is not specified, but the message is claimed, the operation will fail. In other words, claimed messages can only ever be deleted by providing an appropriate claim_id.

Delete Multiple Messages

Template

DELETE /v1.1/queues/{queue_name}/messages{?ids,pop}


Request

DELETE /v1.1/queues/fizbit/messages?ids=50b68a50d6f5b8c8a7c62b01,50b68a50d6f5b8c8a7c62b02 HTTP/1.1
Host: marconi.example.com

...

Response

HTTP/1.1 200 OK
Content-Type: application/json

...

{
  "deleted": 2
}


or, to atomically pop several messages (when only 2 messages are available):

Request

DELETE /v1.1/queues/fizbit/messages?pop=5 HTTP/1.1
Host: marconi.example.com

...

Response

HTTP/1.1 200 OK
Content-Type: application/json

...

{
  "deleted": 2,
  "messages": [
    {
      "href": "/v1.1/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b01?claim_id=a28ee94e-6cb4-11e2-b4d5-7703267a7926",
      "id": "50b68a50d6f5b8c8a7c62b01",
      "ttl": 800,
      "age": 100,
      "body": {
        "object_id": "8a50d6",
        "target": "h.264"
      }
    },
    {
      "href": "/v1.1/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b02?claim_id=a28ee94e-6cb4-11e2-b4d5-7703267a7926",
      "id": "50b68a50d6f5b8c8a7c62b02",
      "ttl": 800,
      "age": 790,
      "body": {
        "object_id": "fb8c8a",
        "target": "h.264"
      }
    }
  ]
}

pop & ids parameters are mutually exclusive. Using them together in a request will result in HTTP 400.

Request

DELETE /v1.1/queues/fizbit/messages?pop=5&ids=50b68a50d6f5b8c8a7c62b02 HTTP/1.1
Host: marconi.example.com

...

Response

HTTP/1.1 400 Bad Request
Content-Type: application/json

...

{
  "message": pop and id params cannot be present together in the delete request
}

Discussion

Bulk delete for messages.

ids specifies the messages to delete, up to a maximum of 20 (the limit is configurable, shares the same setting as the maximum for listing messages)

pop specifies a certain number of messages to pop off the queue, which is equivalent to claiming and deleting those messages atomically (therefore guaranteeing once-and-only-once delivery). The maximum value for this parameter is the same as for the maximum number of messages that can be claimed in a single operation. Note that if the worker crashes after popping the messages, but before processing them, no other workers will be able to reclaim those messages. Therefore, the "pop" operation should only be used when the risk of losing an occasional message can be justified.

Note that the ids and pop parameters are mutually exclusive. In other words, if both params are present the server will respond with a "400 Bad Request" status.

If any of the message IDs are malformed or non-existent, they are ignored. The remaining valid message IDs are deleted.

WARNING: Claimed messages will not be skipped; they will be deleted along with unclaimed messages. If using the worker pool pattern, in which workers are claiming batches of messages, we encourage deleting messages one at a time. This ensures that at most one message could be processed, but not deleted, if a worker happens to crash at just the wrong moment:

    for each message in batch:
        process message
        delete message

As opposed to:

    for each message in batch:
        process message
    bulk-delete batch

Claims

Claim Messages

Template

POST /v1.1/queues/{queue_name}/claims{?limit}
Content-Type: application/json

...

{
    "ttl": {claim_ttl},
    "grace": {message_grace}
}


Request

POST /v1.1/queues/fizbit/claims?limit=5 HTTP/1.1
Host: marconi.example.com
Content-Type: application/json
Accept: application/json

...

{
    "ttl": 300,
    "grace": 300
}


Response

The client receives a claim URI and a list of claimed messages, if any:

HTTP/1.1 201 Created
Content-Type: application/json
Location: https://marconi.example.com/v1.1/queues/fizbit/claims/a28ee94e-6cb4-11e2-b4d5-7703267a7926

...

{
  "messages": [
    {
      "href": "/v1.1/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b01?claim_id=a28ee94e-6cb4-11e2-b4d5-7703267a7926",
      "id": "50b68a50d6f5b8c8a7c62b01",
      "ttl": 800,
      "age": 100,
      "claim" : {
        "id": "a28ee94e-6cb4-11e2-b4d5-7703267a7926",
        "client_id": "2d21ebb4-6747-11e3-89d1-43d6a24410d2"
      }
      "body": {
        "object_id": "8a50d6",
        "target": "h.264"
      }
    },
    {
      "href": "/v1.1/queues/fizbit/messages/50b68a50d6f5b8c8a7c62b02?claim_id=a28ee94e-6cb4-11e2-b4d5-7703267a7926",
      "id": "50b68a50d6f5b8c8a7c62b02",
      "ttl": 800,
      "age": 790,
      "claim": {
        "id": "a28ee94e-6cb4-11e2-b4d5-7703267a7926",
        "client_id": "2d21ebb4-6747-11e3-89d1-43d6a24410d2"
      }
      "body": {
        "object_id": "fb8c8a",
        "target": "h.264"
      }
    }
  ]
}

or, if no messages are available to claim:

HTTP/1.1 201 Created
Content-Type: application/json
Location: https://marconi.example.com/v1.1/queues/fizbit/claims/f9151272-673e-11e3-8e72-43d6a24410d2

...

{
  "messages": []
}


Discussion

Claims a set of messages, up to limit, from oldest to newest, skipping any that are already claimed. If no unclaimed messages are available, returns 204 No Content.

The client should delete the message when it has finished processing it, before the claim expires, to ensure the message is processed only once. As part of the delete operation, all worker clients should specify the claim ID (this is best done by simply using the provided href). That way, the server can return an error if the claim just expired (notifying the client of the race condition), giving the worker a chance to roll back its own processing of the given message, since another worker will likely claim the message and process it.

Just as with a message's age, the age given for the claim is relative to the server's clock, and is useful for determining how quickly messages are getting processed, and whether a given message's claim is about to expire.

When a claim expires, it is removed, allowing another client worker to claim the message in the case that the original worker fails to process it.

limit specifies up to 20 messages (configurable) to claim. If not specified, limit defaults to 10. Note that claim creation is best-effort, meaning the server may claim and return less than the requested number of messages.

ttl is how long the server should wait before releasing the claim. Value MUST be between 60 and 43200 seconds (12 hours, configurable). If ommitted, defaults to the max claim TTL set by the service provider.

grace is the message grace period in seconds. Value MUST be between 60 and 43200 seconds (12 hours, configurable). The server will extend the lifetime of claimed messages to be at least as long as the lifetime of the claim itself, plus a specified grace period to deal with crashed workers (up to 1209600 or 14 days including claim lifetime). If a message's lifetime already extends beyond the claim's TTL plus grace period, the message's lifetime will not be adjusted.

Query Claim

Template

GET /v1.1/queues/{queue_name}/claims/{claim_id}


Request

GET /v1.1/queues/foo-bar/claims/a28ee94e-6cb4-11e2-b4d5-7703267a7926
Host: marconi.example.com

...


Response

HTTP/1.1 200 OK

...

{
  "age": 19,
  "ttl": 30,
  "messages": [
    ...
  ]
}


Discussion

Renew Claim

Template

PATCH /v1.1/queues/{queue_name}/claims/{claim_id}
Content-Type: application/json


Request

PATCH /v1.1/queues/fizbit/claims/a28ee94e-6cb4-11e2-b4d5-7703267a7926 HTTP/1.1
Host: marconi.example.com
Content-Type: application/json

...

{ "ttl": 300 }


Response

HTTP/1.1 204 No Content


Discussion

Clients should periodically renew claims during long-running batches of work to avoid losing a claim in the middle of processing a message. This is done by issuing a PATCH to a specific claim resource and including a new TTL for the claim (which may be different from the original TTL). The server will reset the age of the claim and apply the new TTL.

Release Claim

Template

DELETE /v1.1/queues/{queue_name}/claims/{claim_id}


Request

DELETE /v1.1/queues/foo-bar/claims/a28ee94e-6cb4-11e2-b4d5-7703267a7926 HTTP/1.1
Host: marconi.example.com


Response

HTTP/1.1 204 No Content


Discussion

Use this operation to immediately release a claim, making any (remaining, non-deleted) messages associated with the claim available to other workers.

This operation is useful when a worker is performing a graceful shutdown, fails to process one or more messages, or is taking longer than expected to process messages, and wishes to make the remainder available to other workers.

Sharding

WARNING: in a production deployment, the sharding API should be restricted to administrators through role-based authentication (e.g., via policy middleware). You can also reduce your attack surface by disabling the sharding endpoints completely on public-facing web heads, by setting admin_mode = False in marconi.conf.

Marconi supports heterogeneous sharding through the use of:

  • stevedore for dynamic storage driver loading
  • node weights


As long as an entry point is defined in an installed module that matches the scheme of a shard connection URI, Marconi will be able to use that shard. For example, a shard entry might look like:


{
  "weight": 100,
  "uri": "mongodb://localhost:27017",
  "options": {
    "max_retry_sleep": 1
  }
}

Register a Shard

Template

PUT /v1.1/shards/{shard}

Request

PUT /v1.1/shards/wat HTTP/1.1
Host: marconi.example.com

{
  "weight": 100,
  "uri": "mongodb://localhost:27017",
  "options": {
    "max_retry_sleep": 1,
    "partitions": 8
  }
}


Response

HTTP/1.1 201 Created
Location: /v1.1/shards/wat


Discussion

Registers a shard.

shard is the name to give the shard entry. The name MUST NOT exceed 64 bytes in length, and is limited to US-ASCII letters, digits, underscores and hyphens.

weight is the likelihood that this shard will be selected for the next queue allocation. It must be an integer greater than -1.

uri is a connection string compatible with a storage client (e.g., pymongo) attempting to connect to that shard.

options An optional request component that gives storage-specific options used by storage driver implementations. Valid parameters come from the registered options for a given storage backend, for example: mongodb, sqlite

Read a Shard

Template

GET /v1.1/shards/{shard}?detailed=True

Request

GET /v1.1/shards/wat HTTP/1.1
Host: marconi.example.com


Response

HTTP/1.1 200 OK
Content-Location: /v1.1/shards/wat

{
  "uri": "mongodb://marconi1.example.com:27017",
  "weight": 100
}


Discussion

Returns information on a registered shard.

Delete a Shard

Template

DELETE /v1.1/shard/{shard}

Request

DELETE /v1.1/shards/wat HTTP/1.1
Host: marconi.example.com


Response

HTTP/1.1 204 No Content

Discussion

Removes a shard from the registry.

Update a Shard

Template

PATCH /v1.1/shards/{shard}

Request

PATCH /v1.1/shards/wat HTTP/1.1
Host: marconi.example.com

{
  "uri": "mongodb://marconi3.example.com:27018",
  "weight": 120
}


Response

HTTP/1.1 204 No Content

Discussion

Allows one to update any or all of: `weight`, `uri`, `options`. At least one of these fields must be specified, else, a HTTP 400 is returned.

List Shards

Template

GET /v1.1/shards?detailed=True&limit=10&marker=taco

Request

GET /v1.1/shards HTTP/1.1
Host: marconi.proxy.example.com


Response

HTTP/1.1 200 OK
Content-Location: /v1.1/shards

[
  {"href": "/v1.1/shards/wat", "weight": 100, "uri": "mongodb://marconi1.example.com:27017"},
  {"href": "/v1.1/shards/wot", "weight": 50, "uri": "redis://marconi2.example.com:6379"}
]

Discussion

Lists the registered shards.

detailed if True, returns the options field in the listing. marker used for pagination - what shard do we start listing from? limit how many entries to return per request?