Revision as of 16:03, 23 September 2013

Marconi Proxy: v1 Blueprint

Overview

Marconi's proxy service was written to enable scaling out a Marconi deployment at the partition level. It is controlled through an HTTP API.

Reference

For authoritative header/general details, please refer to the Marconi Queues API v1 Spec.

Endpoints Synopsis

# Proxy-specific endpoints
## Partition operations
GET /partitions
GET /partitions/{partition}
PUT /partitions/{partition}
DELETE /partitions/{partition}

## Catalogue operations
GET /catalogue
GET /catalogue/{queue}
GET /health

# Queues API intercepting endpoints
## Implicitly modify catalogue
GET /queues
GET /queues/{queue}
PUT /queues/{queue}
DELETE /queues/{queue}

# Operations forwarded to Queues API
## Informative
GET /v1

## Claims-forwarding
POST /queues/{queue}/claims
GET /queues/{queue}/claims/{claim_id}
PATCH /queues/{queue}/claims/{claim_id}
DELETE /queues/{queue}/claims/{claim_id}

## Messages forwarding
GET /queues/{queue}/messages
POST /queues/{queue}/messages
DELETE /queues/{queue}/messages
GET /queues/{queue}/messages/{message_id}
DELETE /queues/{queue}/messages/{message_id}

## Metadata handling - also implicitly modifies catalogue
GET /queues/{queue}/metadata
PUT /queues/{queue}/metadata

## Queue stats
GET /queues/{queue}/stats

Handling Partitions

A partition maps a name to a list of nodes associated with that partition and a weight. The weight is used to determine how likely a partition is to be selected as the storage location for a particular queue.

Register a Partition

Template

PUT /v1/partitions/{partition_name}

Request

PUT /v1/partitions/wat HTTP/1.1
Host: marconi.proxy.example.com

{
  "hosts": ["http://marconi1.example.com", "http://marconi2.example.com"],
  "weight": 100
}

Response

HTTP/1.1 201 Created
Location: /v1/partitions/wat

Discussion

Registers a partition.

partition_name is the name to give the queue. The name MUST NOT exceed 64 bytes in length, and is limited to US-ASCII letters, digits, underscores and hyphens.

Read a Partition

Template

GET /v1/partitions/{partition_name}

Request

GET /v1/partitions/wat HTTP/1.1
Host: marconi.proxy.example.com

Response

HTTP/1.1 200 OK
Location: /v1/partitions/wat

{
  "hosts": ["http://marconi1.example.com", "http://marconi2.example.com"],
  "weight": 100
}

Discussion

Returns information on a registered partition.

Delete a Partition

Template

DELETE /v1/partitions/{partition_name}

Request

DELETE /v1/partitions/wat HTTP/1.1
Host: marconi.proxy.example.com

Response

HTTP/1.1 204 No Content

Discussion

Removes a partition from the registry.

List Partitions

Template

GET /v1/partitions

Request

GET /v1/partitions HTTP/1.1
Host: marconi.proxy.example.com

Response

HTTP/1.1 200 OK
Location: /v1/partitions

{
    "wat": {
        "hosts": [
            "http://marconi1.example.com", 
            "http://marconi2.example.com"
        ], 
        "weight": 256
    }, 
    "wat2": {
        "hosts": [
            "http://marconi3.example.com", 
            "http://marconi4.example.com"
            "http://marconi5.example.com", 
            "http://marconi6.example.com"
        ], 
        "weight": 500
    }, 
    "wat3": {
        "hosts": [
        ], 
            "http://marconi7.example.com"
        "weight": 25
    }
}

Discussion

Removes a partition from the registry.

Handling the Catalogue

The catalogue associates a queue + project_id to a partition name. It also tracks metadata for that particular queue.

The catalogue is immutable, as far as the direct API allows. Modifications to the catalogue occur as a side effect to forwarding operations to a Marconi queues partition.

Read a Catalogue Entry

Template

GET /v1/catalogue/{queue_name}

Request

GET /v1/catalogue/taco HTTP/1.1
Host: marconi.proxy.example.com

Response

HTTP/1.1 200 OK
Location: /v1/catalogue/taco

{
  "name": "taco",
  "metadata": {},
  "partition": "wat",
  "host": "http://marconi1.example.com"
}

Discussion

Returns information on a captured catalogue entry.

NOTE: should a host be returned in the representation?

Listing the Catalogue

Template

GET /v1/catalogue

Request

GET /v1/catalogue HTTP/1.1
Host: marconi.proxy.example.com

Response

HTTP/1.1 200 OK
Location: /v1/catalogue

{
  "taco": {
    "name": "taco",
    "metadata": {},
    "partition": "wat",
    "host": "http://marconi1.example.com"
  },
  "lettuce": {...},
  "rice": {...}
}

Discussion

Lists the entries stored in the catalogue. Paging is not supported at this time.