Deprecated

Deploying w/ a Full-Blown Service

A placement service consists of the following components:

A catalogue of mappings: {queue => [read locations], queue => [write locations], queue => status}

   * where status is one of ACTIVE or MIGRATING

A cache maintained in each Marconi instance of the mappings
A migration service

Placement API

The placement service must handle:

storage assignment
cataloging
migrations

An API must be exposed to facilitate these operations.

Storage Management

Add node (POST /v1/storage: {'location': 'mongo1.storage.com' , 'weight': 100})
Delete node (DELETE /v1/storage/mongo1.storage.com)
Change weight (PUT /v1/storage/mongo1.storage.com {'weight': 50})
Query storage (GET /v1/storage => [...], GET /v1/storage/mongo1.storage.com => {'weight': 100})

Catalogue Management

Add entry (POST)
Update an entry (PUT)
Remove entry (DELETE)

Migration

This means: "move object from location A to location B". In terms of the Placement service, it must:

Update the storage write location
Update the storage read location
Update the entry status ("a" => "m")

Catalogue

The catalog, as described above, maintains these mappings from queues to locations.

Marconi can continue to function in a degraded state if the placement service goes down. The following capabilities will cease to function for Marconi if the placement service goes down:

Migration
Queue creation

It may be possible to continue performing queue creation and deletion while the placement service is down by defaulting to the last storage location that Marconi held.

The data contained in the catalogue can be regenerated by querying each Marconi endpoint and synchronizing against each cache.

The Marconi catalogue contains a series of structures that look as follows:

{
    "{project}.{queue}" : {
        "r": [("mongo://192.168.1.105:7777", "db1"), ("mongo://192.168.1.106:7777", "db2")],
        "w": [("mongo://192.168.1.105:7777", "db1")],
        "s": "a"
    }
}

An entry in the queue is found by concatenating a project with a queue name. "r" is the collection of locations where data for a particular queue can be gathered from. "w" is the locations where new data written to this queue are stored. "s" is the state of the queue: active "a" or migrating "m". A storage location is given by a URL and a database name.

The catalogue is filled with many such documents. The local cache for a given Marconi node is populated with the entries from this cache, to avoid lookups on each request.

Adding Items to the Catalogue

Whenever a Marconi queue is created, a hook is called that transfers control the the placement service. It then becomes the placement service's responsibility to:

Take the given project ID and namespace (queue name)
Assign a read storage location to it
Assign a write storage location to it
Store these locations in the catalogue
Return these locations to the Marconi worker

Removing Items from the Catalogue

When a Marconi queue has been deleted, a hook must be called to notify the Placement service. The placement service must then remove the entry from its catalogue and notify all subscribed Marconi listeners.

Placement Policy Engine

The placement service will use a weighted distribution to determine storage assignment for newly created queues. However, for certain tenants, we might want to assign them to special or dedicated storage.

A weighted distribution might look like:

{
    "mongo1.storage.com": 30,
    "mongo2.storage.com": 20,
    "mongo3.storage.com": 30,
    "mongo4.storage.com": 75,
    "mongo5.storage.com": 5,
}

This means that the probability of a queue being stored on mongo1 is (30)/(30 + 20 + 30 + 75 + 5) => 18.75%.

The policy engine is responsible for maintaining this list of special tenants and their preferred mappings. Whenever a request to create a queue is received, Placement iterates through this list, and if the tenant is found, the associated dedicated storage is returned. If no such tenant is found, then the assignment of storage to the queue is determined by Placement.

Marconi Cache: Using PubSub

If the Marconi catalog and the local caches are implemented using a pubsub-based transport, the invalidation procedure can be simplified. The catalog publishes on the cache channel and all listening Marconi workers update their local cache accordingly. Simple messages in the form of:

<action> <queue> [<destination>]
Where action is one of: delete, create, migrate
Where destination is valid only for migrate, and if left unspecified, leaves the decision of destination up the placement service.

Possibilities here include: Redis, ZeroMQ

Placement Service API

To be filled soon: a Python transport/storage API for managing the placement service catalogue and storage nodes.

Advantages

Can be generalized: Sharding as a Service

Disadvantages

Complex - many pieces to implementation
Lots of custom code - RPC, storage drivers, transport, etc.
Adds a dependency to Marconi - not deployment friendly
NIH - rewrites a lot of logic freely available else where