Trove-Replication-And-Clustering-API
Contents
Instance Object Changes
Part of this proposal includes adding new properties to the json structure of an instance. The first would be to add a "nodes" property which would be list of refs back to the nova instance(s) that make up the trove instance.
The second would be a metadata storage that is of an open format to be used for storing critical data that is pertinent to the trove instance.
INSTANCES
Nodes
Because trove instances are a service type with an underlying nova resource(s) we can better display the nova resource(s) that make up the instance of a service type. When looking at a singular trove instance of something like redis or mysql the knowledge of the instance resource isn't all that important. Currently trove knows about the instance from a nova perspective but masks most of this information. This would look something like this:
GET /instances/{id}
{ "instance": { "created": "2013-05-08T22:43:34", "hostname": "mysweetdb.example.com", "id": "b142d7fe-2174-4bb8-ba02-c2ffd49d2e33", "name": "json_instance", "status": "ACTIVE", "updated": "2013-05-08T22:43:34", "service_type": "mysql-5.1", "links": [ { "href": "https://service/v1.0/1234/instances/b142d7fe-2174-4bb8-ba02-c2ffd49d2e33", "rel": "self" }, { "href": "https://service/instances/b142d7fe-2174-4bb8-ba02-c2ffd49d2e33", "rel": "bookmark" } ], "nodes": [ { "id": "dcc5c518-73c7-4471-83e1-15fae67a98eb", "hostname": "master-n01.example.com", "name": "master-n01", "flavor": { "id": "1", "links": [...] }, "links": [ { "href": "https://service/v1.0/1234/instances/{instance_id}/nodes/dcc5c518-73c7-4471-83e1-15fae67a98eb", "rel": "self" }, { "href": "https://service/instances/{instance_id}/nodes/dcc5c518-73c7-4471-83e1-15fae67a98eb", "rel": "bookmark" } ], "volume": { "size": 2, "used": 0.16368598397821188 }, }, ], } }
Metadata
The metadata looks much like our previous proposal. It contains critical information to the configuration or topology of the cluster or replication agreement. This should look something like this:
This would be an example of a 3 node mult-master mysql replication contract GET /instances/{id}
{ "instance": { "created": "2013-05-08T22:43:34", "id": "cfeccbf4-ac5b-494a-99b7-61593a6a71b0", "name": "multi-master-hotness", "status": "ACTIVE", "updated": "2013-05-08T22:43:34", "service_type": "mysql-5.5", "links": [ { "href": "https://service/v1.0/1234/instances/cfeccbf4-ac5b-494a-99b7-61593a6a71b0", "rel": "self" }, { "href": "https://service/instances/cfeccbf4-ac5b-494a-99b7-61593a6a71b0", "rel": "bookmark" } ], "nodes": [ { "id": "dcc5c518-73c7-4471-83e1-15fae67a98eb", "hostname": "master-n01.example.com", "name": "master-n01", "flavor": { "id": "1", "links": [...] }, "links": [ { "href": "https://service/v1.0/1234/instances/{instance_id}/nodes/dcc5c518-73c7-4471-83e1-15fae67a98eb", "rel": "self" }, { "href": "https://service/instances/{instance_id}/nodes/dcc5c518-73c7-4471-83e1-15fae67a98eb", "rel": "bookmark" } ], "volume": { "size": 2, "used": 0.16368598397821188 }, }, { "id": "4530736b-c1ff-4c07-9a9f-fa9bca418157", "hostname": "master-n02.example.com", "name": "master-n02", "flavor": { "id": "1", "links": [...] }, "links": [ { "href": "https://service/v1.0/1234/instances/{instance_id}/nodes/4530736b-c1ff-4c07-9a9f-fa9bca418157", "rel": "self" }, { "href": "https://service/instances/{instance_id}/nodes/4530736b-c1ff-4c07-9a9f-fa9bca418157", "rel": "bookmark" } ], "volume": { "size": 2, "used": 0.16368598397821188 }, }, { "id": "736b9820-7e91-4775-84b0-78e71d60ce4c", "hostname": "master-n03.example.com", "name": "master-n03", "flavor": { "id": "1", "links": [...] }, "links": [ { "href": "https://service/v1.0/1234/instances/{instance_id}/nodes/736b9820-7e91-4775-84b0-78e71d60ce4c", "rel": "self" }, { "href": "https://service/instances/{instance_id}/nodes/736b9820-7e91-4775-84b0-78e71d60ce4c", "rel": "bookmark" } ], "volume": { "size": 2, "used": 0.16368598397821188 }, }, ], "metadata": { "replication_contract": { // We should use something more clear than just nova instance // ids here but lets just use them for now for example. Maybe // links? names? not sure what works best yet. "dcc5c518-73c7-4471-83e1-15fae67a98eb": { "replicates_to": [ "4530736b-c1ff-4c07-9a9f-fa9bca418157", ], "replicates_from": [ "736b9820-7e91-4775-84b0-78e71d60ce4c", ], "writeable": true, }, "4530736b-c1ff-4c07-9a9f-fa9bca418157": { "replicates_to": [ "736b9820-7e91-4775-84b0-78e71d60ce4c", ], "replicates_from": [ "dcc5c518-73c7-4471-83e1-15fae67a98eb", ], "writeable": true, }, "736b9820-7e91-4775-84b0-78e71d60ce4c": { "replicates_to": [ "dcc5c518-73c7-4471-83e1-15fae67a98eb", ], "replicates_from": [ "4530736b-c1ff-4c07-9a9f-fa9bca418157", ], "writeable": true } } } } }
Other Examples
3 Node multi-master without nesting instances
Instance #1 (Master)
{ "instance": { "created": "2013-05-08T22:43:34", "id": "cfeccbf4-ac5b-494a-99b7-61593a6a71b0", "name": "multi-master-hotness-n01", "status": "ACTIVE", "updated": "2013-05-08T22:43:34", "service_type": "mysql-5.5", "links": [ { "href": "https://service/v1.0/1234/instances/cfeccbf4-ac5b-494a-99b7-61593a6a71b0", "rel": "self" }, { "href": "https://service/instances/cfeccbf4-ac5b-494a-99b7-61593a6a71b0", "rel": "bookmark" } ], "nodes": [ { "id": "dcc5c518-73c7-4471-83e1-15fae67a98eb", "hostname": "master-n01.example.com", "name": "master-n01", "flavor": { "id": "1", "links": [...] }, "links": [ { "href": "https://service/v1.0/1234/instances/{instance_id}/nodes/dcc5c518-73c7-4471-83e1-15fae67a98eb", "rel": "self" }, { "href": "https://service/instances/{instance_id}/nodes/dcc5c518-73c7-4471-83e1-15fae67a98eb", "rel": "bookmark" } ], "volume": { "size": 2, "used": 0.16368598397821188 }, }, ], "metadata": { "replication_contract": { // We should use something more clear than just nova instance // ids here but lets just use them for now for example. Maybe // links? names? not sure what works best yet. "dcc5c518-73c7-4471-83e1-15fae67a98eb": { "replicates_to": [ "4530736b-c1ff-4c07-9a9f-fa9bca418157", ], "replicates_from": [ "736b9820-7e91-4775-84b0-78e71d60ce4c", ], "writeable": true, }, } } } }
Instance #2 (Master)
{ "instance": { "created": "2013-05-08T22:43:34", "id": "fdff9fa2-abe3-4536-94dc-887a1bba4274", "name": "multi-master-hotness-n02", "status": "ACTIVE", "updated": "2013-05-08T22:43:34", "service_type": "mysql-5.5", "links": [ { "href": "https://service/v1.0/1234/instances/fdff9fa2-abe3-4536-94dc-887a1bba4274", "rel": "self" }, { "href": "https://service/instances/fdff9fa2-abe3-4536-94dc-887a1bba4274", "rel": "bookmark" } ], "nodes": [ { "id": "4530736b-c1ff-4c07-9a9f-fa9bca418157", "hostname": "master-n02.example.com", "name": "master-n02", "flavor": { "id": "1", "links": [...] }, "links": [ { "href": "https://service/v1.0/1234/instances/{instance_id}/nodes/4530736b-c1ff-4c07-9a9f-fa9bca418157", "rel": "self" }, { "href": "https://service/instances/{instance_id}/nodes/4530736b-c1ff-4c07-9a9f-fa9bca418157", "rel": "bookmark" } ], "volume": { "size": 2, "used": 0.16368598397821188 }, }, ], "metadata": { "replication_contract": { // We should use something more clear than just nova instance // ids here but lets just use them for now for example. Maybe // links? names? not sure what works best yet. "4530736b-c1ff-4c07-9a9f-fa9bca418157": { "replicates_to": [ "736b9820-7e91-4775-84b0-78e71d60ce4c", ], "replicates_from": [ "dcc5c518-73c7-4471-83e1-15fae67a98eb", ], "writeable": true, }, } } } }
Instance #3 (Master)
{ "instance": { "created": "2013-05-08T22:43:34", "id": "e97cffc3-e91c-4526-987e-711fa557891a", "name": "multi-master-hotness-n03", "status": "ACTIVE", "updated": "2013-05-08T22:43:34", "service_type": "mysql-5.5", "links": [ { "href": "https://service/v1.0/1234/instances/e97cffc3-e91c-4526-987e-711fa557891a", "rel": "self" }, { "href": "https://service/instances/e97cffc3-e91c-4526-987e-711fa557891a", "rel": "bookmark" } ], "nodes": [ { "id": "736b9820-7e91-4775-84b0-78e71d60ce4c", "hostname": "master-n03.example.com", "name": "master-n03", "flavor": { "id": "1", "links": [...] }, "links": [ { "href": "https://service/v1.0/1234/instances/{instance_id}/nodes/736b9820-7e91-4775-84b0-78e71d60ce4c", "rel": "self" }, { "href": "https://service/instances/{instance_id}/nodes/736b9820-7e91-4775-84b0-78e71d60ce4c", "rel": "bookmark" } ], "volume": { "size": 2, "used": 0.16368598397821188 }, }, ], "metadata": { "replication_contract": { // We should use something more clear than just nova instance // ids here but lets just use them for now for example. Maybe // links? names? not sure what works best yet. "736b9820-7e91-4775-84b0-78e71d60ce4c": { "replicates_to": [ "dcc5c518-73c7-4471-83e1-15fae67a98eb", ], "replicates_from": [ "4530736b-c1ff-4c07-9a9f-fa9bca418157", ], "writeable": true } } } } }
Replication without nesting slave instances with their masters
Instance #1 (Master)
{ "instance": { "created": "2013-05-08T22:43:34", "id": "cfeccbf4-ac5b-494a-99b7-61593a6a71b0", "name": "master-slave-sweetness-n01", "status": "ACTIVE", "updated": "2013-05-08T22:43:34", "service_type": "mysql-5.5", "links": [ { "href": "https://service/v1.0/1234/instances/cfeccbf4-ac5b-494a-99b7-61593a6a71b0", "rel": "self" }, { "href": "https://service/instances/cfeccbf4-ac5b-494a-99b7-61593a6a71b0", "rel": "bookmark" } ], "nodes": [ { "id": "dcc5c518-73c7-4471-83e1-15fae67a98eb", "hostname": "master-n01.example.com", "name": "master-n01", "flavor": { "id": "1", "links": [...] }, "links": [ { "href": "https://service/v1.0/1234/instances/{instance_id}/nodes/dcc5c518-73c7-4471-83e1-15fae67a98eb", "rel": "self" }, { "href": "https://service/instances/{instance_id}/nodes/dcc5c518-73c7-4471-83e1-15fae67a98eb", "rel": "bookmark" } ], "volume": { "size": 2, "used": 0.16368598397821188 }, }, ], "metadata": { "replication_contract": { // We should use something more clear than just nova instance // ids here but lets just use them for now for example. Maybe // links? names? not sure what works best yet. "dcc5c518-73c7-4471-83e1-15fae67a98eb": { "replicates_to": [ "4530736b-c1ff-4c07-9a9f-fa9bca418157", ], "writeable": true, }, } } } }
Instance #2 (Read-only Slave)
{ "instance": { "created": "2013-05-08T22:43:34", "id": "fdff9fa2-abe3-4536-94dc-887a1bba4274", "name": "master-slave-sweetness-n02", "status": "ACTIVE", "updated": "2013-05-08T22:43:34", "service_type": "mysql-5.5", "links": [ { "href": "https://service/v1.0/1234/instances/fdff9fa2-abe3-4536-94dc-887a1bba4274", "rel": "self" }, { "href": "https://service/instances/fdff9fa2-abe3-4536-94dc-887a1bba4274", "rel": "bookmark" } ], "nodes": [ { "id": "4530736b-c1ff-4c07-9a9f-fa9bca418157", "hostname": "master-n02.example.com", "name": "master-n02", "flavor": { "id": "1", "links": [...] }, "links": [ { "href": "https://service/v1.0/1234/instances/{instance_id}/nodes/4530736b-c1ff-4c07-9a9f-fa9bca418157", "rel": "self" }, { "href": "https://service/instances/{instance_id}/nodes/4530736b-c1ff-4c07-9a9f-fa9bca418157", "rel": "bookmark" } ], "volume": { "size": 2, "used": 0.16368598397821188 }, }, ], "metadata": { "replication_contract": { // We should use something more clear than just nova instance // ids here but lets just use them for now for example. Maybe // links? names? not sure what works best yet. "4530736b-c1ff-4c07-9a9f-fa9bca418157": { "replicates_from": [ "dcc5c518-73c7-4471-83e1-15fae67a98eb", ], "writeable": false, }, } } } }
Clustering and Replication
A cluster is an independent service type.
- This simplifies the configurations (ties to service_type)
- The guest impl can be different for each service_type
- We don't need cluster type and service_type when defining a cluster
Think: A instance is an instance is a cluster is an instance is NOT replication
CLUSTERING
Definition of cluster for purposes of Trove:
A cluster must provide
- High Availablility
- Fault Tolerance
A cluster object is represented as an instance with a metadata attribute
containing node information. A node is a sub-resource of a cluster and
has a limited number of actions that can be performed on them. A cluster
is made of of equal sized nodes using all the same flavor/disk size.
Cluster Operations
Create cluster:
Create a cluster with flavor X and size Y. All nodes are of equal size.
POST /instances
{ "name": "foobar", "flavor": "{flavor_id}", "service_type": "{service_type_id}", "volume": { "size": 50, }, "metadata": { "nodes": 5, "valuable_cluster_info": {} } }
Delete cluster:
- Deletes all nodes in the cluster and the cluster itself.
DELETE /instances/{id} { "empty body?" }
Downsize a cluster (Delete nodes):
PATCH /instances/{id}
{ "metadata": { "nodes": 3 } }
Or remove specific nodes from the cluster. e.g. You have a node acting up in the cluster and you want to pull it out PATCH /instances/{id}
{ "nodes": [ {"id": "32e3c3b8-8d71-4b76-b2be-27fec62bc302"}, {"id": "3c6176fc-bb33-429f-a926-e8307c115752"}, ] }
Restart a cluster:
POST /instances/{id}/restart
{ "empty body?" }
Resize a cluster:
POST /instances/{id}/resize
{ "flavor": "flavor_id", "volume": { "size": 100 } }
Initialize cluster:
Clusters can get mucked up sometimes, you may want to keep a node in the cluster but wipe it's data and let the cluster technology repopulate it's data. POST /instance/{id}/initialize
{ "nodes": [ {"id": "32e3c3b8-8d71-4b76-b2be-27fec62bc302"}, {"id": "3c6176fc-bb33-429f-a926-e8307c115752"}, ] }
Rebalance/restripe cluster:
Changing the hashing algorithm on some cluster technologies may require a rebalance or restriping of the data. This is a maintenance method to support that. POST /instance/{id}/rebalance
{ "nodes": [ {"id": "32e3c3b8-8d71-4b76-b2be-27fec62bc302"}, {"id": "3c6176fc-bb33-429f-a926-e8307c115752"}, ] }
Node Operations:
Restart a node:
POST /instance/{id}/node/{id}/restart
{ "empty body?" }
Initialize a node:
Clusters can get mucked up sometimes, you may want to keep a node in the cluster but wipe it's data and let the cluster technology repopulate it's data. POST /instance/{id}/node/{id}/initialize
{ "empty body?" }
Rebalance/restripe cluster:
Changing the hashing algorithm on some cluster technologies may require a rebalance or restriping of the data. This is a maintenance method to support that. POST /instance/{id}/node/{id}/rebalance
{ "empty body?" }
REPLICATION
Think of replication as a capability of some service types. Not every service type will have this capability and it should be configurable via the service types capabilities feature.
What you cannot do:
- create a master and slave in one call
- one call per action means simplified workflow
Replication workflow:
- Create an instance
- Create an Nth instance with
- Caveat: adding a replicated slave will alter data on the master instance and will use the agent to create the replication.
{ "metadata": { "replicates_from": [ "{master_instance_id}" ], // The writeable flag should actually not be required but would populate based on // the underlying service type's replication setup. It is an informative flag only // and shouldn't block users. "writeable": false } }
Replication Examples
Examples of what data returned back to the user when they query detailed instance info from the api with: GET /instances/<instance_id>
Master Instance
{ "name": "blah", "id": "5c489b9d-96bb-4e15-a79d-412d88a0062e", etc... "metadata": { "replication_contract": { "replicates_to": [ "7ee9664f-1261-4076-8abc-21af3a092132", ], "writeable": true } } }
Slave Instance (Read-Only)
{ "name": "blah-slave", "id": "7ee9664f-1261-4076-8abc-21af3a092132", etc... "metadata": { "replication_contract": { "replicates_from": [ "5c489b9d-96bb-4e15-a79d-412d88a0062e", ], "writeable": true } } }
Replication Operations
Create Replication:
POST /instance
{ "name": "foobar", "flavor": "{flavor_id}", "service_type": "mysql-5.1", "size": 50, "metadata": { "replication_contract": { "replicates_from": [ "32e3c3b8-8d71-4b76-b2be-27fec62bc302", ] } } }