Difference between revisions of "Trove-Replication-And-Clustering-API"
Imsplitbit (talk | contribs) |
Imsplitbit (talk | contribs) |
||
| Line 408: | Line 408: | ||
"created": "2013-05-08T22:43:34", | "created": "2013-05-08T22:43:34", | ||
"id": "cfeccbf4-ac5b-494a-99b7-61593a6a71b0", | "id": "cfeccbf4-ac5b-494a-99b7-61593a6a71b0", | ||
| − | "name": " | + | "name": "master-slave-sweetness-n01", |
"status": "ACTIVE", | "status": "ACTIVE", | ||
"updated": "2013-05-08T22:43:34", | "updated": "2013-05-08T22:43:34", | ||
| Line 467: | Line 467: | ||
"created": "2013-05-08T22:43:34", | "created": "2013-05-08T22:43:34", | ||
"id": "cfeccbf4-ac5b-494a-99b7-61593a6a71b0", | "id": "cfeccbf4-ac5b-494a-99b7-61593a6a71b0", | ||
| − | "name": " | + | "name": "master-slave-sweetness-n02", |
"status": "ACTIVE", | "status": "ACTIVE", | ||
"updated": "2013-05-08T22:43:34", | "updated": "2013-05-08T22:43:34", | ||
Revision as of 15:38, 10 October 2013
Contents
Instance Object Changes
Part of this proposal includes adding new properties to the json structure of an instance. The first would be to add a "nodes" property which would be list of refs back to the nova instance(s) that make up the trove instance.
The second would be a metadata storage that is of an open format to be used for storing critical data that is pertinent to the trove instance.
INSTANCES
Nodes
Because trove instances are a service type with an underlying nova resource(s) we can better display the nova resource(s) that make up the instance of a service type. When looking at a singular trove instance of something like redis or mysql the knowledge of the instance resource isn't all that important. Currently trove knows about the instance from a nova perspective but masks most of this information. This would look something like this:
GET /instances/{id}
{
"instance": {
"created": "2013-05-08T22:43:34",
"hostname": "mysweetdb.example.com",
"id": "dcc5c518-73c7-4471-83e1-15fae67a98eb",
"name": "json_rack_instance",
"status": "ACTIVE",
"updated": "2013-05-08T22:43:34",
"service_type": "mysql-5.1",
"nodes": [
{
"flavor": {
"id": "1",
"links": [
{
"href": "https://service/v1.0/1234/flavors/1",
"rel": "self"
},
{
"href": "https://service/flavors/1",
"rel": "bookmark"
}
]
},
"links": [
{
"href": "https://service/v1.0/1234/instances/dcc5c518-73c7-4471-83e1-15fae67a98eb",
"rel": "self"
},
{
"href": "https://service/instances/dcc5c518-73c7-4471-83e1-15fae67a98eb",
"rel": "bookmark"
}
],
"volume": {
"size": 2,
"used": 0.16368598397821188
},
},
],
}
}
Metadata
The metadata looks much like our previous proposal. It contains critical information to the configuration or topology of the cluster or replication agreement. This should look something like this:
This would be an example of a 3 node mult-master mysql replication contract GET /instances/{id}
{
"instance": {
"created": "2013-05-08T22:43:34",
"id": "cfeccbf4-ac5b-494a-99b7-61593a6a71b0",
"name": "multi-master-hotness",
"status": "ACTIVE",
"updated": "2013-05-08T22:43:34",
"service_type": "mysql-5.5",
"nodes": [
{
"id": "dcc5c518-73c7-4471-83e1-15fae67a98eb",
"hostname": "master-n01.example.com",
"flavor": {
"id": "1",
"links": [
{
"href": "https://service/v1.0/1234/flavors/1",
"rel": "self"
},
{
"href": "https://service/flavors/1",
"rel": "bookmark"
}
]
},
"links": [
{
"href": "https://service/v1.0/1234/instances/dcc5c518-73c7-4471-83e1-15fae67a98eb",
"rel": "self"
},
{
"href": "https://service/instances/dcc5c518-73c7-4471-83e1-15fae67a98eb",
"rel": "bookmark"
}
],
"volume": {
"size": 2,
"used": 0.16368598397821188
},
},
{
"id": "4530736b-c1ff-4c07-9a9f-fa9bca418157",
"hostname": "master-n02.example.com",
"flavor": {
"id": "1",
"links": [
{
"href": "https://service/v1.0/1234/flavors/1",
"rel": "self"
},
{
"href": "https://service/flavors/1",
"rel": "bookmark"
}
]
},
"links": [
{
"href": "https://service/v1.0/1234/instances/4530736b-c1ff-4c07-9a9f-fa9bca418157",
"rel": "self"
},
{
"href": "https://service/instances/4530736b-c1ff-4c07-9a9f-fa9bca418157",
"rel": "bookmark"
}
],
"volume": {
"size": 2,
"used": 0.16368598397821188
},
},
{
"id": "736b9820-7e91-4775-84b0-78e71d60ce4c",
"hostname": "master-n03.example.com",
"flavor": {
"id": "1",
"links": [
{
"href": "https://service/v1.0/1234/flavors/1",
"rel": "self"
},
{
"href": "https://service/flavors/1",
"rel": "bookmark"
}
]
},
"links": [
{
"href": "https://service/v1.0/1234/instances/736b9820-7e91-4775-84b0-78e71d60ce4c",
"rel": "self"
},
{
"href": "https://service/instances/736b9820-7e91-4775-84b0-78e71d60ce4c",
"rel": "bookmark"
}
],
"volume": {
"size": 2,
"used": 0.16368598397821188
},
},
],
"metadata": {
"replication_contract": {
// We should use something more clear than just nova instance
// ids here but lets just use them for now for example. Maybe
// links? names? not sure what works best yet.
"dcc5c518-73c7-4471-83e1-15fae67a98eb": {
"replicates_to": [
"4530736b-c1ff-4c07-9a9f-fa9bca418157",
],
"replicates_from": [
"736b9820-7e91-4775-84b0-78e71d60ce4c",
],
"writeable": true,
},
"4530736b-c1ff-4c07-9a9f-fa9bca418157": {
"replicates_to": [
"736b9820-7e91-4775-84b0-78e71d60ce4c",
],
"replicates_from": [
"dcc5c518-73c7-4471-83e1-15fae67a98eb",
],
"writeable": true,
},
"736b9820-7e91-4775-84b0-78e71d60ce4c": {
"replicates_to": [
"dcc5c518-73c7-4471-83e1-15fae67a98eb",
],
"replicates_from": [
"4530736b-c1ff-4c07-9a9f-fa9bca418157",
],
"writeable": true
}
}
}
}
}
Other Examples
3 Node multi-master without nesting instances
Instance #1 (Master)
{
"instance": {
"created": "2013-05-08T22:43:34",
"id": "cfeccbf4-ac5b-494a-99b7-61593a6a71b0",
"name": "multi-master-hotness-n01",
"status": "ACTIVE",
"updated": "2013-05-08T22:43:34",
"service_type": "mysql-5.5",
"nodes": [
{
"id": "dcc5c518-73c7-4471-83e1-15fae67a98eb",
"hostname": "master-n01.example.com",
"flavor": {
"id": "1",
"links": [
{
"href": "https://service/v1.0/1234/flavors/1",
"rel": "self"
},
{
"href": "https://service/flavors/1",
"rel": "bookmark"
}
]
},
"links": [
{
"href": "https://service/v1.0/1234/instances/dcc5c518-73c7-4471-83e1-15fae67a98eb",
"rel": "self"
},
{
"href": "https://service/instances/dcc5c518-73c7-4471-83e1-15fae67a98eb",
"rel": "bookmark"
}
],
"volume": {
"size": 2,
"used": 0.16368598397821188
},
},
],
"metadata": {
"replication_contract": {
// We should use something more clear than just nova instance
// ids here but lets just use them for now for example. Maybe
// links? names? not sure what works best yet.
"dcc5c518-73c7-4471-83e1-15fae67a98eb": {
"replicates_to": [
"4530736b-c1ff-4c07-9a9f-fa9bca418157",
],
"replicates_from": [
"736b9820-7e91-4775-84b0-78e71d60ce4c",
],
"writeable": true,
},
}
}
}
}
Instance #2 (Master)
{
"instance": {
"created": "2013-05-08T22:43:34",
"id": "cfeccbf4-ac5b-494a-99b7-61593a6a71b0",
"name": "multi-master-hotness-n02",
"status": "ACTIVE",
"updated": "2013-05-08T22:43:34",
"service_type": "mysql-5.5",
"nodes": [
{
"id": "4530736b-c1ff-4c07-9a9f-fa9bca418157",
"hostname": "master-n02.example.com",
"flavor": {
"id": "1",
"links": [
{
"href": "https://service/v1.0/1234/flavors/1",
"rel": "self"
},
{
"href": "https://service/flavors/1",
"rel": "bookmark"
}
]
},
"links": [
{
"href": "https://service/v1.0/1234/instances/4530736b-c1ff-4c07-9a9f-fa9bca418157",
"rel": "self"
},
{
"href": "https://service/instances/4530736b-c1ff-4c07-9a9f-fa9bca418157",
"rel": "bookmark"
}
],
"volume": {
"size": 2,
"used": 0.16368598397821188
},
},
],
"metadata": {
"replication_contract": {
// We should use something more clear than just nova instance
// ids here but lets just use them for now for example. Maybe
// links? names? not sure what works best yet.
"4530736b-c1ff-4c07-9a9f-fa9bca418157": {
"replicates_to": [
"736b9820-7e91-4775-84b0-78e71d60ce4c",
],
"replicates_from": [
"dcc5c518-73c7-4471-83e1-15fae67a98eb",
],
"writeable": true,
},
}
}
}
}
Instance #3 (Master)
{
"instance": {
"created": "2013-05-08T22:43:34",
"id": "cfeccbf4-ac5b-494a-99b7-61593a6a71b0",
"name": "multi-master-hotness-n03",
"status": "ACTIVE",
"updated": "2013-05-08T22:43:34",
"service_type": "mysql-5.5",
"nodes": [
{
"id": "736b9820-7e91-4775-84b0-78e71d60ce4c",
"hostname": "master-n03.example.com",
"flavor": {
"id": "1",
"links": [
{
"href": "https://service/v1.0/1234/flavors/1",
"rel": "self"
},
{
"href": "https://service/flavors/1",
"rel": "bookmark"
}
]
},
"links": [
{
"href": "https://service/v1.0/1234/instances/736b9820-7e91-4775-84b0-78e71d60ce4c",
"rel": "self"
},
{
"href": "https://service/instances/736b9820-7e91-4775-84b0-78e71d60ce4c",
"rel": "bookmark"
}
],
"volume": {
"size": 2,
"used": 0.16368598397821188
},
},
],
"metadata": {
"replication_contract": {
// We should use something more clear than just nova instance
// ids here but lets just use them for now for example. Maybe
// links? names? not sure what works best yet.
"736b9820-7e91-4775-84b0-78e71d60ce4c": {
"replicates_to": [
"dcc5c518-73c7-4471-83e1-15fae67a98eb",
],
"replicates_from": [
"4530736b-c1ff-4c07-9a9f-fa9bca418157",
],
"writeable": true
}
}
}
}
}
Replication without nesting slave instances with their masters
Instance #1 (Master)
{
"instance": {
"created": "2013-05-08T22:43:34",
"id": "cfeccbf4-ac5b-494a-99b7-61593a6a71b0",
"name": "master-slave-sweetness-n01",
"status": "ACTIVE",
"updated": "2013-05-08T22:43:34",
"service_type": "mysql-5.5",
"nodes": [
{
"id": "dcc5c518-73c7-4471-83e1-15fae67a98eb",
"hostname": "master-n01.example.com",
"flavor": {
"id": "1",
"links": [
{
"href": "https://service/v1.0/1234/flavors/1",
"rel": "self"
},
{
"href": "https://service/flavors/1",
"rel": "bookmark"
}
]
},
"links": [
{
"href": "https://service/v1.0/1234/instances/dcc5c518-73c7-4471-83e1-15fae67a98eb",
"rel": "self"
},
{
"href": "https://service/instances/dcc5c518-73c7-4471-83e1-15fae67a98eb",
"rel": "bookmark"
}
],
"volume": {
"size": 2,
"used": 0.16368598397821188
},
},
],
"metadata": {
"replication_contract": {
// We should use something more clear than just nova instance
// ids here but lets just use them for now for example. Maybe
// links? names? not sure what works best yet.
"dcc5c518-73c7-4471-83e1-15fae67a98eb": {
"replicates_to": [
"4530736b-c1ff-4c07-9a9f-fa9bca418157",
],
"writeable": true,
},
}
}
}
}
Instance #2 (Read-only Slave)
{
"instance": {
"created": "2013-05-08T22:43:34",
"id": "cfeccbf4-ac5b-494a-99b7-61593a6a71b0",
"name": "master-slave-sweetness-n02",
"status": "ACTIVE",
"updated": "2013-05-08T22:43:34",
"service_type": "mysql-5.5",
"nodes": [
{
"id": "4530736b-c1ff-4c07-9a9f-fa9bca418157",
"hostname": "master-n02.example.com",
"flavor": {
"id": "1",
"links": [
{
"href": "https://service/v1.0/1234/flavors/1",
"rel": "self"
},
{
"href": "https://service/flavors/1",
"rel": "bookmark"
}
]
},
"links": [
{
"href": "https://service/v1.0/1234/instances/4530736b-c1ff-4c07-9a9f-fa9bca418157",
"rel": "self"
},
{
"href": "https://service/instances/4530736b-c1ff-4c07-9a9f-fa9bca418157",
"rel": "bookmark"
}
],
"volume": {
"size": 2,
"used": 0.16368598397821188
},
},
],
"metadata": {
"replication_contract": {
// We should use something more clear than just nova instance
// ids here but lets just use them for now for example. Maybe
// links? names? not sure what works best yet.
"4530736b-c1ff-4c07-9a9f-fa9bca418157": {
"replicates_from": [
"dcc5c518-73c7-4471-83e1-15fae67a98eb",
],
"writeable": false,
},
}
}
}
}
Clustering and Replication
A cluster is an independent service type.
- This simplifies the configurations (ties to service_type)
- The guest impl can be different for each service_type
- We don't need cluster type and service_type when defining a cluster
Think: A instance is an instance is a cluster is an instance is NOT replication
CLUSTERING
Definition of cluster for purposes of Trove:
A cluster must provide
- High Availablility
- Fault Tolerance
A cluster object is represented as an instance with a metadata attribute
containing node information. A node is a sub-resource of a cluster and
has a limited number of actions that can be performed on them. A cluster
is made of of equal sized nodes using all the same flavor/disk size.
Cluster Operations
Create cluster:
Create a cluster with flavor X and size Y. All nodes are of equal size.
POST /instances
{
"name": "foobar",
"flavor": "{flavor_id}",
"service_type": "{service_type_id}",
"volume": {
"size": 50,
},
"metadata": {
"nodes": 5,
"valuable_cluster_info": {}
}
}
Delete cluster:
- Deletes all nodes in the cluster and the cluster itself.
DELETE /instances/{id} { "empty body?" }
Downsize a cluster (Delete nodes):
PATCH /instances/{id}
{
"metadata": {
"nodes": 3
}
}
Or remove specific nodes from the cluster. e.g. You have a node acting up in the cluster and you want to pull it out PATCH /instances/{id}
{
"nodes": [
{"id": "32e3c3b8-8d71-4b76-b2be-27fec62bc302"},
{"id": "3c6176fc-bb33-429f-a926-e8307c115752"},
]
}
Restart a cluster:
POST /instances/{id}/restart
{
"empty body?"
}
Resize a cluster:
POST /instances/{id}/resize
{
"flavor": "flavor_id",
"volume": {
"size": 100
}
}
Initialize cluster:
Clusters can get mucked up sometimes, you may want to keep a node in the cluster but wipe it's data and let the cluster technology repopulate it's data. POST /instance/{id}/initialize
{
"nodes": [
{"id": "32e3c3b8-8d71-4b76-b2be-27fec62bc302"},
{"id": "3c6176fc-bb33-429f-a926-e8307c115752"},
]
}
Rebalance/restripe cluster:
Changing the hashing algorithm on some cluster technologies may require a rebalance or restriping of the data. This is a maintenance method to support that. POST /instance/{id}/rebalance
{
"nodes": [
{"id": "32e3c3b8-8d71-4b76-b2be-27fec62bc302"},
{"id": "3c6176fc-bb33-429f-a926-e8307c115752"},
]
}
Node Operations:
Restart a node:
POST /instance/{id}/node/{id}/restart
{
"empty body?"
}
Initialize a node:
Clusters can get mucked up sometimes, you may want to keep a node in the cluster but wipe it's data and let the cluster technology repopulate it's data. POST /instance/{id}/node/{id}/initialize
{
"empty body?"
}
Rebalance/restripe cluster:
Changing the hashing algorithm on some cluster technologies may require a rebalance or restriping of the data. This is a maintenance method to support that. POST /instance/{id}/node/{id}/rebalance
{
"empty body?"
}
REPLICATION
Think of replication as a capability of some service types. Not every service type will have this capability and it should be configurable via the service types capabilities feature.
What you cannot do:
- create a master and slave in one call
- one call per action means simplified workflow
Replication workflow:
- Create an instance
- Create an Nth instance with
- Caveat: adding a replicated slave will alter data on the master instance and will use the agent to create the replication.
{
"metadata": {
"replicates_from": [
"{master_instance_id}"
],
// The writeable flag should actually not be required but would populate based on
// the underlying service type's replication setup. It is an informative flag only
// and shouldn't block users.
"writeable": false
}
}
Replication Operations
Create Replication:
POST /instance
{
"name": "foobar",
"flavor": "{flavor_id}",
"service_type": "mysql-5.1",
"size": 50,
"metadata": {
"replication_contract": {
"replicates_from": [
"32e3c3b8-8d71-4b76-b2be-27fec62bc302",
]
}
}
}
