Trove/Replication-And-Clustering-With-Nodes-3
Contents
Example: Cassandra
To illustrate the approach, Cassandra is used in the examples below. The eccentricities of each Datastore will be explained in their own sections.
Create Cluster
Request:
POST /instances { "instance": { "name": "products", "datastore": { "type": "cassandra", "version": "2.0.6" }, "configuration": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6", "flavorRef": "7", "volume": { "size": 1 }, "cluster": { "size": 3, "nodes": [ {"region": "phx"}, {"region": "slc"}, {"region": "lvs"} ] } } }
Response:
{ "instance": { "status": "BUILD", "id": "dfbbd9ca-b5e1-4028-adb7-f78643e17998", "name": "products", "created": "2014-04-25T20:19:23", "updated": "2014-04-25T20:19:23", "links": [{...}], "datastore": { "type": "cassandra", "version": "2.0.6" }, "cluster": { "size": 3, "nodes": [ {"id": "416b0b16-ba55-4302-bbd3-ff566032e1c1", "region": "phx"}, {"id": "7f52e4f9-3fa6-4238-ac08-1ce15197329a", "region": "slc"}, {"id": "ff9d680c-fde3-49c6-a84e-76173b6df39d", "region": "lvs"} ] } } }
Notes:
- For Phase One:
- cluster.nodes{} will not be supported.
- cluster.nodes[].region will not be returned.
- For Phase Two:
- if cluster.allocations{} is not provided, the current region is assumed.
- cluster.nodes[].region will always be returned.
- Cassandra-specific fields that are required to construct the initial cluster (num_tokens, endpoint_snitch, seed ip-list, etc.) are to be determined/calculated based on configuration file values and common-sense.
Show Cluster
Request:
GET /instances/dfbbd9ca-b5e1-4028-adb7-f78643e17998
Response:
{ "instance": { "status": "ACTIVE", "id": "dfbbd9ca-b5e1-4028-adb7-f78643e17998", "name": "products", "created": "2014-04-25T20:19:23", "updated": "2014-04-25T20:19:23", "links": [{...}], "datastore": { "type": "cassandra", "version": "2.0.6" }, "cluster": { "size": 3, "nodes": [ {"id": "416b0b16-ba55-4302-bbd3-ff566032e1c1", "region": "phx"}, {"id": "7f52e4f9-3fa6-4238-ac08-1ce15197329a", "region": "slc"}, {"id": "ff9d680c-fde3-49c6-a84e-76173b6df39d", "region": "lvs"} ] } } }
Notes:
- In Phase One:
- cluster.nodes[].region will not be returned.
- Change: instance.volume.used, instance.ip[], and instance.hostname will never be returned
- It's possible that instance.ip[] can remain if it only returns the seed ips.
- It's possible that instance.hostname can remain if it's converted to an array and only contains the seed hostnames.
Show Node
Request:
GET /instances/dfbbd9ca-b5e1-4028-adb7-f78643e17998/nodes/416b0b16-ba55-4302-bbd3-ff566032e1c1
Response:
{ "node": { "status": "ACTIVE", "id": "416b0b16-ba55-4302-bbd3-ff566032e1c1", "name": "products-1", "created": "2014-04-25T20:19:23", "updated": "2014-04-25T20:19:23", "links": [{...}], "ip": ["10.0.0.1"], "configuration": { "id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6", "links": [{...}], }, "flavor": { "id": "7", "links": [{...}], }, "volume": { "size": 2, "used": 0.17 } } }
Add Node(s)
Request:
POST /instances/dfbbd9ca-b5e1-4028-adb7-f78643e17998/nodes { "nodes": { "num": 2, "allocations": [ {"region": "phx"}, {"region": "phx"} ] } }
Response:
HTTP 202 (Empty Body)
Notes:
- For Phase One:
- nodes.num will be the only supported field (nodes.allocations will not)
- For Phase Two:
- if nodes.allocations[] is not provided, the region of every existing node must match, otherwise the request is failed.
Replace Node
Request:
POST /instances/dfbbd9ca-b5e1-4028-adb7-f78643e17998/action { "replace_node": { "id": "7f52e4f9-3fa6-4238-ac08-1ce15197329a" } }
Response:
HTTP 202 (Empty Body)
Notes:
- http://www.datastax.com/documentation/cassandra/2.0/cassandra/operations/ops_replace_live_node.html
- http://www.datastax.com/documentation/cassandra/2.0/cassandra/operations/ops_replace_seed_node.html
- http://www.datastax.com/documentation/cassandra/2.0/cassandra/operations/ops_replace_node_t.html
Remove Node
Request:
DELETE /instances/dfbbd9ca-b5e1-4028-adb7-f78643e17998/nodes/7f52e4f9-3fa6-4238-ac08-1ce15197329a
Response:
HTTP 202 (Empty Body)
Notes:
Example: MongoDB
Create Cluster
Request:
POST /instances { "instance": { "name": "products", "datastore": { "type": "mongodb", "version": "2.4.10" }, "configuration": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6", "flavorRef": "7", "volume": { "size": 1 }, "cluster": { "size": 5, "nodes": [ {"region": "phx"}, {"region": "phx"}, {"region": "phx"}, {"region": "slc"}, {"region": "slc"} ] } } }
Response:
{ "instance": { "status": "BUILD", "id": "dfbbd9ca-b5e1-4028-adb7-f78643e17998", "name": "products", "created": "2014-04-25T20:19:23", "updated": "2014-04-25T20:19:23", "links": [{...}], "datastore": { "type": "mongodb", "version": "2.4.10" }, "cluster": { "size": 5, "nodes": [ {"id": "416b0b16-ba55-4302-bbd3-ff566032e1c1", "region": "phx"}, {"id": "965ef811-7c1d-47fc-89f2-a89dfdd23ef2", "region": "phx"}, {"id": "3642f41c-e8ad-4164-a089-3891bf7f2d2b", "region": "phx"}, {"id": "7f52e4f9-3fa6-4238-ac08-1ce15197329a", "region": "slc"}, {"id": "ff9d680c-fde3-49c6-a84e-76173b6df39d", "region": "slc"} ] } } }
Show Cluster
Request:
GET /instances/dfbbd9ca-b5e1-4028-adb7-f78643e17998
Response:
{ "instance": { "status": "ACTIVE", "id": "dfbbd9ca-b5e1-4028-adb7-f78643e17998", "name": "products", "created": "2014-04-25T20:19:23", "updated": "2014-04-25T20:19:23", "links": [{...}], "datastore": { "type": "mongodb", "version": "2.4.10" }, "cluster": { "size": 5, "nodes": [ {"id": "416b0b16-ba55-4302-bbd3-ff566032e1c1", "region": "phx"}, {"id": "965ef811-7c1d-47fc-89f2-a89dfdd23ef2", "region": "phx"}, {"id": "3642f41c-e8ad-4164-a089-3891bf7f2d2b", "region": "phx"}, {"id": "7f52e4f9-3fa6-4238-ac08-1ce15197329a", "region": "slc"}, {"id": "ff9d680c-fde3-49c6-a84e-76173b6df39d", "region": "slc"} ] } } }
Show Node
Request:
GET /instances/dfbbd9ca-b5e1-4028-adb7-f78643e17998/nodes/416b0b16-ba55-4302-bbd3-ff566032e1c1
Response:
{ "node": { "status": "ACTIVE", "id": "416b0b16-ba55-4302-bbd3-ff566032e1c1", "name": "products-1", "created": "2014-04-25T20:19:23", "updated": "2014-04-25T20:19:23", "links": [{...}], "ip": ["10.0.0.1"], "configuration": { "id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6", "links": [{...}], }, "flavor": { "id": "7", "links": [{...}], }, "volume": { "size": 2, "used": 0.17 } } }
Create Arbiter(s)
Request:
POST /instances/dfbbd9ca-b5e1-4028-adb7-f78643e17998/nodes { "nodes": { "num": 2, "allocations": [ {"region": "lvs", "type": "arbiter"}, {"region": "lvs", "type": "arbiter"} ] } }
Response:
HTTP 202 (Empty Body)
Show Cluster (After Arbiters)
Request:
GET /instances/dfbbd9ca-b5e1-4028-adb7-f78643e17998
Response:
{ "instance": { "status": "ACTIVE", "id": "dfbbd9ca-b5e1-4028-adb7-f78643e17998", "name": "products", "created": "2014-04-25T20:19:23", "updated": "2014-04-25T20:19:23", "links": [{...}], "datastore": { "type": "mongodb", "version": "2.4.10" }, "cluster": { "size": 7, "nodes": [ {"id": "416b0b16-ba55-4302-bbd3-ff566032e1c1", "region": "phx"}, {"id": "965ef811-7c1d-47fc-89f2-a89dfdd23ef2", "region": "phx"}, {"id": "3642f41c-e8ad-4164-a089-3891bf7f2d2b", "region": "phx"}, {"id": "7f52e4f9-3fa6-4238-ac08-1ce15197329a", "region": "slc"}, {"id": "ff9d680c-fde3-49c6-a84e-76173b6df39d", "region": "slc"}, {"id": "77032c55-4496-4e35-8c0d-6cd1c18e1a9c", "region": "lvs", "type": "arbiter"}, {"id": "1fd054ed-221f-4c99-8d17-570bcff4c1d2", "region": "lvs", "type": "arbiter"} ] } } }
Example: MySQL
Create Master
Request:
POST /instances { "instance": { "name": "products", "datastore": { "type": "mysql", "version": "5.5" }, "configuration": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6", "flavorRef": "7", "volume": { "size": 1 } } }
Response:
{ "instance": { "status": "BUILD", "id": "dfbbd9ca-b5e1-4028-adb7-f78643e17998", "name": "products", "created": "2014-04-25T20:19:23", "updated": "2014-04-25T20:19:23", "links": [{...}], "datastore": { "type": "mysql", "version": "5.5" }, "configuration": { "id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6", "links": [{...}], }, "flavor": { "id": "7", "links": [{...}], }, "volume": { "size": 1 } } }
Create Slave
Request:
POST /instances { "instance": { "name": "products-slave", "datastore": { "type": "mysql", "version": "5.5" }, "configuration": "fc318e00-3a6f-4f93-af99-146b44912188", "flavorRef": "7", "volume": { "size": 1 }, "slave": { "of": "dfbbd9ca-b5e1-4028-adb7-f78643e17998", "read_only": true } } }
Response:
{ "instance": { "status": "BUILD", "id": "061aaf4c-3a57-411e-9df9-2d0f813db859", "name": "products", "created": "2014-04-25T20:19:23", "updated": "2014-04-25T20:19:23", "links": [{...}], "datastore": { "type": "mysql", "version": "5.5" }, "configuration": { "id": "fc318e00-3a6f-4f93-af99-146b44912188", "links": [{...}], }, "flavor": { "id": "7", "links": [{...}], }, "volume": { "size": 1 }, "slave": { "of": "dfbbd9ca-b5e1-4028-adb7-f78643e17998", "read_only": true } } }
Show Master
Request:
GET /instances/dfbbd9ca-b5e1-4028-adb7-f78643e17998
Response:
{ "instance": { "status": "ACTIVE", "id": "dfbbd9ca-b5e1-4028-adb7-f78643e17998", "name": "products", "created": "2014-04-25T20:19:23", "updated": "2014-04-25T20:19:23", "links": [{...}], "datastore": { "type": "mysql", "version": "5.5" }, "configuration": { "id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6", "links": [{...}], }, "flavor": { "id": "7", "links": [{...}], }, "volume": { "size": 1 }, "slave": { "list": [ {"id": "061aaf4c-3a57-411e-9df9-2d0f813db859"} ] } } }
Show Slave
Request:
GET /instances/061aaf4c-3a57-411e-9df9-2d0f813db859
Response:
{ "instance": { "status": "ACTIVE", "id": "061aaf4c-3a57-411e-9df9-2d0f813db859", "name": "products", "created": "2014-04-25T20:19:23", "updated": "2014-04-25T20:19:23", "links": [{...}], "datastore": { "type": "mysql", "version": "5.5" }, "configuration": { "id": "fc318e00-3a6f-4f93-af99-146b44912188", "links": [{...}], }, "flavor": { "id": "7", "links": [{...}], }, "volume": { "size": 1 }, "slave": { "of": "dfbbd9ca-b5e1-4028-adb7-f78643e17998", "read_only": true } } }
Promote (Detach) Slave
Request:
POST /instances/061aaf4c-3a57-411e-9df9-2d0f813db859/action { "promote": {} }
Response:
HTTP 202 (Empty Body)
Delete Slave
Request:
DELETE /instances/061aaf4c-3a57-411e-9df9-2d0f813db859
Response:
HTTP 202 (Empty Body)
Data Model Changes
Nodes Table
Create a new 'nodes' Table:
CREATE TABLE "nodes" ( "id" varchar(36) NOT NULL, "instance_id" varchar(36) NOT NULL, "created" datetime DEFAULT NULL, "updated" datetime DEFAULT NULL, "name" varchar(255) DEFAULT NULL, "hostname" varchar(255) DEFAULT NULL, "compute_instance_id" varchar(36) DEFAULT NULL, "task_id" int(11) DEFAULT NULL, "task_description" varchar(32) DEFAULT NULL, "task_start_time" datetime DEFAULT NULL, "volume_id" varchar(36) DEFAULT NULL, "flavor_id" int(11) DEFAULT NULL, "volume_size" int(11) DEFAULT NULL, "tenant_id" varchar(36) DEFAULT NULL, "server_status" varchar(64) DEFAULT NULL, "deleted" tinyint(1) DEFAULT NULL, "deleted_at" datetime DEFAULT NULL, "datastore_version_id" varchar(36) NOT NULL, "configuration_id" varchar(36) DEFAULT NULL, PRIMARY KEY ("id"), KEY "instance_id" ("instance_id"), KEY "datastore_version_id" ("datastore_version_id"), KEY "configuration_id" ("configuration_id"), KEY "instances_tenant_id" ("tenant_id"), KEY "instances_deleted" ("deleted"), CONSTRAINT "nodes_ibfk_3" FOREIGN KEY ("instance_id") REFERENCES "instances" ("id"), CONSTRAINT "nodes_ibfk_2" FOREIGN KEY ("configuration_id") REFERENCES "configurations" ("id"), CONSTRAINT "nodes_ibfk_1" FOREIGN KEY ("datastore_version_id") REFERENCES "datastore_versions" ("id") ) ENGINE=InnoDB DEFAULT CHARSET=utf8;
aka the same table as instances, except:
- addition of: "instance_id" varchar(36) NOT NULL
- addition of: KEY "instance_id" ("instance_id")
- addition of: CONSTRAINT "instances_ibfk_3" FOREIGN KEY ("instance_id") REFERENCES "instances" ("id"),
- TODO: changing of 'DEFAULT NULL' to 'NOT NULL' whenever possible (ex: things like CREATED should never be NULL)
- TODO: addition of removal of indexes as deemed necessary
Alter Instances Table
Add slave_of Column to Instances Table (+ Constraint + Index):
ALTER TABLE nodes ADD COLUMN slave_of VARCHAR(36) DEFAULT NULL; KEY "slave_of" ("slave_of"), CONSTRAINT "instances_ibfk_3" FOREIGN KEY ("slave_of") REFERENCES "instances" ("id")
Alter Other Tables
Add node_id column to the following tables:
- agent_heartbeats
- backups
- conductor_lastseen
- root_enabled_history
- security_group_instance_associations
- service_statuses
- usage_events
ALTER TABLE <table> ADD COLUMN node_id VARCHAR(36) DEFAULT NULL; KEY "node_id" ("node_id"), CONSTRAINT "<table>_ibfk_<num>" FOREIGN KEY ("node_id") REFERENCES "nodes" ("id")
TaskManager
- add node_id to /etc/guest_info (if it's a node in a cluster). guest_id remains as-is.
- for-loop create each node.
- poll until all nodes are active.
- for each node: use trove/nova to get ip/hostname
- for couchbase:
- send ip/hostname list via rpc cast to guest
- for cassandra:
- send seed ip list via rpc cast to guest seed nodes, one by one (polling on REBOOT => ACTIVE), then to rest of nodes.
- for mongodb:
- send ip/hostname list via rpc cast to guest that is the db.isMaster()
Guest
- update heartbeat payload ( heartbeat(guest_id, payload, sent) ) from {"service_status": "<status>"} to {"service_status": "<status>", "node_id": "<node-id>"}
- add method to each datastore guest manager for handling ip/hostname list
Conductor
- update heartbeat logic to update the nodes table (node status)
Capabilities
A capability might be supported for a datastore-version for standalone instances, but not for clusters.
Therefore, the capability tables must be amended to include a cluster-enabled flag.
ALTER TABLE capabilities ADD COLUMN enabled_cluster TINYINT(1) DEFAULT NULL;
ALTER TABLE capability_overrides ADD COLUMN enabled_cluster TINYINT(1) DEFAULT NULL;