Trove/Replication-And-Clustering-With-Nodes-5

Example: Cassandra
To illustrate the approach, Cassandra is used in the examples below. The eccentricities of each Datastore will be explained in their own sections.

Create Cluster
Request:

POST /clusters { "cluster": { "name": "products", "datastore": { "type": "cassandra", "version": "2.0.6" },   "instances": [ {       "flavorRef": "7", "volume": { "size": 1 },       "configuration_id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6", "region": "west" },     {        "flavorRef": "7", "volume": { "size": 1 },       "configuration_id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6", "region": "east" },     {        "flavorRef": "7", "volume": { "size": 1 },       "configuration_id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6", "region": "eu" }   ]  } }

Response:

Add Me Future (TBD): Defaults POST /clusters { "cluster": { "name": "products", "datastore": { "type": "cassandra", "version": "2.0.6" },   "defaults": { "flavorRef": "7" ...   }

Show Cluster
Request:

GET /clusters/dfbbd9ca-b5e1-4028-adb7-f78643e17998

Response:

{ "cluster": { "status": "ACTIVE", "id": "dfbbd9ca-b5e1-4028-adb7-f78643e17998", "name": "products", "created": "2014-04-25T20:19:23", "updated": "2014-04-25T20:19:23", "links": [{...}], "datastore": { "type": "cassandra", "version": "2.0.6" },   "instances": [ {       "id": "416b0b16-ba55-4302-bbd3-ff566032e1c1", "flavor": { "id": "7", "links": [{...}] },       "configuration": { "id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6", "links": [{...}] },       "volume": { "size": 1 },       "region": "west" },     {        "id": "7f52e4f9-3fa6-4238-ac08-1ce15197329a" "flavor": { "id": "7", "links": [{...}] },       "configuration": { "id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6", "links": [{...}] },       "volume": { "size": 1 },       "region": "east" },     {        "id": "ff9d680c-fde3-49c6-a84e-76173b6df39d" "flavor": { "id": "7", "links": [{...}] },       "configuration": { "id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6", "links": [{...}] },       "volume": { "size": 1 },       "region": "eu" }   ]  } }

Show Instance
Request:

GET /clusters/dfbbd9ca-b5e1-4028-adb7-f78643e17998/instances/416b0b16-ba55-4302-bbd3-ff566032e1c1

Response:

{ "instance": { "status": "ACTIVE", "id": "416b0b16-ba55-4302-bbd3-ff566032e1c1", "name": "products-1", "created": "2014-04-25T20:19:23", "updated": "2014-04-25T20:19:23", "links": [{...}], "ip": ["10.0.0.1"], "configuration": { "id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6", "links": [{...}], },   "flavor": { "id": "7", "links": [{...}], },   "volume": { "size": 1, "used": 0.17 } } }

Add Instance(s)
Request:

POST /clusters/dfbbd9ca-b5e1-4028-adb7-f78643e17998/instances { "instances": [ {     "flavorRef": "7", "volume": { "size": 1 },     "configuration_id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6", "region": "west" },   {      "flavorRef": "7", "volume": { "size": 1 },     "configuration_id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6", "region": "west" } ] } Response: HTTP 202 (Empty Body)

Replace Instance
Request:

POST /clusters/dfbbd9ca-b5e1-4028-adb7-f78643e17998/action

{ "replace_instance": { "id": "7f52e4f9-3fa6-4238-ac08-1ce15197329a" } } Response: HTTP 202 (Empty Body) Notes:
 * http://www.datastax.com/documentation/cassandra/2.0/cassandra/operations/ops_replace_live_node.html
 * http://www.datastax.com/documentation/cassandra/2.0/cassandra/operations/ops_replace_seed_node.html
 * http://www.datastax.com/documentation/cassandra/2.0/cassandra/operations/ops_replace_node_t.html

Remove Instance
Request:

DELETE /clusters/dfbbd9ca-b5e1-4028-adb7-f78643e17998/instances/7f52e4f9-3fa6-4238-ac08-1ce15197329a Response: HTTP 202 (Empty Body) Notes:
 * http://www.datastax.com/documentation/cassandra/2.0/cassandra/operations/ops_remove_node_t.html

Create Cluster
Request:

POST /clusters { "cluster": { "name": "products", "datastore": { "type": "mongodb", "version": "2.4.10" },   "instances": [ {       "flavorRef": "7", "volume": { "size": 1 },       "configuration_id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6", "region": "west" },     {        "flavorRef": "7", "volume": { "size": 1 },       "configuration_id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6", "region": "west" },     {        "flavorRef": "7", "volume": { "size": 1 },       "configuration_id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6", "region": "west" },     {        "flavorRef": "7", "volume": { "size": 1 },       "configuration_id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6", "region": "east" },     {        "flavorRef": "7", "volume": { "size": 1 },       "configuration_id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6", "region": "east" }   ]  } }

Response:

Add Me

Show Cluster
Request:

GET /clusters/dfbbd9ca-b5e1-4028-adb7-f78643e17998

Response:

{ "cluster": { "status": "ACTIVE", "id": "dfbbd9ca-b5e1-4028-adb7-f78643e17998", "name": "products", "created": "2014-04-25T20:19:23", "updated": "2014-04-25T20:19:23", "links": [{...}], "datastore": { "type": "mongodb", "version": "2.4.10" },   "instances": [ {       "id": "416b0b16-ba55-4302-bbd3-ff566032e1c1", "flavor": { "id": "7", "links": [{...}] },       "configuration": { "id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6", "links": [{...}] },       "volume": { "size": 1 },       "region": "west" },     {        "id": "965ef811-7c1d-47fc-89f2-a89dfdd23ef2" "flavor": { "id": "7", "links": [{...}] },       "configuration": { "id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6", "links": [{...}] },       "volume": { "size": 1 },       "region": "west" },     {        "id": "3642f41c-e8ad-4164-a089-3891bf7f2d2b" "flavor": { "id": "7", "links": [{...}] },       "configuration": { "id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6", "links": [{...}] },       "volume": { "size": 1 },       "region": "west" },     {        "id": "7f52e4f9-3fa6-4238-ac08-1ce15197329a" "flavor": { "id": "7", "links": [{...}] },       "configuration": { "id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6", "links": [{...}] },       "volume": { "size": 1 },       "region": "east" },     {        "id": "ff9d680c-fde3-49c6-a84e-76173b6df39d" "flavor": { "id": "7", "links": [{...}] },       "configuration": { "id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6", "links": [{...}] },       "volume": { "size": 1 },       "region": "east" }   ]  } }

Show Instance
Request:

GET /clusters/dfbbd9ca-b5e1-4028-adb7-f78643e17998/instances/416b0b16-ba55-4302-bbd3-ff566032e1c1

Response:

{ "instance": { "status": "ACTIVE", "id": "416b0b16-ba55-4302-bbd3-ff566032e1c1", "name": "products-1", "created": "2014-04-25T20:19:23", "updated": "2014-04-25T20:19:23", "links": [{...}], "ip": ["10.0.0.1"], "configuration": { "id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6", "links": [{...}], },   "flavor": { "id": "7", "links": [{...}], },   "volume": { "size": 1, "used": 0.17 } } }

Create Arbiter(s)
Request:

POST /clusters/dfbbd9ca-b5e1-4028-adb7-f78643e17998/instances

{ "instances": [ {     "flavorRef": "7", "volume": { "size": 1 },     "configuration_id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6", "region": "eu", "type": "arbiter" },   {      "flavorRef": "7", "volume": { "size": 1 },     "configuration_id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6", "region": "eu", "type": "arbiter" } ] }

Response:

HTTP 202 (Empty Body)

Show Cluster (After Arbiters)
Request:

GET /clusters/dfbbd9ca-b5e1-4028-adb7-f78643e17998

Response:

{ "cluster": { "status": "ACTIVE", "id": "dfbbd9ca-b5e1-4028-adb7-f78643e17998", "name": "products", "created": "2014-04-25T20:19:23", "updated": "2014-04-25T20:19:23", "links": [{...}], "datastore": { "type": "mongodb", "version": "2.4.10" },   "instances": [ {       "id": "416b0b16-ba55-4302-bbd3-ff566032e1c1", "flavor": { "id": "7", "links": [{...}] },       "configuration": { "id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6", "links": [{...}] },       "volume": { "size": 1 },       "region": "west" },     {        "id": "965ef811-7c1d-47fc-89f2-a89dfdd23ef2" "flavor": { "id": "7", "links": [{...}] },       "configuration": { "id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6", "links": [{...}] },       "volume": { "size": 1 },       "region": "west" },     {        "id": "3642f41c-e8ad-4164-a089-3891bf7f2d2b" "flavor": { "id": "7", "links": [{...}] },       "configuration": { "id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6", "links": [{...}] },       "volume": { "size": 1 },       "region": "west" },     {        "id": "7f52e4f9-3fa6-4238-ac08-1ce15197329a" "flavor": { "id": "7", "links": [{...}] },       "configuration": { "id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6", "links": [{...}] },       "volume": { "size": 1 },       "region": "east" },     {        "id": "ff9d680c-fde3-49c6-a84e-76173b6df39d" "flavor": { "id": "7", "links": [{...}] },       "configuration": { "id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6", "links": [{...}] },       "volume": { "size": 1 },       "region": "east" },     {        "id": "77032c55-4496-4e35-8c0d-6cd1c18e1a9c" "flavor": { "id": "7", "links": [{...}] },       "configuration": { "id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6", "links": [{...}] },       "volume": { "size": 1 },       "region": "eu", "type": "arbiter" },     {        "id": "1fd054ed-221f-4c99-8d17-570bcff4c1d2" "flavor": { "id": "7", "links": [{...}] },       "configuration": { "id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6", "links": [{...}] },       "volume": { "size": 1 },       "region": "eu", "type": "arbiter" }   ]  } }

Create Master
Request:

POST /instances { "instance": { "name": "products", "datastore": { "type": "mysql", "version": "5.5" },   "configuration_id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6", "flavorRef": "7", "volume": { "size": 1 } } }

Response:

{ "instance": { "status": "BUILD", "id": "dfbbd9ca-b5e1-4028-adb7-f78643e17998", "name": "products", "created": "2014-04-25T20:19:23", "updated": "2014-04-25T20:19:23", "links": [{...}], "datastore": { "type": "mysql", "version": "5.5" },   "configuration": { "id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6", "links": [{...}], },   "flavor": { "id": "7", "links": [{...}], },   "volume": { "size": 1 } } }

Create Slave
Request:

POST /instances { "instance": { "name": "products-slave", "datastore": { "type": "mysql", "version": "5.5" },   "configuration_id": "fc318e00-3a6f-4f93-af99-146b44912188", "flavorRef": "7", "volume": { "size": 1 },   "slave": { "of": "dfbbd9ca-b5e1-4028-adb7-f78643e17998", "read_only": true } } }

Response:

{ "instance": { "status": "BUILD", "id": "061aaf4c-3a57-411e-9df9-2d0f813db859", "name": "products", "created": "2014-04-25T20:19:23", "updated": "2014-04-25T20:19:23", "links": [{...}], "datastore": { "type": "mysql", "version": "5.5" },   "configuration": { "id": "fc318e00-3a6f-4f93-af99-146b44912188", "links": [{...}], },   "flavor": { "id": "7", "links": [{...}], },   "volume": { "size": 1 },   "slave": { "of": "dfbbd9ca-b5e1-4028-adb7-f78643e17998", "read_only": true } } }

Show Master
Request:

GET /instances/dfbbd9ca-b5e1-4028-adb7-f78643e17998

Response:

{ "instance": { "status": "ACTIVE", "id": "dfbbd9ca-b5e1-4028-adb7-f78643e17998", "name": "products", "created": "2014-04-25T20:19:23", "updated": "2014-04-25T20:19:23", "links": [{...}], "datastore": { "type": "mysql", "version": "5.5" },   "configuration": { "id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6", "links": [{...}], },   "flavor": { "id": "7", "links": [{...}], },   "volume": { "size": 1 },   "slave": { "list": [ {"id": "061aaf4c-3a57-411e-9df9-2d0f813db859"} ]   }  } }

Show Slave
Request:

GET /instances/061aaf4c-3a57-411e-9df9-2d0f813db859

Response:

{ "instance": { "status": "ACTIVE", "id": "061aaf4c-3a57-411e-9df9-2d0f813db859", "name": "products", "created": "2014-04-25T20:19:23", "updated": "2014-04-25T20:19:23", "links": [{...}], "datastore": { "type": "mysql", "version": "5.5" },   "configuration": { "id": "fc318e00-3a6f-4f93-af99-146b44912188", "links": [{...}], },   "flavor": { "id": "7", "links": [{...}], },   "volume": { "size": 1 },   "slave": { "of": "dfbbd9ca-b5e1-4028-adb7-f78643e17998", "read_only": true } } }

Detach Slave
Request:

POST /instances/061aaf4c-3a57-411e-9df9-2d0f813db859/action

{ "detach": {} }

Response:

HTTP 202 (Empty Body)

Delete Master
Request:

DELETE /instances/dfbbd9ca-b5e1-4028-adb7-f78643e17998 Response: HTTP 202 (Empty Body) Notes:
 * How to handle situation in which a slave is attached to a master, and the user attempts to delete the master?

Delete Slave
Request:

DELETE /instances/061aaf4c-3a57-411e-9df9-2d0f813db859 Response: HTTP 202 (Empty Body)

clusters Table
Create a new 'clusters' Table: CREATE TABLE "clusters" ( "id" varchar(36) NOT NULL,  "created" datetime DEFAULT NULL,  "updated" datetime DEFAULT NULL,  "name" varchar(255) DEFAULT NULL,  "task_id" int(11) DEFAULT NULL,  "task_description" varchar(32) DEFAULT NULL,  "task_start_time" datetime DEFAULT NULL,  "tenant_id" varchar(36) DEFAULT NULL,  "server_status" varchar(64) DEFAULT NULL,  "datastore_version_id" varchar(36) NOT NULL,  "deleted" tinyint(1) DEFAULT NULL,  "deleted_at" datetime DEFAULT NULL,  PRIMARY KEY ("id"),  KEY "datastore_version_id" ("datastore_version_id"),  KEY "clusters_tenant_id" ("tenant_id"),  KEY "clusters_deleted" ("deleted"),  CONSTRAINT "clusters_ibfk_1" FOREIGN KEY ("datastore_version_id") REFERENCES "datastore_versions" ("id") ) ENGINE=InnoDB DEFAULT CHARSET=utf8; Notes:
 * todo: task_* related columns were simply copy/pasted from instances; could diverge
 * todo: change 'default null' to 'not null' whenever possible

Alter Instances Table
Add slave_of Column to Instances Table (+ Constraint + Index): ALTER TABLE instances ADD COLUMN slave_of VARCHAR(36) DEFAULT NULL; KEY "slave_of" ("slave_of"), CONSTRAINT "instances_ibfk_3" FOREIGN KEY ("slave_of") REFERENCES "instances" ("id") Add cluster_id Column to Instances Table (+ Constraint + Index): ALTER TABLE instances ADD COLUMN cluster_id VARCHAR(36) DEFAULT NULL; KEY "cluster_id" ("cluster_id"), CONSTRAINT "instances_ibfk_4" FOREIGN KEY ("cluster_id") REFERENCES "clusters" ("id")

TaskManager

 * add cluster_id to /etc/guest_info (if it's an instance in a cluster). guest_id remains as-is.
 * for-loop create each instance.
 * poll until all instances are active.
 * for each instance: use trove/nova to get ip/hostname
 * for couchbase:
 * send ip/hostname list via rpc cast to guest
 * for cassandra:
 * send seed ip list via rpc cast to guest seed instances, one by one (polling on REBOOT => ACTIVE), then to rest of instances.
 * for mongodb:
 * send ip/hostname list via rpc cast to guest that is the db.isMaster

Guest

 * update heartbeat payload ( heartbeat(guest_id, payload, sent) ) from {"service_status": " "} to {"service_status": " ", "cluster_id": ""}
 * add method to each datastore guest manager for handling ip/hostname list

Conductor

 * update heartbeat logic to update the clusters table (cluster status) if an instance's status changes the overall state of the cluster

Capabilities
A capability might be supported for a datastore-version for standalone instances, but not for clusters. Therefore, the capability tables must be amended to include a cluster-enabled flag.

ALTER TABLE capabilities ADD COLUMN enabled_cluster TINYINT(1) DEFAULT NULL; ALTER TABLE capability_overrides ADD COLUMN enabled_cluster TINYINT(1) DEFAULT NULL; The following capabilities should have enabled_cluster set to false for the first iteration of clusters:
 * backup-create + list-instance
 * configuration-attach + detach + instances
 * resize-
 * database-
 * root-
 * secgroup-
 * user-

Introduce read_only and hidden Parameters
Need to introduce two additional attributes for configuration group parameters: read_only and hidden.
 * read_only fields include cluster_name, num_tokens, seed_provider, seeds, endpoint_snitch (cassandra) + replSet (mongodb) + server_id, log_bin (mysql).
 * depending on the provider, some of the read_only fields should also be hidden from the user on a configuration-show.
 * once read_only + hidden are available, a parallel effort should move configuration-default to configurations-show if a configuration-group is attached.
 * amcrn (talk) 20:43, 8 May 2014 (UTC): update: mysql master/slave will not be required to do this because the overrides.cnf functionality handles this nicely. to be determined as to how clustering will handle this. it could be this, or it could be a copy of the original conf, or a mixture thereof.

Auto-Create and Attach

 * cassandra & mongodb need to have configuration-groups automatically created and attached to each instance (for cluster_name, replset, etc.) during provisioning.
 * unique configuration-group per instance.
 * auto-created+attached configuration-groups need to not be detachable from the instance.
 * dependency: configuration-group support for mongodb + cassandra
 * amcrn (talk) 20:43, 8 May 2014 (UTC): update: mysql master/slave will not be required to do this because the overrides.cnf functionality handles this nicely. to be determined as to how clustering will handle this. it could be this, or it could be a copy of the original conf, or a mixture thereof.

python-troveclient
Discussed options are documented at https://gist.github.com/amcrn/c4ae2210e9d9864c21fd. An option was chosen on 05/21 and is described below.

Create-Cluster Help
$ trove help create-cluster usage: trove create-cluster  [--instance ]

Creates a new cluster.

Positional arguments: Name of the cluster A datastore name or UUID   A datastore version name or UUID

Optional arguments:

--instance  Create an instance for the cluster. Specify multiple times to create multiple instances.

Create Cluster Request
$ trove create-cluster products mongodb "2.4.10" --instance flavor=12,disk=100,config-group=e0a6a893-1e41-4a05-8252-3c2cb934dd79 \ --instance flavor=12,disk=100,config-group=e0a6a893-1e41-4a05-8252-3c2cb934dd79 \ --instance flavor=12,disk=100,config-group=e0a6a893-1e41-4a05-8252-3c2cb934dd79 \ --instance flavor=12,disk=100,config-group=e0a6a893-1e41-4a05-8252-3c2cb934dd79 \ --instance flavor=12,disk=100,config-group=e0a6a893-1e41-4a05-8252-3c2cb934dd79 \ --instance flavor=7,disk=1,config-group=b9c8a3f8-7ace-4aea-9908-7b555586d7b6,type=arbiter \ --instance flavor=7,disk=1,config-group=b9c8a3f8-7ace-4aea-9908-7b555586d7b6,type=arbiter

Create Cluster Response
TBD