Jump to: navigation, search

Difference between revisions of "Trove/Replication-And-Clustering-With-Nodes-5"

m
(Change "configuration" to "configuration_id" for requests.)
 
(3 intermediate revisions by one other user not shown)
Line 17: Line 17:
 
       "version": "2.0.6"
 
       "version": "2.0.6"
 
     },
 
     },
     "nodes": [
+
     "instances": [
 
       {
 
       {
 
         "flavorRef": "7",
 
         "flavorRef": "7",
Line 23: Line 23:
 
           "size": 1
 
           "size": 1
 
         },
 
         },
         "configuration": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
+
         "configuration_id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
 
         "region": "west"
 
         "region": "west"
 
       },
 
       },
Line 31: Line 31:
 
           "size": 1
 
           "size": 1
 
         },
 
         },
         "configuration": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
+
         "configuration_id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
 
         "region": "east"
 
         "region": "east"
 
       },
 
       },
Line 39: Line 39:
 
           "size": 1
 
           "size": 1
 
         },
 
         },
         "configuration": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
+
         "configuration_id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
 
         "region": "eu"
 
         "region": "eu"
 
       }
 
       }
Line 93: Line 93:
 
       "version": "2.0.6"
 
       "version": "2.0.6"
 
     },
 
     },
     "nodes": [
+
     "instances": [
 
       {
 
       {
 
         "id": "416b0b16-ba55-4302-bbd3-ff566032e1c1",
 
         "id": "416b0b16-ba55-4302-bbd3-ff566032e1c1",
Line 144: Line 144:
 
</pre>
 
</pre>
 
<br>
 
<br>
==== Show Node ====
+
==== Show Instance ====
 
<br>
 
<br>
 
Request:
 
Request:
  
 
<pre>
 
<pre>
GET /clusters/dfbbd9ca-b5e1-4028-adb7-f78643e17998/nodes/416b0b16-ba55-4302-bbd3-ff566032e1c1
+
GET /clusters/dfbbd9ca-b5e1-4028-adb7-f78643e17998/instances/416b0b16-ba55-4302-bbd3-ff566032e1c1
 
</pre>
 
</pre>
  
Line 156: Line 156:
 
<pre>
 
<pre>
 
{
 
{
   "node": {
+
   "instance": {
 
     "status": "ACTIVE",
 
     "status": "ACTIVE",
 
     "id": "416b0b16-ba55-4302-bbd3-ff566032e1c1",
 
     "id": "416b0b16-ba55-4302-bbd3-ff566032e1c1",
Line 180: Line 180:
 
</pre>
 
</pre>
 
<br>
 
<br>
==== Add Node(s) ====
+
==== Add Instance(s) ====
 
<br>
 
<br>
 
Request:
 
Request:
  
 
<pre>
 
<pre>
POST /clusters/dfbbd9ca-b5e1-4028-adb7-f78643e17998/nodes
+
POST /clusters/dfbbd9ca-b5e1-4028-adb7-f78643e17998/instances
 
{
 
{
   "nodes": {
+
   "instances": [
 
     {
 
     {
 
       "flavorRef": "7",
 
       "flavorRef": "7",
Line 193: Line 193:
 
         "size": 1
 
         "size": 1
 
       },
 
       },
       "configuration": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
+
       "configuration_id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
 
       "region": "west"
 
       "region": "west"
 
     },
 
     },
Line 201: Line 201:
 
         "size": 1
 
         "size": 1
 
       },
 
       },
       "configuration": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
+
       "configuration_id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
 
       "region": "west"
 
       "region": "west"
 
     }
 
     }
   }
+
   ]
 
}
 
}
 
</pre>
 
</pre>
Line 213: Line 213:
 
</pre>
 
</pre>
 
<br>
 
<br>
==== Replace Node ====
+
==== Replace Instance ====
 
<br>
 
<br>
 
Request:
 
Request:
Line 221: Line 221:
  
 
{
 
{
   "replace_node": {
+
   "replace_instance": {
 
     "id": "7f52e4f9-3fa6-4238-ac08-1ce15197329a"
 
     "id": "7f52e4f9-3fa6-4238-ac08-1ce15197329a"
 
   }
 
   }
Line 238: Line 238:
 
<br>
 
<br>
  
==== Remove Node ====
+
==== Remove Instance ====
 
<br>
 
<br>
 
Request:
 
Request:
  
 
<pre>
 
<pre>
DELETE /clusters/dfbbd9ca-b5e1-4028-adb7-f78643e17998/nodes/7f52e4f9-3fa6-4238-ac08-1ce15197329a
+
DELETE /clusters/dfbbd9ca-b5e1-4028-adb7-f78643e17998/instances/7f52e4f9-3fa6-4238-ac08-1ce15197329a
 
</pre>
 
</pre>
 
<br>
 
<br>
Line 270: Line 270:
 
       "version": "2.4.10"
 
       "version": "2.4.10"
 
     },
 
     },
     "nodes": [
+
     "instances": [
 
       {
 
       {
 
         "flavorRef": "7",
 
         "flavorRef": "7",
Line 276: Line 276:
 
           "size": 1
 
           "size": 1
 
         },
 
         },
         "configuration": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
+
         "configuration_id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
 
         "region": "west"
 
         "region": "west"
 
       },
 
       },
Line 284: Line 284:
 
           "size": 1
 
           "size": 1
 
         },
 
         },
         "configuration": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
+
         "configuration_id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
 
         "region": "west"
 
         "region": "west"
 
       },
 
       },
Line 292: Line 292:
 
           "size": 1
 
           "size": 1
 
         },
 
         },
         "configuration": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
+
         "configuration_id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
 
         "region": "west"
 
         "region": "west"
 
       },
 
       },
Line 300: Line 300:
 
           "size": 1
 
           "size": 1
 
         },
 
         },
         "configuration": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
+
         "configuration_id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
 
         "region": "east"
 
         "region": "east"
 
       },
 
       },
Line 308: Line 308:
 
           "size": 1
 
           "size": 1
 
         },
 
         },
         "configuration": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
+
         "configuration_id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
 
         "region": "east"
 
         "region": "east"
 
       }
 
       }
Line 346: Line 346:
 
       "version": "2.4.10"
 
       "version": "2.4.10"
 
     },
 
     },
     "nodes": [
+
     "instances": [
 
       {
 
       {
 
         "id": "416b0b16-ba55-4302-bbd3-ff566032e1c1",
 
         "id": "416b0b16-ba55-4302-bbd3-ff566032e1c1",
Line 428: Line 428:
 
<br>
 
<br>
  
==== Show Node ====
+
==== Show Instance ====
 
<br>
 
<br>
 
Request:
 
Request:
  
 
<pre>
 
<pre>
GET /clusters/dfbbd9ca-b5e1-4028-adb7-f78643e17998/nodes/416b0b16-ba55-4302-bbd3-ff566032e1c1
+
GET /clusters/dfbbd9ca-b5e1-4028-adb7-f78643e17998/instances/416b0b16-ba55-4302-bbd3-ff566032e1c1
 
</pre>
 
</pre>
  
Line 440: Line 440:
 
<pre>
 
<pre>
 
{
 
{
   "node": {
+
   "instance": {
 
     "status": "ACTIVE",
 
     "status": "ACTIVE",
 
     "id": "416b0b16-ba55-4302-bbd3-ff566032e1c1",
 
     "id": "416b0b16-ba55-4302-bbd3-ff566032e1c1",
Line 469: Line 469:
  
 
<pre>
 
<pre>
POST /clusters/dfbbd9ca-b5e1-4028-adb7-f78643e17998/nodes
+
POST /clusters/dfbbd9ca-b5e1-4028-adb7-f78643e17998/instances
  
 
{
 
{
   "nodes": {
+
   "instances": [
 
     {
 
     {
 
       "flavorRef": "7",
 
       "flavorRef": "7",
Line 478: Line 478:
 
         "size": 1
 
         "size": 1
 
       },
 
       },
       "configuration": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
+
       "configuration_id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
 
       "region": "eu",
 
       "region": "eu",
 
       "type": "arbiter"
 
       "type": "arbiter"
Line 487: Line 487:
 
         "size": 1
 
         "size": 1
 
       },
 
       },
       "configuration": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
+
       "configuration_id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
 
       "region": "eu",
 
       "region": "eu",
 
       "type": "arbiter"
 
       "type": "arbiter"
 
     }
 
     }
   }
+
   ]
 
}
 
}
 
</pre>
 
</pre>
Line 525: Line 525:
 
       "version": "2.4.10"
 
       "version": "2.4.10"
 
     },
 
     },
     "nodes": [
+
     "instances": [
 
       {
 
       {
 
         "id": "416b0b16-ba55-4302-bbd3-ff566032e1c1",
 
         "id": "416b0b16-ba55-4302-bbd3-ff566032e1c1",
Line 654: Line 654:
 
       "version": "5.5"
 
       "version": "5.5"
 
     },
 
     },
     "configuration": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
+
     "configuration_id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
 
     "flavorRef": "7",
 
     "flavorRef": "7",
 
     "volume": {
 
     "volume": {
Line 707: Line 707:
 
       "version": "5.5"
 
       "version": "5.5"
 
     },
 
     },
     "configuration": "fc318e00-3a6f-4f93-af99-146b44912188",
+
     "configuration_id": "fc318e00-3a6f-4f93-af99-146b44912188",
 
     "flavorRef": "7",
 
     "flavorRef": "7",
 
     "volume": {
 
     "volume": {
Line 889: Line 889:
  
 
== Data Model Changes ==
 
== Data Model Changes ==
=== Nodes Table ===
+
=== clusters Table ===
Create a new 'nodes' Table:
+
Create a new 'clusters' Table:
 
<pre>
 
<pre>
CREATE TABLE "nodes" (
+
CREATE TABLE "clusters" (
 
   "id" varchar(36) NOT NULL,
 
   "id" varchar(36) NOT NULL,
  "instance_id" varchar(36) NOT NULL,
 
 
   "created" datetime DEFAULT NULL,
 
   "created" datetime DEFAULT NULL,
 
   "updated" datetime DEFAULT NULL,
 
   "updated" datetime DEFAULT NULL,
 
   "name" varchar(255) DEFAULT NULL,
 
   "name" varchar(255) DEFAULT NULL,
  "hostname" varchar(255) DEFAULT NULL,
 
  "compute_instance_id" varchar(36) DEFAULT NULL,
 
 
   "task_id" int(11) DEFAULT NULL,
 
   "task_id" int(11) DEFAULT NULL,
 
   "task_description" varchar(32) DEFAULT NULL,
 
   "task_description" varchar(32) DEFAULT NULL,
 
   "task_start_time" datetime DEFAULT NULL,
 
   "task_start_time" datetime DEFAULT NULL,
  "volume_id" varchar(36) DEFAULT NULL,
 
  "flavor_id" int(11) DEFAULT NULL,
 
  "volume_size" int(11) DEFAULT NULL,
 
 
   "tenant_id" varchar(36) DEFAULT NULL,
 
   "tenant_id" varchar(36) DEFAULT NULL,
 
   "server_status" varchar(64) DEFAULT NULL,
 
   "server_status" varchar(64) DEFAULT NULL,
 +
  "datastore_version_id" varchar(36) NOT NULL,
 
   "deleted" tinyint(1) DEFAULT NULL,
 
   "deleted" tinyint(1) DEFAULT NULL,
 
   "deleted_at" datetime DEFAULT NULL,
 
   "deleted_at" datetime DEFAULT NULL,
  "datastore_version_id" varchar(36) NOT NULL,
 
  "configuration_id" varchar(36) DEFAULT NULL,
 
 
   PRIMARY KEY ("id"),
 
   PRIMARY KEY ("id"),
  KEY "instance_id" ("instance_id"),
 
 
   KEY "datastore_version_id" ("datastore_version_id"),
 
   KEY "datastore_version_id" ("datastore_version_id"),
   KEY "configuration_id" ("configuration_id"),
+
   KEY "clusters_tenant_id" ("tenant_id"),
  KEY "instances_tenant_id" ("tenant_id"),
+
   KEY "clusters_deleted" ("deleted"),
   KEY "instances_deleted" ("deleted"),
+
   CONSTRAINT "clusters_ibfk_1" FOREIGN KEY ("datastore_version_id") REFERENCES "datastore_versions" ("id")
   CONSTRAINT "nodes_ibfk_3" FOREIGN KEY ("instance_id") REFERENCES "instances" ("id"),
 
  CONSTRAINT "nodes_ibfk_2" FOREIGN KEY ("configuration_id") REFERENCES "configurations" ("id"),
 
  CONSTRAINT "nodes_ibfk_1" FOREIGN KEY ("datastore_version_id") REFERENCES "datastore_versions" ("id")
 
 
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
 
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
 
</pre>
 
</pre>
 
<br>
 
<br>
aka the same table as instances, except:
+
Notes:
* addition of: "instance_id" varchar(36) NOT NULL
+
* todo: task_* related columns were simply copy/pasted from instances; could diverge
* addition of: KEY "instance_id" ("instance_id")
+
* todo: change 'default null' to 'not null' whenever possible
* addition of: CONSTRAINT "instances_ibfk_3" FOREIGN KEY ("instance_id") REFERENCES "instances" ("id"),
 
* TODO: changing of 'DEFAULT NULL' to 'NOT NULL' whenever possible (ex: things like CREATED should never be NULL)
 
* TODO: addition of removal of indexes as deemed necessary
 
 
<br>
 
<br>
 
=== Alter Instances Table ===
 
=== Alter Instances Table ===
Line 938: Line 924:
 
CONSTRAINT "instances_ibfk_3" FOREIGN KEY ("slave_of") REFERENCES "instances" ("id")
 
CONSTRAINT "instances_ibfk_3" FOREIGN KEY ("slave_of") REFERENCES "instances" ("id")
 
</pre>
 
</pre>
<br>
+
Add cluster_id Column to Instances Table (+ Constraint + Index):
 
 
=== Alter Other Tables ===
 
Add node_id column to the following tables:
 
* agent_heartbeats
 
* backups
 
* conductor_lastseen
 
* root_enabled_history
 
* security_group_instance_associations
 
* service_statuses
 
* usage_events
 
<br>
 
 
<pre>
 
<pre>
ALTER TABLE <table> ADD COLUMN node_id VARCHAR(36) DEFAULT NULL;
+
ALTER TABLE instances ADD COLUMN cluster_id VARCHAR(36) DEFAULT NULL;
KEY "node_id" ("node_id"),
+
KEY "cluster_id" ("cluster_id"),
CONSTRAINT "<table>_ibfk_<num>" FOREIGN KEY ("node_id") REFERENCES "nodes" ("id")
+
CONSTRAINT "instances_ibfk_4" FOREIGN KEY ("cluster_id") REFERENCES "clusters" ("id")
 
</pre>
 
</pre>
 
<br>
 
<br>
 +
 
== TaskManager ==
 
== TaskManager ==
  
* add node_id to /etc/guest_info (if it's a node in a cluster). guest_id remains as-is.
+
* add cluster_id to /etc/guest_info (if it's an instance in a cluster). guest_id remains as-is.
* for-loop create each node.
+
* for-loop create each instance.
* poll until all nodes are active.
+
* poll until all instances are active.
* for each node: use trove/nova to get ip/hostname
+
* for each instance: use trove/nova to get ip/hostname
 
* for couchbase:
 
* for couchbase:
 
** send ip/hostname list via rpc cast to guest
 
** send ip/hostname list via rpc cast to guest
 
* for cassandra:
 
* for cassandra:
** send seed ip list via rpc cast to guest seed nodes, one by one (polling on REBOOT => ACTIVE), then to rest of nodes.
+
** send seed ip list via rpc cast to guest seed instances, one by one (polling on REBOOT => ACTIVE), then to rest of instances.
 
* for mongodb:
 
* for mongodb:
 
** send ip/hostname list via rpc cast to guest that is the db.isMaster()
 
** send ip/hostname list via rpc cast to guest that is the db.isMaster()
Line 972: Line 948:
 
== Guest ==
 
== Guest ==
  
* update heartbeat payload ( heartbeat(guest_id, payload, sent) ) from {"service_status": "<status>"} to {"service_status": "<status>", "node_id": "<node-id>"}
+
* update heartbeat payload ( heartbeat(guest_id, payload, sent) ) from {"service_status": "<status>"} to {"service_status": "<status>", "cluster_id": "<cluster-id>"}
 
* add method to each datastore guest manager for handling ip/hostname list
 
* add method to each datastore guest manager for handling ip/hostname list
 
<br>
 
<br>
Line 978: Line 954:
 
== Conductor ==
 
== Conductor ==
  
* update heartbeat logic to update the nodes table (node status)
+
* update heartbeat logic to update the clusters table (cluster status) if an instance's status changes the overall state of the cluster
 
<br>
 
<br>
  
Line 1,012: Line 988:
  
 
=== Auto-Create and Attach ===
 
=== Auto-Create and Attach ===
* cassandra & mongodb need to have configuration-groups automatically created and attached to each node (for cluster_name, replset, etc.) during provisioning.
+
* cassandra & mongodb need to have configuration-groups automatically created and attached to each instance (for cluster_name, replset, etc.) during provisioning.
* unique configuration-group per node.
+
* unique configuration-group per instance.
 
* auto-created+attached configuration-groups need to not be detachable from the instance.
 
* auto-created+attached configuration-groups need to not be detachable from the instance.
 
* dependency: configuration-group support for mongodb + cassandra
 
* dependency: configuration-group support for mongodb + cassandra
 
* [[User:Amcrn|amcrn]] ([[User talk:Amcrn|talk]]) 20:43, 8 May 2014 (UTC): update: mysql master/slave will not be required to do this because the overrides.cnf functionality handles this nicely. to be determined as to how clustering will handle this. it could be this, or it could be a copy of the original conf, or a mixture thereof.
 
* [[User:Amcrn|amcrn]] ([[User talk:Amcrn|talk]]) 20:43, 8 May 2014 (UTC): update: mysql master/slave will not be required to do this because the overrides.cnf functionality handles this nicely. to be determined as to how clustering will handle this. it could be this, or it could be a copy of the original conf, or a mixture thereof.
 
<br>
 
<br>
 +
== python-troveclient ==
 +
<br>
 +
Discussed options are documented at https://gist.github.com/amcrn/c4ae2210e9d9864c21fd . An option was chosen on 05/21 and is described below.
 +
<br>
 +
=== Create-Cluster Help ===
  
== Feedback ==
+
<pre>
 +
$ trove help create-cluster
 +
</pre>
 +
<pre>
 +
usage: trove create-cluster <name> <datastore> <datastore_version>
 +
                            [--instance <instance>]
  
==== Create Slave ====
+
Creates a new cluster.
  
*glucas: Replication will require capturing a snapshot of the master's state and passing that to the slave. It would be preferable to create multiple slaves from a single snapshot and then clean up, rather than repeating the snapshot process multiple times. For that reason we propose a 'replicate' action that can create N slaves from an existing master in one call. I believe this approach works with the schema changes proposed here, i.e. adding the ''slave_of'' reference to the instance table.
+
Positional arguments:
* [[User:Amcrn|amcrn]] ([[User talk:Amcrn|talk]]) 17:49, 8 May 2014 (UTC): this is not usually cost-effective, depending on the provider. ex: if you have at least one slave in two or more regions, it makes sense to seed additional slaves with a backup from their home region. Here's a question: the backup of the master, will it stay put in Swift even after the replication has been setup? Re-worded, does the user have to delete the backup that's an artifact of setting up replication, or will Trove take care of it?
+
  <name>                Name of the cluster
*mwj: I thought the point of the topology design was to not introduce properties to the instance table that would not be relevant to all (i.e., non-replication) instances.  I take it this is no longer an issue?
+
  <datastore>          A datastore name or UUID
* [[User:Amcrn|amcrn]] ([[User talk:Amcrn|talk]]) 17:49, 8 May 2014 (UTC): don't understand the question because the instances table has not been modified at all, except the addition of the slave_of column, and no clustered nodes will be inserted in the instances table.
+
  <datastore_version>  A datastore version name or UUID
  
==== Capabilities ====
+
Optional arguments:
  
*glucas: We should use capabilities to indicate whether a datastore supports replication (and potentially read-only vs. read-write replication). In the first iteration, replication will not be supported for clusters.  mwj: Is it really necessary to have a capability for replication?  We are designing replication to use backup/restore functionality, so the backup/restore capability should be enough, no?
+
  --instance <flavor=flavor-id,disk=disk-size,config-group=group-uuid,type=type-name>
* [[User:Amcrn|amcrn]] ([[User talk:Amcrn|talk]]) 17:36, 8 May 2014 (UTC): there definitely needs to be a capability flag for clusters when it comes to features, as to whether master/slave needs it, that's an open question. does the logic for creating a backup wildly change when targeted against a slave vs. a standalone, and if so, are you guaranteeing you'll support this in your first iteration? you'll need to answer this for configuration-groups, users, databases, security-groups, root-enable/history, etc.
+
                        Create an instance for the cluster. Specify multiple
 +
                        times to create multiple instances.
 +
</pre>
  
==== Configuration Groups ====
+
=== Create Cluster Request ===
*dougshelley66: What if the end user wants to specify some configuration parameters for nodes or replicas? It appears (from the description above) that each node will have an auto generated group attached. Given an instance can only have one config group, this would allow the user to specify one?
 
* [[User:Amcrn|amcrn]] ([[User talk:Amcrn|talk]]) 17:33, 8 May 2014 (UTC): here's how it'd work: you want a master/slave setup, with 2 slaves. all three nodes would get their own configuration-group automatically created and attached (3 unique ids). if the user wishes to configure some parameters, they can do so. if the user provides a configuration-id on provisioning, then we'd add the read_only/hidden parameters to their configuration-group automatically.
 
  
==== Promote Slave ====
+
<pre>
*mwj: I would rather call this "detach slave" as I expect will may want to have "promote slave" be used in future fail-over designs.
+
$ trove create-cluster products mongodb "2.4.10"
* [[User:Amcrn|amcrn]] ([[User talk:Amcrn|talk]]) 17:30, 8 May 2014 (UTC): agreed, just went with the industry standard as a default. changed it to "detach" because i agree with you.
+
    --instance flavor=12,disk=100,config-group=e0a6a893-1e41-4a05-8252-3c2cb934dd79 \
 +
    --instance flavor=12,disk=100,config-group=e0a6a893-1e41-4a05-8252-3c2cb934dd79 \
 +
    --instance flavor=12,disk=100,config-group=e0a6a893-1e41-4a05-8252-3c2cb934dd79 \
 +
    --instance flavor=12,disk=100,config-group=e0a6a893-1e41-4a05-8252-3c2cb934dd79 \
 +
    --instance flavor=12,disk=100,config-group=e0a6a893-1e41-4a05-8252-3c2cb934dd79 \
 +
    --instance flavor=7,disk=1,config-group=b9c8a3f8-7ace-4aea-9908-7b555586d7b6,type=arbiter \
 +
    --instance flavor=7,disk=1,config-group=b9c8a3f8-7ace-4aea-9908-7b555586d7b6,type=arbiter
 +
</pre>
  
==== Delete Slave ====
+
=== Create Cluster Response ===
*mwj: Shouldn't "delete instance" just work, even for slaves?
 
* [[User:Amcrn|amcrn]] ([[User talk:Amcrn|talk]]) 17:29, 8 May 2014 (UTC): didn't have the example above, but yes it should. added an example.
 
  
==== Clusters ====
+
<pre>
*drewford: Should there be a "List Clusters" call?
+
TBD
*drewford: "size" and "num" attributes seem like workarounds when dealing with nodes and allocations.  Using "num"/"size" to tell the API how many objects you are adding in an array adds the dependency that the "num" value must match the length of the array.  What if they are not the same?  Unless there is a long-term use case for "size" and "num", they seem like they will become artifacts.
+
</pre>
*drewford: "allocations" adds one more level to the create object that seems like it doesn't need to be there.  Why not get straight to the point when adding nodes, like: <br />
 
 
 
    "nodes": [
 
    {"region":"west"},
 
    {"region":"west"}
 
    ]
 
 
 
*drewford: When dealing with lists of child objects, preventing individual "detail" calls for each child is a good thing.  Example - when getting details on a cluster - in a UI you would most likely also want to see a list of the nodes in the cluster.  It would be good to prevent implementors from having to call the API for each individual node just to show some important details in a nodes list.  Here are a couple options:
 
# Use the complete node detail object in the nodes array on the "Show Cluster" call
 
# Add a "List Cluster Nodes" call that returns a list of detailed node objects.
 

Latest revision as of 21:18, 6 June 2014

Example: Cassandra


To illustrate the approach, Cassandra is used in the examples below. The eccentricities of each Datastore will be explained in their own sections.

Create Cluster


Request:

POST /clusters
{
  "cluster": {
    "name": "products",
    "datastore": {
      "type": "cassandra",
      "version": "2.0.6"
    },
    "instances": [
      {
        "flavorRef": "7",
        "volume": {
          "size": 1
        },
        "configuration_id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
        "region": "west"
      },
      {
        "flavorRef": "7",
        "volume": {
          "size": 1
        },
        "configuration_id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
        "region": "east"
      },
      {
        "flavorRef": "7",
        "volume": {
          "size": 1
        },
        "configuration_id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
        "region": "eu"
      }
    ]
  }
}

Response:

Add Me


Future (TBD): Defaults

POST /clusters
{
  "cluster": {
    "name": "products",
    "datastore": {
      "type": "cassandra",
      "version": "2.0.6"
    },
    "defaults": {
      "flavorRef": "7"
      ...
    }


Show Cluster


Request:

GET /clusters/dfbbd9ca-b5e1-4028-adb7-f78643e17998

Response:

{
  "cluster": {
    "status": "ACTIVE",
    "id": "dfbbd9ca-b5e1-4028-adb7-f78643e17998",
    "name": "products",
    "created": "2014-04-25T20:19:23",
    "updated": "2014-04-25T20:19:23",
    "links": [{...}],
    "datastore": {
      "type": "cassandra",
      "version": "2.0.6"
    },
    "instances": [
      {
        "id": "416b0b16-ba55-4302-bbd3-ff566032e1c1",
        "flavor": {
          "id": "7",
          "links": [{...}]
        },
        "configuration": {
          "id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
          "links": [{...}]
        },
        "volume": {
          "size": 1
        },
        "region": "west"
      },
      {
        "id": "7f52e4f9-3fa6-4238-ac08-1ce15197329a"
        "flavor": {
          "id": "7",
          "links": [{...}]
        },
        "configuration": {
          "id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
          "links": [{...}]
        },
        "volume": {
          "size": 1
        },
        "region": "east"
      },
      {
        "id": "ff9d680c-fde3-49c6-a84e-76173b6df39d"
        "flavor": {
          "id": "7",
          "links": [{...}]
        },
        "configuration": {
          "id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
          "links": [{...}]
        },
        "volume": {
          "size": 1
        },
        "region": "eu"
      }
    ]
  }
}


Show Instance


Request:

GET /clusters/dfbbd9ca-b5e1-4028-adb7-f78643e17998/instances/416b0b16-ba55-4302-bbd3-ff566032e1c1

Response:

{
  "instance": {
    "status": "ACTIVE",
    "id": "416b0b16-ba55-4302-bbd3-ff566032e1c1",
    "name": "products-1",
    "created": "2014-04-25T20:19:23",
    "updated": "2014-04-25T20:19:23",
    "links": [{...}],
    "ip": ["10.0.0.1"],
    "configuration": {
      "id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
      "links": [{...}],
    },
    "flavor": {
      "id": "7",
      "links": [{...}],
    },
    "volume": {
      "size": 1,
      "used": 0.17
    }
  }
}


Add Instance(s)


Request:

POST /clusters/dfbbd9ca-b5e1-4028-adb7-f78643e17998/instances
{
  "instances": [
    {
      "flavorRef": "7",
      "volume": {
        "size": 1
      },
      "configuration_id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
      "region": "west"
    },
    {
      "flavorRef": "7",
      "volume": {
        "size": 1
      },
      "configuration_id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
      "region": "west"
    }
  ]
}


Response:

HTTP 202 (Empty Body)


Replace Instance


Request:

POST /clusters/dfbbd9ca-b5e1-4028-adb7-f78643e17998/action

{
  "replace_instance": {
    "id": "7f52e4f9-3fa6-4238-ac08-1ce15197329a"
  }
}


Response:

HTTP 202 (Empty Body)


Notes:


Remove Instance


Request:

DELETE /clusters/dfbbd9ca-b5e1-4028-adb7-f78643e17998/instances/7f52e4f9-3fa6-4238-ac08-1ce15197329a


Response:

HTTP 202 (Empty Body)


Notes:


Example: MongoDB


Create Cluster


Request:

POST /clusters
{
  "cluster": {
    "name": "products",
    "datastore": {
      "type": "mongodb",
      "version": "2.4.10"
    },
    "instances": [
      {
        "flavorRef": "7",
        "volume": {
          "size": 1
        },
        "configuration_id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
        "region": "west"
      },
      {
        "flavorRef": "7",
        "volume": {
          "size": 1
        },
        "configuration_id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
        "region": "west"
      },
      {
        "flavorRef": "7",
        "volume": {
          "size": 1
        },
        "configuration_id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
        "region": "west"
      },
      {
        "flavorRef": "7",
        "volume": {
          "size": 1
        },
        "configuration_id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
        "region": "east"
      },
      {
        "flavorRef": "7",
        "volume": {
          "size": 1
        },
        "configuration_id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
        "region": "east"
      }
    ]
  }
}

Response:

Add Me


Show Cluster


Request:

GET /clusters/dfbbd9ca-b5e1-4028-adb7-f78643e17998

Response:

{
  "cluster": {
    "status": "ACTIVE",
    "id": "dfbbd9ca-b5e1-4028-adb7-f78643e17998",
    "name": "products",
    "created": "2014-04-25T20:19:23",
    "updated": "2014-04-25T20:19:23",
    "links": [{...}],
    "datastore": {
      "type": "mongodb",
      "version": "2.4.10"
    },
    "instances": [
      {
        "id": "416b0b16-ba55-4302-bbd3-ff566032e1c1",
        "flavor": {
          "id": "7",
          "links": [{...}]
        },
        "configuration": {
          "id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
          "links": [{...}]
        },
        "volume": {
          "size": 1
        },
        "region": "west"
      },
      {
        "id": "965ef811-7c1d-47fc-89f2-a89dfdd23ef2"
        "flavor": {
          "id": "7",
          "links": [{...}]
        },
        "configuration": {
          "id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
          "links": [{...}]
        },
        "volume": {
          "size": 1
        },
        "region": "west"
      },
      {
        "id": "3642f41c-e8ad-4164-a089-3891bf7f2d2b"
        "flavor": {
          "id": "7",
          "links": [{...}]
        },
        "configuration": {
          "id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
          "links": [{...}]
        },
        "volume": {
          "size": 1
        },
        "region": "west"
      },
      {
        "id": "7f52e4f9-3fa6-4238-ac08-1ce15197329a"
        "flavor": {
          "id": "7",
          "links": [{...}]
        },
        "configuration": {
          "id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
          "links": [{...}]
        },
        "volume": {
          "size": 1
        },
        "region": "east"
      },
      {
        "id": "ff9d680c-fde3-49c6-a84e-76173b6df39d"
        "flavor": {
          "id": "7",
          "links": [{...}]
        },
        "configuration": {
          "id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
          "links": [{...}]
        },
        "volume": {
          "size": 1
        },
        "region": "east"
      }
    ]
  }
}


Show Instance


Request:

GET /clusters/dfbbd9ca-b5e1-4028-adb7-f78643e17998/instances/416b0b16-ba55-4302-bbd3-ff566032e1c1

Response:

{
  "instance": {
    "status": "ACTIVE",
    "id": "416b0b16-ba55-4302-bbd3-ff566032e1c1",
    "name": "products-1",
    "created": "2014-04-25T20:19:23",
    "updated": "2014-04-25T20:19:23",
    "links": [{...}],
    "ip": ["10.0.0.1"],
    "configuration": {
      "id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
      "links": [{...}],
    },
    "flavor": {
      "id": "7",
      "links": [{...}],
    },
    "volume": {
      "size": 1,
      "used": 0.17
    }
  }
}


Create Arbiter(s)


Request:

POST /clusters/dfbbd9ca-b5e1-4028-adb7-f78643e17998/instances

{
  "instances": [
    {
      "flavorRef": "7",
      "volume": {
        "size": 1
      },
      "configuration_id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
      "region": "eu",
      "type": "arbiter"
    },
    {
      "flavorRef": "7",
      "volume": {
        "size": 1
      },
      "configuration_id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
      "region": "eu",
      "type": "arbiter"
    }
  ]
}

Response:

HTTP 202 (Empty Body)


Show Cluster (After Arbiters)


Request:

GET /clusters/dfbbd9ca-b5e1-4028-adb7-f78643e17998

Response:

{
  "cluster": {
    "status": "ACTIVE",
    "id": "dfbbd9ca-b5e1-4028-adb7-f78643e17998",
    "name": "products",
    "created": "2014-04-25T20:19:23",
    "updated": "2014-04-25T20:19:23",
    "links": [{...}],
    "datastore": {
      "type": "mongodb",
      "version": "2.4.10"
    },
    "instances": [
      {
        "id": "416b0b16-ba55-4302-bbd3-ff566032e1c1",
        "flavor": {
          "id": "7",
          "links": [{...}]
        },
        "configuration": {
          "id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
          "links": [{...}]
        },
        "volume": {
          "size": 1
        },
        "region": "west"
      },
      {
        "id": "965ef811-7c1d-47fc-89f2-a89dfdd23ef2"
        "flavor": {
          "id": "7",
          "links": [{...}]
        },
        "configuration": {
          "id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
          "links": [{...}]
        },
        "volume": {
          "size": 1
        },
        "region": "west"
      },
      {
        "id": "3642f41c-e8ad-4164-a089-3891bf7f2d2b"
        "flavor": {
          "id": "7",
          "links": [{...}]
        },
        "configuration": {
          "id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
          "links": [{...}]
        },
        "volume": {
          "size": 1
        },
        "region": "west"
      },
      {
        "id": "7f52e4f9-3fa6-4238-ac08-1ce15197329a"
        "flavor": {
          "id": "7",
          "links": [{...}]
        },
        "configuration": {
          "id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
          "links": [{...}]
        },
        "volume": {
          "size": 1
        },
        "region": "east"
      },
      {
        "id": "ff9d680c-fde3-49c6-a84e-76173b6df39d"
        "flavor": {
          "id": "7",
          "links": [{...}]
        },
        "configuration": {
          "id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
          "links": [{...}]
        },
        "volume": {
          "size": 1
        },
        "region": "east"
      },
      {
        "id": "77032c55-4496-4e35-8c0d-6cd1c18e1a9c"
        "flavor": {
          "id": "7",
          "links": [{...}]
        },
        "configuration": {
          "id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
          "links": [{...}]
        },
        "volume": {
          "size": 1
        },
        "region": "eu",
        "type": "arbiter"
      },
      {
        "id": "1fd054ed-221f-4c99-8d17-570bcff4c1d2"
        "flavor": {
          "id": "7",
          "links": [{...}]
        },
        "configuration": {
          "id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
          "links": [{...}]
        },
        "volume": {
          "size": 1
        },
        "region": "eu",
        "type": "arbiter"
      }
    ]
  }
}


Example: MySQL


Create Master


Request:

POST /instances
{
  "instance": {
    "name": "products",
    "datastore": {
      "type": "mysql",
      "version": "5.5"
    },
    "configuration_id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
    "flavorRef": "7",
    "volume": {
      "size": 1
    }
  }
}

Response:

{
  "instance": {
    "status": "BUILD",
    "id": "dfbbd9ca-b5e1-4028-adb7-f78643e17998",
    "name": "products",
    "created": "2014-04-25T20:19:23",
    "updated": "2014-04-25T20:19:23",
    "links": [{...}],
    "datastore": {
      "type": "mysql",
      "version": "5.5"
    },
    "configuration": {
      "id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
      "links": [{...}],
    },
    "flavor": {
      "id": "7",
      "links": [{...}],
    },
    "volume": {
      "size": 1
    }
  }
}


Create Slave


Request:

POST /instances
{
  "instance": {
    "name": "products-slave",
    "datastore": {
      "type": "mysql",
      "version": "5.5"
    },
    "configuration_id": "fc318e00-3a6f-4f93-af99-146b44912188",
    "flavorRef": "7",
    "volume": {
      "size": 1
    },
    "slave": {
      "of": "dfbbd9ca-b5e1-4028-adb7-f78643e17998",
      "read_only": true
    }
  }
}

Response:

{
  "instance": {
    "status": "BUILD",
    "id": "061aaf4c-3a57-411e-9df9-2d0f813db859",
    "name": "products",
    "created": "2014-04-25T20:19:23",
    "updated": "2014-04-25T20:19:23",
    "links": [{...}],
    "datastore": {
      "type": "mysql",
      "version": "5.5"
    },
    "configuration": {
      "id": "fc318e00-3a6f-4f93-af99-146b44912188",
      "links": [{...}],
    },
    "flavor": {
      "id": "7",
      "links": [{...}],
    },
    "volume": {
      "size": 1
    },
    "slave": {
      "of": "dfbbd9ca-b5e1-4028-adb7-f78643e17998",
      "read_only": true
    }
  }
}


Show Master


Request:

GET /instances/dfbbd9ca-b5e1-4028-adb7-f78643e17998

Response:

{
  "instance": {
    "status": "ACTIVE",
    "id": "dfbbd9ca-b5e1-4028-adb7-f78643e17998",
    "name": "products",
    "created": "2014-04-25T20:19:23",
    "updated": "2014-04-25T20:19:23",
    "links": [{...}],
    "datastore": {
      "type": "mysql",
      "version": "5.5"
    },
    "configuration": {
      "id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
      "links": [{...}],
    },
    "flavor": {
      "id": "7",
      "links": [{...}],
    },
    "volume": {
      "size": 1
    },
    "slave": {
      "list": [
        {"id": "061aaf4c-3a57-411e-9df9-2d0f813db859"}
      ]
    }
  }
}


Show Slave


Request:

GET /instances/061aaf4c-3a57-411e-9df9-2d0f813db859

Response:

{
  "instance": {
    "status": "ACTIVE",
    "id": "061aaf4c-3a57-411e-9df9-2d0f813db859",
    "name": "products",
    "created": "2014-04-25T20:19:23",
    "updated": "2014-04-25T20:19:23",
    "links": [{...}],
    "datastore": {
      "type": "mysql",
      "version": "5.5"
    },
    "configuration": {
      "id": "fc318e00-3a6f-4f93-af99-146b44912188",
      "links": [{...}],
    },
    "flavor": {
      "id": "7",
      "links": [{...}],
    },
    "volume": {
      "size": 1
    },
    "slave": {
      "of": "dfbbd9ca-b5e1-4028-adb7-f78643e17998",
      "read_only": true
    }
  }
}


Detach Slave


Request:

POST /instances/061aaf4c-3a57-411e-9df9-2d0f813db859/action

{
  "detach": {}
}

Response:

HTTP 202 (Empty Body)


Delete Master


Request:

DELETE /instances/dfbbd9ca-b5e1-4028-adb7-f78643e17998


Response:

HTTP 202 (Empty Body)


Notes:

  • How to handle situation in which a slave is attached to a master, and the user attempts to delete the master?


Delete Slave


Request:

DELETE /instances/061aaf4c-3a57-411e-9df9-2d0f813db859


Response:

HTTP 202 (Empty Body)


Data Model Changes

clusters Table

Create a new 'clusters' Table:

CREATE TABLE "clusters" (
  "id" varchar(36) NOT NULL,
  "created" datetime DEFAULT NULL,
  "updated" datetime DEFAULT NULL,
  "name" varchar(255) DEFAULT NULL,
  "task_id" int(11) DEFAULT NULL,
  "task_description" varchar(32) DEFAULT NULL,
  "task_start_time" datetime DEFAULT NULL,
  "tenant_id" varchar(36) DEFAULT NULL,
  "server_status" varchar(64) DEFAULT NULL,
  "datastore_version_id" varchar(36) NOT NULL,
  "deleted" tinyint(1) DEFAULT NULL,
  "deleted_at" datetime DEFAULT NULL,
  PRIMARY KEY ("id"),
  KEY "datastore_version_id" ("datastore_version_id"),
  KEY "clusters_tenant_id" ("tenant_id"),
  KEY "clusters_deleted" ("deleted"),
  CONSTRAINT "clusters_ibfk_1" FOREIGN KEY ("datastore_version_id") REFERENCES "datastore_versions" ("id")
) ENGINE=InnoDB DEFAULT CHARSET=utf8;


Notes:

  • todo: task_* related columns were simply copy/pasted from instances; could diverge
  • todo: change 'default null' to 'not null' whenever possible


Alter Instances Table

Add slave_of Column to Instances Table (+ Constraint + Index):

ALTER TABLE instances ADD COLUMN slave_of VARCHAR(36) DEFAULT NULL;
KEY "slave_of" ("slave_of"),
CONSTRAINT "instances_ibfk_3" FOREIGN KEY ("slave_of") REFERENCES "instances" ("id")

Add cluster_id Column to Instances Table (+ Constraint + Index):

ALTER TABLE instances ADD COLUMN cluster_id VARCHAR(36) DEFAULT NULL;
KEY "cluster_id" ("cluster_id"),
CONSTRAINT "instances_ibfk_4" FOREIGN KEY ("cluster_id") REFERENCES "clusters" ("id")


TaskManager

  • add cluster_id to /etc/guest_info (if it's an instance in a cluster). guest_id remains as-is.
  • for-loop create each instance.
  • poll until all instances are active.
  • for each instance: use trove/nova to get ip/hostname
  • for couchbase:
    • send ip/hostname list via rpc cast to guest
  • for cassandra:
    • send seed ip list via rpc cast to guest seed instances, one by one (polling on REBOOT => ACTIVE), then to rest of instances.
  • for mongodb:
    • send ip/hostname list via rpc cast to guest that is the db.isMaster()


Guest

  • update heartbeat payload ( heartbeat(guest_id, payload, sent) ) from {"service_status": "<status>"} to {"service_status": "<status>", "cluster_id": "<cluster-id>"}
  • add method to each datastore guest manager for handling ip/hostname list


Conductor

  • update heartbeat logic to update the clusters table (cluster status) if an instance's status changes the overall state of the cluster


Capabilities


A capability might be supported for a datastore-version for standalone instances, but not for clusters. Therefore, the capability tables must be amended to include a cluster-enabled flag.

ALTER TABLE capabilities ADD COLUMN enabled_cluster TINYINT(1) DEFAULT NULL;
ALTER TABLE capability_overrides ADD COLUMN enabled_cluster TINYINT(1) DEFAULT NULL;

The following capabilities should have enabled_cluster set to false for the first iteration of clusters:

  • backup-create + list-instance
  • configuration-attach + detach + instances
  • resize-<all>
  • database-<all>
  • root-<all>
  • secgroup-<all>
  • user-<all>


Configuration Groups


Introduce read_only and hidden Parameters

Need to introduce two additional attributes for configuration group parameters: read_only and hidden.

  • read_only fields include cluster_name, num_tokens, seed_provider, seeds, endpoint_snitch (cassandra) + replSet (mongodb) + server_id, log_bin (mysql).
  • depending on the provider, some of the read_only fields should also be hidden from the user on a configuration-show.
  • once read_only + hidden are available, a parallel effort should move configuration-default to configurations-show if a configuration-group is attached.
  • amcrn (talk) 20:43, 8 May 2014 (UTC): update: mysql master/slave will not be required to do this because the overrides.cnf functionality handles this nicely. to be determined as to how clustering will handle this. it could be this, or it could be a copy of the original conf, or a mixture thereof.


Auto-Create and Attach

  • cassandra & mongodb need to have configuration-groups automatically created and attached to each instance (for cluster_name, replset, etc.) during provisioning.
  • unique configuration-group per instance.
  • auto-created+attached configuration-groups need to not be detachable from the instance.
  • dependency: configuration-group support for mongodb + cassandra
  • amcrn (talk) 20:43, 8 May 2014 (UTC): update: mysql master/slave will not be required to do this because the overrides.cnf functionality handles this nicely. to be determined as to how clustering will handle this. it could be this, or it could be a copy of the original conf, or a mixture thereof.


python-troveclient


Discussed options are documented at https://gist.github.com/amcrn/c4ae2210e9d9864c21fd . An option was chosen on 05/21 and is described below.

Create-Cluster Help

$ trove help create-cluster
usage: trove create-cluster <name> <datastore> <datastore_version>
                            [--instance <instance>]

Creates a new cluster.

Positional arguments:
  <name>                Name of the cluster
  <datastore>           A datastore name or UUID
  <datastore_version>   A datastore version name or UUID

Optional arguments:

  --instance <flavor=flavor-id,disk=disk-size,config-group=group-uuid,type=type-name>
                        Create an instance for the cluster. Specify multiple
                        times to create multiple instances.

Create Cluster Request

$ trove create-cluster products mongodb "2.4.10"
    --instance flavor=12,disk=100,config-group=e0a6a893-1e41-4a05-8252-3c2cb934dd79 \
    --instance flavor=12,disk=100,config-group=e0a6a893-1e41-4a05-8252-3c2cb934dd79 \
    --instance flavor=12,disk=100,config-group=e0a6a893-1e41-4a05-8252-3c2cb934dd79 \
    --instance flavor=12,disk=100,config-group=e0a6a893-1e41-4a05-8252-3c2cb934dd79 \
    --instance flavor=12,disk=100,config-group=e0a6a893-1e41-4a05-8252-3c2cb934dd79 \
    --instance flavor=7,disk=1,config-group=b9c8a3f8-7ace-4aea-9908-7b555586d7b6,type=arbiter \
    --instance flavor=7,disk=1,config-group=b9c8a3f8-7ace-4aea-9908-7b555586d7b6,type=arbiter

Create Cluster Response

TBD