Difference between revisions of "Trove/Clusters-MongoDB"
(Show Instance) |
(CLI Examples) |
||
(9 intermediate revisions by the same user not shown) | |||
Line 1: | Line 1: | ||
== Summary == | == Summary == | ||
+ | <br> | ||
+ | Note: Not intimately familiar with the existing API? Quickly glance through the [http://docs.rackspace.com/cdb/api/v1.0/cdb-devguide/content/API_Operations-d1e2264.html RackSpace Database API Guide], otherwise this might be a bit befuddling. | ||
<br> | <br> | ||
* Will remain on /v1.0 of the API (aka will not introduce /v2.0, meaning everything remains backwards compatible) | * Will remain on /v1.0 of the API (aka will not introduce /v2.0, meaning everything remains backwards compatible) | ||
Line 6: | Line 8: | ||
* All MongoDB clusters will be sharded clusters, with shards being three member replica-sets. Upon cluster creation, a single shard is assumed; once the initial cluster is provisioned, additional shards can be added on-demand. | * All MongoDB clusters will be sharded clusters, with shards being three member replica-sets. Upon cluster creation, a single shard is assumed; once the initial cluster is provisioned, additional shards can be added on-demand. | ||
* Each sharded cluster will have at least one query router (mongos), and at least one mongo config server (mongod --configsvr). | * Each sharded cluster will have at least one query router (mongos), and at least one mongo config server (mongod --configsvr). | ||
− | ** Query routers and config servers will not be represented as instances in the API. See more information under the Query Routers and Config Servers section. | + | ** Query routers and config servers will not be represented as instances in the API. See more information under the [[#Query_Routers_and_Config_Servers|Query Routers and Config Servers]] section. |
<br> | <br> | ||
<br> | <br> | ||
− | == | + | == API == |
<br> | <br> | ||
==== Create Cluster ==== | ==== Create Cluster ==== | ||
Line 53: | Line 55: | ||
{ | { | ||
"cluster": { | "cluster": { | ||
− | |||
"id": "dfbbd9ca-b5e1-4028-adb7-f78643e17998", | "id": "dfbbd9ca-b5e1-4028-adb7-f78643e17998", | ||
+ | "task": { | ||
+ | "id": 2, | ||
+ | "name": "BUILDING", | ||
+ | "description": "Building the initial cluster." | ||
+ | }, | ||
"name": "products", | "name": "products", | ||
"created": "2014-04-25T20:19:23", | "created": "2014-04-25T20:19:23", | ||
Line 67: | Line 73: | ||
"id": "416b0b16-ba55-4302-bbd3-ff566032e1c1", | "id": "416b0b16-ba55-4302-bbd3-ff566032e1c1", | ||
"shard_id": "5415b62f-f301-4e84-ba90-8ab0734d15a7", | "shard_id": "5415b62f-f301-4e84-ba90-8ab0734d15a7", | ||
+ | "status": "BUILD", | ||
"flavor": { | "flavor": { | ||
"id": "7", | "id": "7", | ||
Line 78: | Line 85: | ||
"id": "965ef811-7c1d-47fc-89f2-a89dfdd23ef2", | "id": "965ef811-7c1d-47fc-89f2-a89dfdd23ef2", | ||
"shard_id": "5415b62f-f301-4e84-ba90-8ab0734d15a7", | "shard_id": "5415b62f-f301-4e84-ba90-8ab0734d15a7", | ||
+ | "status": "BUILD", | ||
"flavor": { | "flavor": { | ||
"id": "7", | "id": "7", | ||
Line 89: | Line 97: | ||
"id": "3642f41c-e8ad-4164-a089-3891bf7f2d2b", | "id": "3642f41c-e8ad-4164-a089-3891bf7f2d2b", | ||
"shard_id": "5415b62f-f301-4e84-ba90-8ab0734d15a7", | "shard_id": "5415b62f-f301-4e84-ba90-8ab0734d15a7", | ||
+ | "status": "BUILD", | ||
"flavor": { | "flavor": { | ||
"id": "7", | "id": "7", | ||
Line 104: | Line 113: | ||
Notes: | Notes: | ||
* Creates a sharded cluster, with a single shard (3 member replica-set), each replica-set member with the same flavor (memory + vcpu) and volume-size. | * Creates a sharded cluster, with a single shard (3 member replica-set), each replica-set member with the same flavor (memory + vcpu) and volume-size. | ||
− | ** Also creates mongos query routers and config-servers (see Query | + | ** Also creates mongos query routers and config-servers (see [[#Query_Routers_and_Config_Servers|Query Routers and Config Servers]] for more information) |
* The replica-set name (replSet) will be automatically generated; it will be the cluster name with a "rs-<num>" suffix. | * The replica-set name (replSet) will be automatically generated; it will be the cluster name with a "rs-<num>" suffix. | ||
** In this example, the first replica-set name will be "product-rs1", the next "product-rs2", etc. | ** In this example, the first replica-set name will be "product-rs1", the next "product-rs2", etc. | ||
Line 125: | Line 134: | ||
{ | { | ||
"cluster": { | "cluster": { | ||
− | |||
"id": "dfbbd9ca-b5e1-4028-adb7-f78643e17998", | "id": "dfbbd9ca-b5e1-4028-adb7-f78643e17998", | ||
+ | "task": { | ||
+ | "id": 1, | ||
+ | "name": "NONE", | ||
+ | "description": "No tasks for the cluster." | ||
+ | }, | ||
"name": "products", | "name": "products", | ||
"created": "2014-04-25T20:19:23", | "created": "2014-04-25T20:19:23", | ||
Line 140: | Line 153: | ||
"id": "416b0b16-ba55-4302-bbd3-ff566032e1c1", | "id": "416b0b16-ba55-4302-bbd3-ff566032e1c1", | ||
"shard_id": "5415b62f-f301-4e84-ba90-8ab0734d15a7", | "shard_id": "5415b62f-f301-4e84-ba90-8ab0734d15a7", | ||
+ | "status": "ACTIVE", | ||
"flavor": { | "flavor": { | ||
"id": "7", | "id": "7", | ||
Line 151: | Line 165: | ||
"id": "965ef811-7c1d-47fc-89f2-a89dfdd23ef2", | "id": "965ef811-7c1d-47fc-89f2-a89dfdd23ef2", | ||
"shard_id": "5415b62f-f301-4e84-ba90-8ab0734d15a7", | "shard_id": "5415b62f-f301-4e84-ba90-8ab0734d15a7", | ||
+ | "status": "ACTIVE", | ||
"flavor": { | "flavor": { | ||
"id": "7", | "id": "7", | ||
Line 162: | Line 177: | ||
"id": "3642f41c-e8ad-4164-a089-3891bf7f2d2b", | "id": "3642f41c-e8ad-4164-a089-3891bf7f2d2b", | ||
"shard_id": "5415b62f-f301-4e84-ba90-8ab0734d15a7", | "shard_id": "5415b62f-f301-4e84-ba90-8ab0734d15a7", | ||
+ | "status": "ACTIVE", | ||
"flavor": { | "flavor": { | ||
"id": "7", | "id": "7", | ||
Line 176: | Line 192: | ||
Notes: | Notes: | ||
− | * The ip/hostname returned is the list of mongos query routers. See Query Routers and Config | + | * The ip/hostname returned is the list of mongos query routers. See [[#Query_Routers_and_Config_Servers|Query Routers and Config Servers]] for more information. |
<br> | <br> | ||
==== Show Instance ==== | ==== Show Instance ==== | ||
Line 258: | Line 274: | ||
* ip/hostname is not returned in GET /v1.0/<tenant_id>/clusters/<cluster-id>/instances/<instance-id> because in a sharded deployment, a client should never connect directly to the replica-set. | * ip/hostname is not returned in GET /v1.0/<tenant_id>/clusters/<cluster-id>/instances/<instance-id> because in a sharded deployment, a client should never connect directly to the replica-set. | ||
** Need to determine whether it can be omitted in GET /v1.0/<tenant_id>/instances/<instance-id> (or is it considered a breaking change?) Could consider returning the mongos query router list. | ** Need to determine whether it can be omitted in GET /v1.0/<tenant_id>/instances/<instance-id> (or is it considered a breaking change?) Could consider returning the mongos query router list. | ||
+ | <br> | ||
+ | ==== List Clusters ==== | ||
+ | <br> | ||
+ | Request: | ||
+ | |||
+ | <pre> | ||
+ | GET /v1.0/<tenant_id>/clusters | ||
+ | </pre> | ||
+ | |||
+ | Response: | ||
+ | |||
+ | <pre> | ||
+ | { | ||
+ | "clusters": [ | ||
+ | { | ||
+ | "id": "dfbbd9ca-b5e1-4028-adb7-f78643e17998", | ||
+ | "task": { | ||
+ | "id": 1, | ||
+ | "name": "NONE", | ||
+ | "description": "No tasks for the cluster." | ||
+ | }, | ||
+ | "name": "products", | ||
+ | "created": "2014-04-25T20:19:23", | ||
+ | "updated": "2014-04-25T20:19:23", | ||
+ | "links": [{...}], | ||
+ | "datastore": { | ||
+ | "type": "mongodb", | ||
+ | "version": "2.6" | ||
+ | }, | ||
+ | "instances": [ | ||
+ | { | ||
+ | "id": "416b0b16-ba55-4302-bbd3-ff566032e1c1", | ||
+ | "shard_id": "5415b62f-f301-4e84-ba90-8ab0734d15a7", | ||
+ | "status": "ACTIVE", | ||
+ | "flavor": { | ||
+ | "id": "7", | ||
+ | "links": [{...}] | ||
+ | }, | ||
+ | "volume": { | ||
+ | "size": 100 | ||
+ | } | ||
+ | }, | ||
+ | { | ||
+ | "id": "965ef811-7c1d-47fc-89f2-a89dfdd23ef2", | ||
+ | "shard_id": "5415b62f-f301-4e84-ba90-8ab0734d15a7", | ||
+ | "status": "ACTIVE", | ||
+ | "flavor": { | ||
+ | "id": "7", | ||
+ | "links": [{...}] | ||
+ | }, | ||
+ | "volume": { | ||
+ | "size": 100 | ||
+ | } | ||
+ | }, | ||
+ | { | ||
+ | "id": "3642f41c-e8ad-4164-a089-3891bf7f2d2b", | ||
+ | "shard_id": "5415b62f-f301-4e84-ba90-8ab0734d15a7", | ||
+ | "status": "ACTIVE", | ||
+ | "flavor": { | ||
+ | "id": "7", | ||
+ | "links": [{...}] | ||
+ | }, | ||
+ | "volume": { | ||
+ | "size": 100 | ||
+ | } | ||
+ | } | ||
+ | ] | ||
+ | }, | ||
+ | ... | ||
+ | ] | ||
+ | } | ||
+ | </pre> | ||
+ | <br> | ||
+ | ==== List Instances ==== | ||
+ | <br> | ||
+ | Request: | ||
+ | |||
+ | <pre> | ||
+ | GET /v1.0/<tenant_id>/instances | ||
+ | </pre> | ||
+ | |||
+ | Response: | ||
+ | |||
+ | <pre> | ||
+ | {"instances": []} | ||
+ | </pre> | ||
+ | |||
+ | Explanation | ||
+ | * GET /instances prior to clustering returned only standalone instances, and to honor that contract, we will filter out instances that belong to a cluster. | ||
+ | <br> | ||
+ | Request: | ||
+ | |||
+ | <pre> | ||
+ | GET /v1.0/<tenant_id>/instances?include_clustered=false | ||
+ | </pre> | ||
+ | |||
+ | Response: | ||
+ | |||
+ | <pre> | ||
+ | {"instances": []} | ||
+ | </pre> | ||
+ | |||
+ | Request: | ||
+ | |||
+ | <pre> | ||
+ | GET /v1.0/<tenant_id>/instances?include_clustered=true | ||
+ | </pre> | ||
+ | |||
+ | Response: | ||
+ | |||
+ | <pre> | ||
+ | { | ||
+ | "instances": [ | ||
+ | { | ||
+ | "status": "ACTIVE", | ||
+ | "id": "416b0b16-ba55-4302-bbd3-ff566032e1c1", | ||
+ | "cluster_id": "dfbbd9ca-b5e1-4028-adb7-f78643e17998", | ||
+ | "shard_id": "5415b62f-f301-4e84-ba90-8ab0734d15a7", | ||
+ | "name": "products-rs1-1", | ||
+ | "created": "2014-04-25T20:19:23", | ||
+ | "updated": "2014-04-25T20:19:23", | ||
+ | "links": [{...}], | ||
+ | "datastore": { | ||
+ | "type": "mongodb", | ||
+ | "version": "2.6" | ||
+ | }, | ||
+ | "ip": ["10.0.0.1"], | ||
+ | "flavor": { | ||
+ | "id": "7", | ||
+ | "links": [{...}], | ||
+ | }, | ||
+ | "volume": { | ||
+ | "size": 100, | ||
+ | "used": 0.17 | ||
+ | } | ||
+ | }, | ||
+ | { | ||
+ | "status": "ACTIVE", | ||
+ | "id": "416b0b16-ba55-4302-bbd3-ff566032e1c1", | ||
+ | "cluster_id": "dfbbd9ca-b5e1-4028-adb7-f78643e17998", | ||
+ | "shard_id": "5415b62f-f301-4e84-ba90-8ab0734d15a7", | ||
+ | "name": "products-rs1-2", | ||
+ | "created": "2014-04-25T20:19:23", | ||
+ | "updated": "2014-04-25T20:19:23", | ||
+ | "links": [{...}], | ||
+ | "datastore": { | ||
+ | "type": "mongodb", | ||
+ | "version": "2.6" | ||
+ | }, | ||
+ | "ip": ["10.0.0.2"], | ||
+ | "flavor": { | ||
+ | "id": "7", | ||
+ | "links": [{...}], | ||
+ | }, | ||
+ | "volume": { | ||
+ | "size": 100, | ||
+ | "used": 0.17 | ||
+ | } | ||
+ | }, | ||
+ | { | ||
+ | "status": "ACTIVE", | ||
+ | "id": "416b0b16-ba55-4302-bbd3-ff566032e1c1", | ||
+ | "cluster_id": "dfbbd9ca-b5e1-4028-adb7-f78643e17998", | ||
+ | "shard_id": "5415b62f-f301-4e84-ba90-8ab0734d15a7", | ||
+ | "name": "products-rs1-3", | ||
+ | "created": "2014-04-25T20:19:23", | ||
+ | "updated": "2014-04-25T20:19:23", | ||
+ | "links": [{...}], | ||
+ | "datastore": { | ||
+ | "type": "mongodb", | ||
+ | "version": "2.6" | ||
+ | }, | ||
+ | "ip": ["10.0.0.3"], | ||
+ | "flavor": { | ||
+ | "id": "7", | ||
+ | "links": [{...}], | ||
+ | }, | ||
+ | "volume": { | ||
+ | "size": 100, | ||
+ | "used": 0.17 | ||
+ | } | ||
+ | } | ||
+ | ] | ||
+ | } | ||
+ | </pre> | ||
+ | <br> | ||
+ | ==== Add Shard ==== | ||
+ | <br> | ||
+ | Request: | ||
+ | |||
+ | <pre> | ||
+ | POST /v1.0/<tenant_id>/clusters/dfbbd9ca-b5e1-4028-adb7-f78643e17998 | ||
+ | { | ||
+ | "add_shard": {} | ||
+ | } | ||
+ | </pre> | ||
+ | |||
+ | The additional shard will be a three member replica-set, with the flavor and volume-size matching that of the existing shard. | ||
+ | <br> | ||
+ | <br> | ||
+ | A "Show Cluster" operation for this cluster would now result in: | ||
+ | |||
+ | <pre> | ||
+ | { | ||
+ | "cluster": { | ||
+ | "id": "dfbbd9ca-b5e1-4028-adb7-f78643e17998", | ||
+ | "task": { | ||
+ | "id": 4, | ||
+ | "name": "ADDING_SHARD", | ||
+ | "description": "Adding a shard to the cluster." | ||
+ | }, | ||
+ | "name": "products", | ||
+ | "created": "2014-04-25T20:19:23", | ||
+ | "updated": "2014-04-25T20:19:23", | ||
+ | "links": [{...}], | ||
+ | "datastore": { | ||
+ | "type": "mongodb", | ||
+ | "version": "2.6" | ||
+ | }, | ||
+ | "ip": ["10.0.0.1"], | ||
+ | "instances": [ | ||
+ | { | ||
+ | "id": "416b0b16-ba55-4302-bbd3-ff566032e1c1", | ||
+ | "shard_id": "5415b62f-f301-4e84-ba90-8ab0734d15a7", | ||
+ | "status": "ACTIVE", | ||
+ | "flavor": { | ||
+ | "id": "7", | ||
+ | "links": [{...}] | ||
+ | }, | ||
+ | "volume": { | ||
+ | "size": 100 | ||
+ | } | ||
+ | }, | ||
+ | { | ||
+ | "id": "965ef811-7c1d-47fc-89f2-a89dfdd23ef2", | ||
+ | "shard_id": "5415b62f-f301-4e84-ba90-8ab0734d15a7", | ||
+ | "status": "ACTIVE", | ||
+ | "flavor": { | ||
+ | "id": "7", | ||
+ | "links": [{...}] | ||
+ | }, | ||
+ | "volume": { | ||
+ | "size": 100 | ||
+ | } | ||
+ | }, | ||
+ | { | ||
+ | "id": "3642f41c-e8ad-4164-a089-3891bf7f2d2b", | ||
+ | "shard_id": "5415b62f-f301-4e84-ba90-8ab0734d15a7", | ||
+ | "status": "ACTIVE", | ||
+ | "flavor": { | ||
+ | "id": "7", | ||
+ | "links": [{...}] | ||
+ | }, | ||
+ | "volume": { | ||
+ | "size": 100 | ||
+ | } | ||
+ | }, | ||
+ | { | ||
+ | "id": "1557208f-5c23-4537-a9f2-52a69db38d3a", | ||
+ | "shard_id": "2ba22204-f43e-44ea-a34f-f2d31e4fa709", | ||
+ | "status": "BUILD", | ||
+ | "flavor": { | ||
+ | "id": "7", | ||
+ | "links": [{...}] | ||
+ | }, | ||
+ | "volume": { | ||
+ | "size": 100 | ||
+ | } | ||
+ | }, | ||
+ | { | ||
+ | "id": "badc9861-9f68-448d-9f48-7acdcd753208", | ||
+ | "shard_id": "2ba22204-f43e-44ea-a34f-f2d31e4fa709", | ||
+ | "status": "BUILD", | ||
+ | "flavor": { | ||
+ | "id": "7", | ||
+ | "links": [{...}] | ||
+ | }, | ||
+ | "volume": { | ||
+ | "size": 100 | ||
+ | } | ||
+ | }, | ||
+ | { | ||
+ | "id": "a7f78b7c-6282-4b34-8060-693f2b67ef6b", | ||
+ | "shard_id": "2ba22204-f43e-44ea-a34f-f2d31e4fa709", | ||
+ | "status": "BUILD", | ||
+ | "flavor": { | ||
+ | "id": "7", | ||
+ | "links": [{...}] | ||
+ | }, | ||
+ | "volume": { | ||
+ | "size": 100 | ||
+ | } | ||
+ | } | ||
+ | ] | ||
+ | } | ||
+ | } | ||
+ | </pre> | ||
+ | |||
+ | With "Show Instance", "List Instances", and "List Clusters" impacted as one would expect. | ||
+ | <br> | ||
+ | <br> | ||
+ | ==== Delete Cluster ==== | ||
+ | <br> | ||
+ | Request: | ||
+ | |||
+ | <pre> | ||
+ | DELETE /v1.0/<tenant_id>/clusters/dfbbd9ca-b5e1-4028-adb7-f78643e17998 | ||
+ | </pre> | ||
+ | |||
+ | Response | ||
+ | |||
+ | <pre> | ||
+ | HTTP 202 (Empty Body) | ||
+ | </pre> | ||
+ | <br> | ||
+ | == Notes == | ||
+ | <br> | ||
+ | ==== Configuration Groups ==== | ||
+ | * Including a configuration-group in create-cluster will not be supported in the first public iteration, nor will attaching one post-creation to a cluster or an instance in a cluster. | ||
+ | * The replica-set name will be set in /etc/mongodb.conf | ||
+ | <br> | ||
+ | ==== Secondary Members and Arbiters ==== | ||
+ | * For the first iteration, given that region-awareness is not supported, and given that arbiters must be on a separate physical machine than the member(s), and given that arbiters are generally used to resolve conflicts in the event of a network partition across regions, this is also deferred until the next iteration. | ||
+ | * With the number of instances in a cluster being limited to 3, the need to support priority, hidden, or slaveDelay is also deferred until the next iteration. | ||
+ | <br> | ||
+ | ==== Converting a Standalone to a Replica-Set ==== | ||
+ | * For the first iteration, it will not be possible to convert an existing standalone MongoDB into a replica-set. | ||
+ | <br> | ||
+ | ==== Query Routers and Config Servers ==== | ||
+ | * In trove.conf, add CONF.mongodb.num_config_servers | ||
+ | ** The number of configuration servers (to hold chunk metadata). The only valid values are 1 or 3. Default to 1 (for DevStack purposes). | ||
+ | * In trove.conf, add CONF.mongodb.num_query_routers | ||
+ | ** The number of mongos query routers. The only valid values for the first iteration are 1 or 2 or 3. Default to 1 (for DevStack purposes). | ||
+ | * For the first iteration, default to the same flavor as the replica-set members for the query routers and config-servers. This is inefficient, but can be tuned in the second iteration. | ||
+ | * Query routers and Config Servers will still show up in Nova (and by inference, Horizon under "Computes") | ||
+ | * The ip/hostname list returned on a cluster-show (or the response of a cluster-create) will be the ip/hostnames of the mongos query routers. | ||
+ | * The nova instance names of the query routers will follow <clustername>-mongos-<num> and the config servers will follow <clustername>-configsvr-<num> | ||
+ | <br> | ||
+ | ==== Databases and Collections ==== | ||
+ | * As mentioned earlier, CRUD APIs will not be introduced for MongoDB databases or collections. | ||
+ | * From http://docs.mongodb.org/manual/tutorial/deploy-shard-cluster/#enable-sharding-for-a-database: | ||
+ | "Before you can shard a collection, you must enable sharding for the collection’s database. Enabling sharding for a database does not redistribute data but make it possible to shard the collections in that database. Once you enable sharding for a database, MongoDB assigns a primary shard for that database where MongoDB stores all data before sharding begins. You enable sharding on a per-collection basis." | ||
+ | <br> | ||
+ | <br> | ||
+ | ==== Supported Versions ==== | ||
+ | * Developed and tested against 2.4 and 2.6 | ||
+ | <br> | ||
+ | == CLI == | ||
+ | <br> | ||
+ | ==== Create Cluster ==== | ||
+ | <pre> | ||
+ | $ trove help cluster-create | ||
+ | |||
+ | usage: trove cluster-create <name> <datastore> <datastore_version> | ||
+ | [--instance <instance>] | ||
+ | |||
+ | Creates a new cluster. | ||
+ | |||
+ | Positional arguments: | ||
+ | <name> Name of the cluster. | ||
+ | <datastore> A datastore name or UUID. | ||
+ | <datastore_version> A datastore version name or UUID. | ||
+ | |||
+ | Optional arguments: | ||
+ | |||
+ | --instance <flavor_id=flavor_id,volume=volume> | ||
+ | Create an instance for the cluster. Specify multiple | ||
+ | times to create multiple instances. | ||
+ | </pre> | ||
+ | <br> | ||
+ | Request: | ||
+ | <pre> | ||
+ | $ trove cluster-create products mongodb "2.6" \ | ||
+ | --instance flavor_id=7,volume=2 --instance flavor_id=7,volume=2 --instance flavor_id=7,volume=2 | ||
+ | </pre> | ||
+ | |||
+ | Response: | ||
+ | <pre> | ||
+ | +-------------------+--------------------------------------+ | ||
+ | | Property | Value | | ||
+ | +-------------------+--------------------------------------+ | ||
+ | | created | 2014-08-16T01:46:51 | | ||
+ | | datastore | mongodb | | ||
+ | | datastore_version | 2.6 | | ||
+ | | id | aa6ef0f5-dbef-48cd-8952-573ad881e717 | | ||
+ | | name | products | | ||
+ | | task_description | Building the initial cluster. | | ||
+ | | task_name | BUILDING | | ||
+ | | updated | 2014-08-16T01:46:51 | | ||
+ | +-------------------+--------------------------------------+ | ||
+ | </pre> | ||
+ | <br> | ||
+ | ==== Show Cluster ==== | ||
+ | <pre> | ||
+ | $ trove help cluster-show | ||
+ | |||
+ | usage: trove cluster-show <cluster> | ||
+ | |||
+ | Shows details of a cluster. | ||
+ | |||
+ | Positional arguments: | ||
+ | <cluster> ID or name of the cluster. | ||
+ | </pre> | ||
+ | <br> | ||
+ | Request: | ||
+ | <pre> | ||
+ | $ trove cluster-show aa6ef0f5-dbef-48cd-8952-573ad881e717 | ||
+ | </pre> | ||
+ | |||
+ | Response: | ||
+ | <pre> | ||
+ | +-------------------+--------------------------------------+ | ||
+ | | Property | Value | | ||
+ | +-------------------+--------------------------------------+ | ||
+ | | created | 2014-08-16T01:46:51 | | ||
+ | | datastore | mongodb | | ||
+ | | datastore_version | 2.6 | | ||
+ | | id | aa6ef0f5-dbef-48cd-8952-573ad881e717 | | ||
+ | | ip | 10.0.0.2 | | ||
+ | | name | products | | ||
+ | | task_description | No tasks for the cluster. | | ||
+ | | task_name | NONE | | ||
+ | | updated | 2014-08-16T01:59:33 | | ||
+ | +-------------------+--------------------------------------+ | ||
+ | </pre> | ||
+ | <br> | ||
+ | ==== Show Cluster Instances ==== | ||
+ | <pre> | ||
+ | $ trove help cluster-instances | ||
+ | |||
+ | usage: trove cluster-instances <cluster> | ||
+ | |||
+ | Lists all instances of a cluster. | ||
+ | |||
+ | Positional arguments: | ||
+ | <cluster> ID or name of the cluster. | ||
+ | </pre> | ||
+ | <br> | ||
+ | Request: | ||
+ | <pre> | ||
+ | $ trove cluster-instances aa6ef0f5-dbef-48cd-8952-573ad881e717 | ||
+ | </pre> | ||
+ | |||
+ | Response: | ||
+ | <pre> | ||
+ | +--------------------------------------+----------------+-----------+------+ | ||
+ | | ID | Name | Flavor ID | Size | | ||
+ | +--------------------------------------+----------------+-----------+------+ | ||
+ | | 45532fc4-661c-4030-8ca4-18f02aa2b337 | products-rs1-1 | 7 | 2 | | ||
+ | | 7458a98d-6f89-4dfd-bb61-5cf1dd65c121 | products-rs1-2 | 7 | 2 | | ||
+ | | b37634fb-e33c-4846-8fe8-cf2b2c95e731 | products-rs1-3 | 7 | 2 | | ||
+ | +--------------------------------------+----------------+-----------+------+ | ||
+ | </pre> | ||
+ | <br> | ||
+ | ==== List Clusters ==== | ||
+ | <pre> | ||
+ | $ trove help cluster-list | ||
+ | |||
+ | usage: trove cluster-list [--limit <limit>] [--marker <ID>] | ||
+ | |||
+ | Lists all the clusters. | ||
+ | |||
+ | Optional arguments: | ||
+ | --limit <limit> Limit the number of results displayed. | ||
+ | --marker <ID> Begin displaying the results for IDs greater than the | ||
+ | specified marker. When used with --limit, set this to the | ||
+ | last ID displayed in the previous run. | ||
+ | </pre> | ||
+ | <br> | ||
+ | Request: | ||
+ | <pre> | ||
+ | $ trove cluster-list | ||
+ | </pre> | ||
+ | |||
+ | Response: | ||
+ | <pre> | ||
+ | +--------------------------------------+----------+-----------+-------------------+-----------+ | ||
+ | | ID | Name | Datastore | Datastore Version | Task Name | | ||
+ | +--------------------------------------+----------+-----------+-------------------+-----------+ | ||
+ | | aa6ef0f5-dbef-48cd-8952-573ad881e717 | products | mongodb | 2.6 | NONE | | ||
+ | | b8829c2a-b03a-49d3-a5b1-21ec974223ee | items | mongodb | 2.6 | BUILDING | | ||
+ | +--------------------------------------+----------+-----------+-------------------+-----------+ | ||
+ | </pre> | ||
+ | <br> | ||
+ | ==== Delete Cluster ==== | ||
+ | <pre> | ||
+ | $ trove help cluster-delete | ||
+ | |||
+ | usage: trove cluster-delete <cluster> | ||
+ | |||
+ | Deletes a cluster. | ||
+ | |||
+ | Positional arguments: | ||
+ | <cluster> ID of the cluster. | ||
+ | </pre> | ||
+ | <br> | ||
+ | Request: | ||
+ | <pre> | ||
+ | $ trove cluster-delete aa6ef0f5-dbef-48cd-8952-573ad881e717 | ||
+ | </pre> | ||
+ | |||
+ | Response: | ||
+ | <pre> | ||
+ | (None) | ||
+ | </pre> |
Latest revision as of 02:02, 16 August 2014
Summary
Note: Not intimately familiar with the existing API? Quickly glance through the RackSpace Database API Guide, otherwise this might be a bit befuddling.
- Will remain on /v1.0 of the API (aka will not introduce /v2.0, meaning everything remains backwards compatible)
- Will introduce /clusters resource
- Previously, an "instance" was defined as a standalone (unclustered) instance of a datastore. Now, an "instance" is either a standalone instance, or an instance that belongs to a cluster. By inference, a cluster is a collection of instances.
- All MongoDB clusters will be sharded clusters, with shards being three member replica-sets. Upon cluster creation, a single shard is assumed; once the initial cluster is provisioned, additional shards can be added on-demand.
- Each sharded cluster will have at least one query router (mongos), and at least one mongo config server (mongod --configsvr).
- Query routers and config servers will not be represented as instances in the API. See more information under the Query Routers and Config Servers section.
API
Create Cluster
Request:
POST /v1.0/<tenant_id>/clusters { "cluster": { "name": "products", "datastore": { "type": "mongodb", "version": "2.6" }, "instances": [ { "flavorRef": "7", "volume": { "size": 100 } }, { "flavorRef": "7", "volume": { "size": 100 } }, { "flavorRef": "7", "volume": { "size": 100 } } ] } }
Response:
{ "cluster": { "id": "dfbbd9ca-b5e1-4028-adb7-f78643e17998", "task": { "id": 2, "name": "BUILDING", "description": "Building the initial cluster." }, "name": "products", "created": "2014-04-25T20:19:23", "updated": "2014-04-25T20:19:23", "links": [{...}], "datastore": { "type": "mongodb", "version": "2.6" }, "instances": [ { "id": "416b0b16-ba55-4302-bbd3-ff566032e1c1", "shard_id": "5415b62f-f301-4e84-ba90-8ab0734d15a7", "status": "BUILD", "flavor": { "id": "7", "links": [{...}] }, "volume": { "size": 100 } }, { "id": "965ef811-7c1d-47fc-89f2-a89dfdd23ef2", "shard_id": "5415b62f-f301-4e84-ba90-8ab0734d15a7", "status": "BUILD", "flavor": { "id": "7", "links": [{...}] }, "volume": { "size": 100 } }, { "id": "3642f41c-e8ad-4164-a089-3891bf7f2d2b", "shard_id": "5415b62f-f301-4e84-ba90-8ab0734d15a7", "status": "BUILD", "flavor": { "id": "7", "links": [{...}] }, "volume": { "size": 100 } } ] } }
Notes:
- Creates a sharded cluster, with a single shard (3 member replica-set), each replica-set member with the same flavor (memory + vcpu) and volume-size.
- Also creates mongos query routers and config-servers (see Query Routers and Config Servers for more information)
- The replica-set name (replSet) will be automatically generated; it will be the cluster name with a "rs-<num>" suffix.
- In this example, the first replica-set name will be "product-rs1", the next "product-rs2", etc.
- The name of each instance will be automatically generated; it will be the <cluster_name>-rs<rs_num>-<instance_num>
- In this example, the instance names will be "product-rs1-1", "product-rs1-2", and "product-rs1-3".
- In the first iteration, flavor and volume-size must be identical for all instances in a replica-set.
- In the first iteration, the number of instances in a replica-set must be 3.
Show Cluster
Request:
GET /v1.0/<tenant_id>/clusters/dfbbd9ca-b5e1-4028-adb7-f78643e17998
Response:
{ "cluster": { "id": "dfbbd9ca-b5e1-4028-adb7-f78643e17998", "task": { "id": 1, "name": "NONE", "description": "No tasks for the cluster." }, "name": "products", "created": "2014-04-25T20:19:23", "updated": "2014-04-25T20:19:23", "links": [{...}], "datastore": { "type": "mongodb", "version": "2.6" }, "ip": ["10.0.0.1"], "instances": [ { "id": "416b0b16-ba55-4302-bbd3-ff566032e1c1", "shard_id": "5415b62f-f301-4e84-ba90-8ab0734d15a7", "status": "ACTIVE", "flavor": { "id": "7", "links": [{...}] }, "volume": { "size": 100 } }, { "id": "965ef811-7c1d-47fc-89f2-a89dfdd23ef2", "shard_id": "5415b62f-f301-4e84-ba90-8ab0734d15a7", "status": "ACTIVE", "flavor": { "id": "7", "links": [{...}] }, "volume": { "size": 100 } }, { "id": "3642f41c-e8ad-4164-a089-3891bf7f2d2b", "shard_id": "5415b62f-f301-4e84-ba90-8ab0734d15a7", "status": "ACTIVE", "flavor": { "id": "7", "links": [{...}] }, "volume": { "size": 100 } } ] } }
Notes:
- The ip/hostname returned is the list of mongos query routers. See Query Routers and Config Servers for more information.
Show Instance
Request:
GET /v1.0/<tenant_id>/clusters/dfbbd9ca-b5e1-4028-adb7-f78643e17998/instances/416b0b16-ba55-4302-bbd3-ff566032e1c1
Response:
{ "instance": { "status": "ACTIVE", "id": "416b0b16-ba55-4302-bbd3-ff566032e1c1", "cluster_id": "dfbbd9ca-b5e1-4028-adb7-f78643e17998", "shard_id": "5415b62f-f301-4e84-ba90-8ab0734d15a7", "name": "products-rs1-1", "created": "2014-04-25T20:19:23", "updated": "2014-04-25T20:19:23", "links": [{...}], "datastore": { "type": "mongodb", "version": "2.6" }, "flavor": { "id": "7", "links": [{...}], }, "volume": { "size": 100, "used": 0.17 } } }
or alternatively, the existing route can be used:
GET /v1.0/<tenant_id>/instances/416b0b16-ba55-4302-bbd3-ff566032e1c1
Response:
{ "instance": { "status": "ACTIVE", "id": "416b0b16-ba55-4302-bbd3-ff566032e1c1", "cluster_id": "dfbbd9ca-b5e1-4028-adb7-f78643e17998", "shard_id": "5415b62f-f301-4e84-ba90-8ab0734d15a7", "name": "products-rs1-1", "created": "2014-04-25T20:19:23", "updated": "2014-04-25T20:19:23", "links": [{...}], "datastore": { "type": "mongodb", "version": "2.6" }, "ip": ["10.0.0.1"], "flavor": { "id": "7", "links": [{...}], }, "volume": { "size": 100, "used": 0.17 } } }
Explanation
- GET /instance/<id> must function as before to avoid introducing a breaking change
Notes
- cluster_id must be added to GET /instance/<id>, otherwise users will not be able to discover the cluster-id for an instance that is discovered via GET /instances?include_clustered=true
- ip/hostname is not returned in GET /v1.0/<tenant_id>/clusters/<cluster-id>/instances/<instance-id> because in a sharded deployment, a client should never connect directly to the replica-set.
- Need to determine whether it can be omitted in GET /v1.0/<tenant_id>/instances/<instance-id> (or is it considered a breaking change?) Could consider returning the mongos query router list.
List Clusters
Request:
GET /v1.0/<tenant_id>/clusters
Response:
{ "clusters": [ { "id": "dfbbd9ca-b5e1-4028-adb7-f78643e17998", "task": { "id": 1, "name": "NONE", "description": "No tasks for the cluster." }, "name": "products", "created": "2014-04-25T20:19:23", "updated": "2014-04-25T20:19:23", "links": [{...}], "datastore": { "type": "mongodb", "version": "2.6" }, "instances": [ { "id": "416b0b16-ba55-4302-bbd3-ff566032e1c1", "shard_id": "5415b62f-f301-4e84-ba90-8ab0734d15a7", "status": "ACTIVE", "flavor": { "id": "7", "links": [{...}] }, "volume": { "size": 100 } }, { "id": "965ef811-7c1d-47fc-89f2-a89dfdd23ef2", "shard_id": "5415b62f-f301-4e84-ba90-8ab0734d15a7", "status": "ACTIVE", "flavor": { "id": "7", "links": [{...}] }, "volume": { "size": 100 } }, { "id": "3642f41c-e8ad-4164-a089-3891bf7f2d2b", "shard_id": "5415b62f-f301-4e84-ba90-8ab0734d15a7", "status": "ACTIVE", "flavor": { "id": "7", "links": [{...}] }, "volume": { "size": 100 } } ] }, ... ] }
List Instances
Request:
GET /v1.0/<tenant_id>/instances
Response:
{"instances": []}
Explanation
- GET /instances prior to clustering returned only standalone instances, and to honor that contract, we will filter out instances that belong to a cluster.
Request:
GET /v1.0/<tenant_id>/instances?include_clustered=false
Response:
{"instances": []}
Request:
GET /v1.0/<tenant_id>/instances?include_clustered=true
Response:
{ "instances": [ { "status": "ACTIVE", "id": "416b0b16-ba55-4302-bbd3-ff566032e1c1", "cluster_id": "dfbbd9ca-b5e1-4028-adb7-f78643e17998", "shard_id": "5415b62f-f301-4e84-ba90-8ab0734d15a7", "name": "products-rs1-1", "created": "2014-04-25T20:19:23", "updated": "2014-04-25T20:19:23", "links": [{...}], "datastore": { "type": "mongodb", "version": "2.6" }, "ip": ["10.0.0.1"], "flavor": { "id": "7", "links": [{...}], }, "volume": { "size": 100, "used": 0.17 } }, { "status": "ACTIVE", "id": "416b0b16-ba55-4302-bbd3-ff566032e1c1", "cluster_id": "dfbbd9ca-b5e1-4028-adb7-f78643e17998", "shard_id": "5415b62f-f301-4e84-ba90-8ab0734d15a7", "name": "products-rs1-2", "created": "2014-04-25T20:19:23", "updated": "2014-04-25T20:19:23", "links": [{...}], "datastore": { "type": "mongodb", "version": "2.6" }, "ip": ["10.0.0.2"], "flavor": { "id": "7", "links": [{...}], }, "volume": { "size": 100, "used": 0.17 } }, { "status": "ACTIVE", "id": "416b0b16-ba55-4302-bbd3-ff566032e1c1", "cluster_id": "dfbbd9ca-b5e1-4028-adb7-f78643e17998", "shard_id": "5415b62f-f301-4e84-ba90-8ab0734d15a7", "name": "products-rs1-3", "created": "2014-04-25T20:19:23", "updated": "2014-04-25T20:19:23", "links": [{...}], "datastore": { "type": "mongodb", "version": "2.6" }, "ip": ["10.0.0.3"], "flavor": { "id": "7", "links": [{...}], }, "volume": { "size": 100, "used": 0.17 } } ] }
Add Shard
Request:
POST /v1.0/<tenant_id>/clusters/dfbbd9ca-b5e1-4028-adb7-f78643e17998 { "add_shard": {} }
The additional shard will be a three member replica-set, with the flavor and volume-size matching that of the existing shard.
A "Show Cluster" operation for this cluster would now result in:
{ "cluster": { "id": "dfbbd9ca-b5e1-4028-adb7-f78643e17998", "task": { "id": 4, "name": "ADDING_SHARD", "description": "Adding a shard to the cluster." }, "name": "products", "created": "2014-04-25T20:19:23", "updated": "2014-04-25T20:19:23", "links": [{...}], "datastore": { "type": "mongodb", "version": "2.6" }, "ip": ["10.0.0.1"], "instances": [ { "id": "416b0b16-ba55-4302-bbd3-ff566032e1c1", "shard_id": "5415b62f-f301-4e84-ba90-8ab0734d15a7", "status": "ACTIVE", "flavor": { "id": "7", "links": [{...}] }, "volume": { "size": 100 } }, { "id": "965ef811-7c1d-47fc-89f2-a89dfdd23ef2", "shard_id": "5415b62f-f301-4e84-ba90-8ab0734d15a7", "status": "ACTIVE", "flavor": { "id": "7", "links": [{...}] }, "volume": { "size": 100 } }, { "id": "3642f41c-e8ad-4164-a089-3891bf7f2d2b", "shard_id": "5415b62f-f301-4e84-ba90-8ab0734d15a7", "status": "ACTIVE", "flavor": { "id": "7", "links": [{...}] }, "volume": { "size": 100 } }, { "id": "1557208f-5c23-4537-a9f2-52a69db38d3a", "shard_id": "2ba22204-f43e-44ea-a34f-f2d31e4fa709", "status": "BUILD", "flavor": { "id": "7", "links": [{...}] }, "volume": { "size": 100 } }, { "id": "badc9861-9f68-448d-9f48-7acdcd753208", "shard_id": "2ba22204-f43e-44ea-a34f-f2d31e4fa709", "status": "BUILD", "flavor": { "id": "7", "links": [{...}] }, "volume": { "size": 100 } }, { "id": "a7f78b7c-6282-4b34-8060-693f2b67ef6b", "shard_id": "2ba22204-f43e-44ea-a34f-f2d31e4fa709", "status": "BUILD", "flavor": { "id": "7", "links": [{...}] }, "volume": { "size": 100 } } ] } }
With "Show Instance", "List Instances", and "List Clusters" impacted as one would expect.
Delete Cluster
Request:
DELETE /v1.0/<tenant_id>/clusters/dfbbd9ca-b5e1-4028-adb7-f78643e17998
Response
HTTP 202 (Empty Body)
Notes
Configuration Groups
- Including a configuration-group in create-cluster will not be supported in the first public iteration, nor will attaching one post-creation to a cluster or an instance in a cluster.
- The replica-set name will be set in /etc/mongodb.conf
Secondary Members and Arbiters
- For the first iteration, given that region-awareness is not supported, and given that arbiters must be on a separate physical machine than the member(s), and given that arbiters are generally used to resolve conflicts in the event of a network partition across regions, this is also deferred until the next iteration.
- With the number of instances in a cluster being limited to 3, the need to support priority, hidden, or slaveDelay is also deferred until the next iteration.
Converting a Standalone to a Replica-Set
- For the first iteration, it will not be possible to convert an existing standalone MongoDB into a replica-set.
Query Routers and Config Servers
- In trove.conf, add CONF.mongodb.num_config_servers
- The number of configuration servers (to hold chunk metadata). The only valid values are 1 or 3. Default to 1 (for DevStack purposes).
- In trove.conf, add CONF.mongodb.num_query_routers
- The number of mongos query routers. The only valid values for the first iteration are 1 or 2 or 3. Default to 1 (for DevStack purposes).
- For the first iteration, default to the same flavor as the replica-set members for the query routers and config-servers. This is inefficient, but can be tuned in the second iteration.
- Query routers and Config Servers will still show up in Nova (and by inference, Horizon under "Computes")
- The ip/hostname list returned on a cluster-show (or the response of a cluster-create) will be the ip/hostnames of the mongos query routers.
- The nova instance names of the query routers will follow <clustername>-mongos-<num> and the config servers will follow <clustername>-configsvr-<num>
Databases and Collections
- As mentioned earlier, CRUD APIs will not be introduced for MongoDB databases or collections.
- From http://docs.mongodb.org/manual/tutorial/deploy-shard-cluster/#enable-sharding-for-a-database:
"Before you can shard a collection, you must enable sharding for the collection’s database. Enabling sharding for a database does not redistribute data but make it possible to shard the collections in that database. Once you enable sharding for a database, MongoDB assigns a primary shard for that database where MongoDB stores all data before sharding begins. You enable sharding on a per-collection basis."
Supported Versions
- Developed and tested against 2.4 and 2.6
CLI
Create Cluster
$ trove help cluster-create usage: trove cluster-create <name> <datastore> <datastore_version> [--instance <instance>] Creates a new cluster. Positional arguments: <name> Name of the cluster. <datastore> A datastore name or UUID. <datastore_version> A datastore version name or UUID. Optional arguments: --instance <flavor_id=flavor_id,volume=volume> Create an instance for the cluster. Specify multiple times to create multiple instances.
Request:
$ trove cluster-create products mongodb "2.6" \ --instance flavor_id=7,volume=2 --instance flavor_id=7,volume=2 --instance flavor_id=7,volume=2
Response:
+-------------------+--------------------------------------+ | Property | Value | +-------------------+--------------------------------------+ | created | 2014-08-16T01:46:51 | | datastore | mongodb | | datastore_version | 2.6 | | id | aa6ef0f5-dbef-48cd-8952-573ad881e717 | | name | products | | task_description | Building the initial cluster. | | task_name | BUILDING | | updated | 2014-08-16T01:46:51 | +-------------------+--------------------------------------+
Show Cluster
$ trove help cluster-show usage: trove cluster-show <cluster> Shows details of a cluster. Positional arguments: <cluster> ID or name of the cluster.
Request:
$ trove cluster-show aa6ef0f5-dbef-48cd-8952-573ad881e717
Response:
+-------------------+--------------------------------------+ | Property | Value | +-------------------+--------------------------------------+ | created | 2014-08-16T01:46:51 | | datastore | mongodb | | datastore_version | 2.6 | | id | aa6ef0f5-dbef-48cd-8952-573ad881e717 | | ip | 10.0.0.2 | | name | products | | task_description | No tasks for the cluster. | | task_name | NONE | | updated | 2014-08-16T01:59:33 | +-------------------+--------------------------------------+
Show Cluster Instances
$ trove help cluster-instances usage: trove cluster-instances <cluster> Lists all instances of a cluster. Positional arguments: <cluster> ID or name of the cluster.
Request:
$ trove cluster-instances aa6ef0f5-dbef-48cd-8952-573ad881e717
Response:
+--------------------------------------+----------------+-----------+------+ | ID | Name | Flavor ID | Size | +--------------------------------------+----------------+-----------+------+ | 45532fc4-661c-4030-8ca4-18f02aa2b337 | products-rs1-1 | 7 | 2 | | 7458a98d-6f89-4dfd-bb61-5cf1dd65c121 | products-rs1-2 | 7 | 2 | | b37634fb-e33c-4846-8fe8-cf2b2c95e731 | products-rs1-3 | 7 | 2 | +--------------------------------------+----------------+-----------+------+
List Clusters
$ trove help cluster-list usage: trove cluster-list [--limit <limit>] [--marker <ID>] Lists all the clusters. Optional arguments: --limit <limit> Limit the number of results displayed. --marker <ID> Begin displaying the results for IDs greater than the specified marker. When used with --limit, set this to the last ID displayed in the previous run.
Request:
$ trove cluster-list
Response:
+--------------------------------------+----------+-----------+-------------------+-----------+ | ID | Name | Datastore | Datastore Version | Task Name | +--------------------------------------+----------+-----------+-------------------+-----------+ | aa6ef0f5-dbef-48cd-8952-573ad881e717 | products | mongodb | 2.6 | NONE | | b8829c2a-b03a-49d3-a5b1-21ec974223ee | items | mongodb | 2.6 | BUILDING | +--------------------------------------+----------+-----------+-------------------+-----------+
Delete Cluster
$ trove help cluster-delete usage: trove cluster-delete <cluster> Deletes a cluster. Positional arguments: <cluster> ID of the cluster.
Request:
$ trove cluster-delete aa6ef0f5-dbef-48cd-8952-573ad881e717
Response:
(None)