Difference between revisions of "Trove/Replication-And-Clustering"

Revision as of 00:38, 25 April 2014

1 Summary/Viewpoints/Strategy
2 MySQL Master/Slave
3 MongoDB
4 Cassandra
5 Couchbase
6 Redis
7 Redis Cluster
8 Summary
- 8.1 Summary of cluster{} Fields
- 8.2 Summary of Route Changes
9 Open Questions
10 Discussion Points
- 10.1 Replica-Set Member Type/Attribute Updates
- 10.2 Remove a Member from a Replica-Set

Summary/Viewpoints/Strategy

Purposefully avoids introducing a /cluster API
Purposefully avoids the proliferation of configuration-groups to accommodate mandatory fields in the context of clusters/topologies
Purposefully avoids introducing a v2 API
Instead, introduces cluster{} field and /instance/<id>/cluster routes
cluster{}'s charter is to include cluster-related fields that are absolutely required to construct the initial cluster.
Cluster-influenced actions are handled via: 'PUT /instances/<id>/cluster', where <id> is any arbitrary instance in the cluster.
Requests against 'PUT /instances/<id>/cluster' will follow the format of: { "<action>": { <action_specific_fields> } }
Where possible and logical, consolidate similar <action>s amongst datastores.

MySQL Master/Slave

Create Master

Request:

POST /instances
{
  "instance": {
    "name": "products",
    "datastore": {
      "type": "mysql",
      "version": "5.5"
    },
    "configuration": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
    "flavorRef": "7",
    "volume": {
      "size": 1
    }
  }
}

Response:

{
  "instance": {
    "status": "BUILD",
    "id": "dfbbd9ca-b5e1-4028-adb7-f78643e17998",
    "name": "products",
    "configuration": {
      "id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
      "name": "config-a",
      "links": [{...}]
    },
    ...
  }
}

Create Slave

--vipuls (talk) 00:38, 25 April 2014 (UTC) I'm not a big fan of introducing the 'cluster' attribute to wrap what is a read-only slave. Request:

POST /instances
{
  "instance": {
    "name": "product-b",
    "datastore": {
      "type": "mysql",
      "version": "5.5"
    },
    "cluster": {
      "slave_of": "dfbbd9ca-b5e1-4028-adb7-f78643e17998",
      "read_only": true
    },
    "configuration": "fc318e00-3a6f-4f93-af99-146b44912188",
    "flavorRef": "7",
    "volume": {
      "size": 1
    }
  }
}

Response:

{
  "instance": {
    "status": "BUILD",
    "id": "061aaf4c-3a57-411e-9df9-2d0f813db859",
    "name": "product-b",
    "cluster": {
      "slave_of": "dfbbd9ca-b5e1-4028-adb7-f78643e17998",
      "read_only": true
    },
    "configuration": {
      "id": "fc318e00-3a6f-4f93-af99-146b44912188",
      "name": "config-b",
      "links": [{...}]
    },
    ...
  }
}

Notes:

For master/slave wirings, the 'server_id' must differ between master and slave, and optionally the slave can specify whether it is read-only or not to avoid accidental writes.
Update: Agreed on 03/14/14 that the user should not have to specify 'server_id'. Instead, trove will be responsible for setting it and ensuring that a slave does not have the same server_id as its master, or any sibling slaves. As a part of this agreement, this requires removing 'server_id' from configuration-groups overrides (to avoid the user meddling with our bookkeeping).
'read_only' was removed from configuration-groups and moved to cluster{} because forcing the user to create a configuration-group for every MySQL slave is arduous and a poor user experience.
'read_only' is only permitted if 'slave_of' is set, otherwise the request will be failed.
Opinion: 'read_only' should default to true if slave_of is set and read_only is not provided.
Note: 'slave_of' is purposefully not an array to support multi-source replication coming in 5.7. Instead, a new 'channels'-like field will be introduced.
For now, 'slave_of' requires a vanilla trove instance uuid, but will inevitably need to be prefixed with namespacing to support multiple dcs and sources (e.g. trove:us-west:tenant_id:instance:dfbbd9ca-b5e1-4028-adb7-f78643e17998) and/or additional fields will be added.
Note: The master could have optionally included instance.cluster{} to provide a cluster_name to avoid it being the same as the master's name (see below for context)

Show Instance

Request:

GET /instances/dfbbd9ca-b5e1-4028-adb7-f78643e17998

Response:

{
  "instance": {
    "status": "ACTIVE",
    "updated": "2014-02-16T03:38:49"
    "id": "dfbbd9ca-b5e1-4028-adb7-f78643e17998",
    "name": "products",
    "datastore": {
      "version": "5.5",
      "type": "mysql",
    },
    "flavor": {
      "id": "7",
      "links": [{...}]
    },
    "configuration": {
      "id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
      "name": "config-a",
      "links": [{...}]
    }
  }
}

Request:

GET /instances/061aaf4c-3a57-411e-9df9-2d0f813db859

Response:

{
  "instance": {
    "status": "ACTIVE",
    "updated": "2014-02-16T03:38:49"
    "id": "061aaf4c-3a57-411e-9df9-2d0f813db859",
    "name": "product-b",
    "datastore": {
      "version": "5.5",
      "type": "mysql",
    },
    "flavor": {
      "id": "7",
      "links": [{...}]
    },
    "cluster": {
      "slave_of": "dfbbd9ca-b5e1-4028-adb7-f78643e17998",
      "read_only": true
    },
    "configuration": {
      "id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
      "name": "config-a",
      "links": [{...}]
    }
  }
}

Show Cluster

Request:

GET /instances/dfbbd9ca-b5e1-4028-adb7-f78643e17998/cluster

Response:

{
  "cluster": {
    "name": "products",
    "instances": [
      {
        "id": "dfbbd9ca-b5e1-4028-adb7-f78643e17998",
        "name": "products"
      },
      {
        "id": "061aaf4c-3a57-411e-9df9-2d0f813db859",
        "name": "product-b",
        "slave_of": "dfbbd9ca-b5e1-4028-adb7-f78643e17998",
        "read_only": true
      }
    ]
  }
}

Remove Replication (aka "Promote" to Standalone)

Request:

   POST or PUT /instances/dfbbd9ca-b5e1-4028-adb7-f78643e17998/cluster/action
or POST or PUT /instances/dfbbd9ca-b5e1-4028-adb7-f78643e17998/cluster

{
  "promote": {
    "id": "061aaf4c-3a57-411e-9df9-2d0f813db859"
  }
}

Response:

TBD

Notes:

The PUT /cluster option is more "correct", but POST /action is already an established pattern in the codebase. We must choose one of the four approaches. For the sake of brevity, it is assumed that PUT /cluster is chosen (and will be used in examples beyond this point in the document)

MongoDB

Create Single Instance Replica-Set

Request:

POST /instances
{
  "instance": {
    "name": "product-a",
    ...
    "datastore": {
      "type": "mongodb",
      "version": "2.4.10"
    },
    "cluster": {
      "cluster_name": "products",
      "instance_type": "member",
      "join": false
    },
    ...
  }
}

Response:

{
  "instance": {
    "id": "dfbbd9ca-b5e1-4028-adb7-f78643e17998",
    ...
  }
}

Notes:

The existing instance-create payload for single instance MongoDB will be treated exactly the same as this example (with instance.name becoming cluster_name, instance_type defaulting to member, and join defaulting to false)
'cluster_name' is used as 'replSet'
Enforce 'cluster_name' field to be provided for MongoDB, even in the case of a standalone/single instance. See http://www.mongodb.com/blog/post/dont-let-your-standalone-mongodb-server-stand-alone for reasoning.
'join' indicates whether you're joining an existing replica-set, or creating a new one. If 'join' is false, and an active replica-set by that name for the tenant already exists, the request will be failed. 'join' by default will be false, but was included above for illustrative purposes.
'instance_type' is 'member' vs. 'primary' because in a replica-set, the primary is dynamic and can change in an election.

Create 3 Instance Replica-Set

Request:

POST /instances
{
  "instance": {
    "name": "",
    ...
    "datastore": {
      "type": "mongodb",
      "version": "2.4.10"
    },
    "cluster": {
      "cluster_name": "products",
      "instance_type": "member",
      "num_instances": 3,
      "join": false
    },
    ...
  }
}

Response:

{
  TBD
}

Notes:

cluster.num_instances defaults to 1
instance.name cannot be provided if instance.cluster.num_instances != 1 (implicitly or explicitly)
cluster.num_instances is not supported for mysql and possibly others (see the Summary table at the bottom of this document)
the names of the instances for num_instances != 1 will be "<cluster_name> - <suffix>", where suffix is either an incrementing number (e.g. product-1, product-2) or some variation thereof.
it's understood that num_instances itself will not be sufficient in the future because users will want to split the allocation across different availability-zones. tying num_instance split allocations to az(s) and/or region(s) is beyond the scope of this first iteration, and therefore, all nodes will land as they would land today (az/region-wise). it's worth pointing out that building node by node vs. using the convenience of num_instances is a workaround.

Add Member to Replica-Set

Request:

POST /instances
{
  "instance": {
    "name": "product-b",
    ...
    "cluster": {
      "instance_type": "member",
      "cluster_name": "products",
      "join": true
    },
    ...
  }
}

Response:

{
  "instance": {
    "status": "BUILD",
    "id": "061aaf4c-3a57-411e-9df9-2d0f813db859",
    ...
  }
}

Notes:

If 'join' is true, and there is no existing replica-set for the tenant matching the 'cluster_name' value, the request will be failed.
Will have to use 'db.isMaster()' to determine the current primary to execute replica-set commands against (since it can be dynamic due to elections)
Will use http://docs.mongodb.org/manual/tutorial/expand-replica-set/#configure-and-add-a-member
Should protect against adding more than 12 members to a replica-set
Should protect against adding more than 7 voting members to a replica-set
Should return warning when number of voting members is even and there is no arbiter

Add Another Member to Replica-Set

Request:

POST /instances
{
  "instance": {
    "name": "product-c",
    ...
    "cluster": {
      "instance_type": "member",
      "cluster_name": "products",
      "join": true
    },
    ...
  }
}

Response:

{
  "instance": {
    "status": "BUILD",
    "id": "3a72ee87-cf3e-40f1-a1e1-fe8c7263a782",
    ...
  }
}

Show Instance

Request:

GET /instances/dfbbd9ca-b5e1-4028-adb7-f78643e17998

Response:

{
  "instance": {
    ...
    "cluster": {
      "instance_type": "member",
      "cluster_name": "products"
    },
    ...
  }
}

Show Cluster

Request:

GET /instances/dfbbd9ca-b5e1-4028-adb7-f78643e17998/cluster

Response:

{
  "cluster": {
    "name": "products",
    "instances": [
      {
        "id": "dfbbd9ca-b5e1-4028-adb7-f78643e17998",
        "name": "product-a",
        "instance_type": "member"
      },
      {
        "id": "061aaf4c-3a57-411e-9df9-2d0f813db859",
        "name": "product-b",
        "instance_type": "member"
      },
      {
        "id": "3a72ee87-cf3e-40f1-a1e1-fe8c7263a782",
        "name": "product-c",
        "instance_type": "member"
      }
    ]
  }
}

Add Arbiter

Request:

POST /instances
{
  "instance": {
    "name": "product-arbiter",
    ...
    "cluster": {
      "instance_type": "arbiter",
      "cluster_name": "products",
      "join": true
    },
    ...
  }
}

Response:

{
  "instance": {
    "status": "BUILD",
    "id": "a1b62aaa-7863-4384-8250-59024141c1f8",
    ...
  }
}

Add a Delayed Member

Request:

POST /instances
{
  "instance": {
    "name": "product-delayed",
    ...
    "cluster": {
      "instance_type": "member",
      "cluster_name": "products",
      "priority": 0,
      "hidden": true,
      "slaveDelay": 3600,
      "join": true
    },
    ...
  }
}

Response:

{
  "instance": {
    "status": "BUILD",
    "id": "7d8eb019-931b-4b2a-88d2-4c9f0ca1b29e",
    ...
  }
}

Notes:

'instance_type', 'cluster_name', 'join', 'priority', 'hidden', and 'slaveDelay' are the only fields supported in cluster{} for mongodb. All other configuration values must be set via a configuration-group. After more thought, consider supporting 'hostname' and 'votes' as well.
Why isn't 'priority', 'hidden' and 'slaveDelay' in a configuration-group you ask? This is explained in "Modifying a Replica-Set" below.

Modifying a Replica-Set

Thus far we've been able to model building a replica-set, adding an arbiter, adding a delayed secondary member, etc. Let's continue with how to modify a replica-set.

Example:

# from http://docs.mongodb.org/manual/tutorial/configure-secondary-only-replica-set-member/#example
cfg = rs.conf()
cfg.members[0].priority = 2
cfg.members[1].priority = 1
cfg.members[2].priority = 0.5
cfg.members[3].priority = 0
rs.reconfig(cfg)

Executing these priority changes one at a time can have catastrophic results, so it must be done as a transaction (with rs.reconfig() commiting). However, without the ability to address the cluster (i.e. multiple members at once), this becomes impossible. The only backdoor solution would be to guarantee that the MongoDB user(s) presented to the cloud tenant all have the clusterAdmin role, as this would allow them to connect to the primary and execute such transactions themselves via the native client. Obviously however, granting clusterAdmin to every DBaaS user in MongoDB is unacceptable in most deployments. The solution is as follows:

Request:

PUT /instances/dfbbd9ca-b5e1-4028-adb7-f78643e17998/cluster
{
  "update_member_attributes": {
    "instances": [
      {
        "id": "dfbbd9ca-b5e1-4028-adb7-f78643e17998",
        "priority": 2
      },
      {
        "id": "061aaf4c-3a57-411e-9df9-2d0f813db859",
        "priority": 1
      },
      {
        "id": "3a72ee87-cf3e-40f1-a1e1-fe8c7263a782",
        "priority": 0.5
      }
    ]
  }
}

Notes:

'update_member_attributes.members[]' elements will only permit 'priority', 'hidden', and 'slaveDelay' (possibly 'votes' and 'hostname' as mentioned earlier).
See https://wiki.openstack.org/w/index.php?title=Trove/Replication-And-Clustering&action=submit#Replica-Set_Member_Type.2FAttribute_Updates for a list of approaches considered.

Remove a Member

Request:

PUT /instances/dfbbd9ca-b5e1-4028-adb7-f78643e17998/cluster

{
  "promote": {
    "id": "dfbbd9ca-b5e1-4028-adb7-f78643e17998"
  }
}

Notes:

Note that 'promote' was used here vs. "remove_member" because the action is akin to promoting a mysql/redis slave instance (see http://docs.mongodb.org/manual/reference/method/rs.remove/#rs.remove)
See https://wiki.openstack.org/w/index.php?title=Trove/Replication-And-Clustering&action=submit#Remove_a_Member_from_a_Replica-Set for a list of approaches considered.

MongoDB TokuMX

TokuMUX will require a new datastore-version and *possibly* a new manager class (same reasoning as why Tungsten/Galera will have their own datastore-version for MySQL)

Cassandra

Create Cluster

Request:

POST /instances
{
  "instance": {
    "name": "product-a",
    ...
    "datastore": {
      "type": "cassandra",
      "version": "cassandra-2.0.5"
    },
    "cluster": {
      "cluster_name": "products",
      "num_tokens": 256,
      "is_seed": true,
      "endpoint_snitch": "RackInferringSnitch",
      "join": false
    },
    ...
  }
}

Response:

{
  "instance": {
    "id": "dfbbd9ca-b5e1-4028-adb7-f78643e17998",
    ...
  }
}

Notes:

Unlike in MongoDB, the 'instance_type' field is not required (because all instances of the cluster are of the same type)
'cluster_name', 'num_tokens', and 'is_seed' are always required, with 'endpoint_snitch' being required if 'join' is false (if 'join' is true, the endpoint_snitch is inherited) and 'auto_bootstrap' required if 'join' is true.
'seed_provider' can optionally be provided, but conveniently defaults to 'org.apache.cassandra.locator.SimpleSeedProvider'
'join' indicates whether you're joining an existing cluster, or creating a new one. If 'join' is false, and an active cluster by that name for the tenant already exists, the request will be failed. 'join' by default will be false, but was included above for illustrative purposes.
Likely the Keystone region and the availability-zone inherent to the trove request can be used for the data-center and rack (cassandra-topology.properties), but if it turns out that the naming schemes are incompatible, 'data_center' and 'rack' can be introduced.
'is_seed' is used vs. a seed list of ip-addresses because (1) the ip-address is not yet known and (2) when additional seeds are added, each instance in the cluster must be notified and updated. More on this later.

Add Instance to Cluster

Request:

POST /instances
{
  "instance": {
    "name": "product-b",
    ...
    "cluster": {
      "cluster_name": "products",
      "num_tokens": 256,
      "is_seed": false,
      "auto_bootstrap": false,
      "join": true
    },
    ...
  }
}

Response:

{
  "instance": {
    "status": "BUILD",
    "id": "061aaf4c-3a57-411e-9df9-2d0f813db859",
    ...
  }
}

Notes:

If 'join' is true, and there is no existing cluster for the tenant matching the 'cluster_name' value, the request will be failed.
If 'endpoint_snitch' is provided, and the value does not match that of the existing instances(s) in the cluster, the request will be failed.

Add Another Instance to Cluster

Request:

POST /instances
{
  "instance": {
    "name": "product-c",
    ...
    "cluster": {
      "cluster_name": "products",
      "num_tokens": 256,
      "is_seed": false,
      "auto_bootstrap": false,
      "join": true
    },
    ...
  }
}

Response:

{
  "instance": {
    "status": "BUILD",
    "id": "3a72ee87-cf3e-40f1-a1e1-fe8c7263a782",
    ...
  }
}

Show Instance

Request:

GET /instances/dfbbd9ca-b5e1-4028-adb7-f78643e17998

Response:

{
  "instance": {
    ...
    "cluster": {
      "cluster_name": "products",
      "num_tokens": 256,
      "is_seed": true
    },
    ...
  }
}

Show Cluster

Request:

GET /instances/dfbbd9ca-b5e1-4028-adb7-f78643e17998/cluster

Response:

{
  "cluster": {
    "name": "products",
    "instances": [
      {
        "id": "dfbbd9ca-b5e1-4028-adb7-f78643e17998",
        "name": "product-a",
        "num_tokens": 256,
        "is_seed": true
      },
      {
        "id": "061aaf4c-3a57-411e-9df9-2d0f813db859",
        "name": "product-b",
        "num_tokens": 256,
        "is_seed": false
      },
      {
        "id": "3a72ee87-cf3e-40f1-a1e1-fe8c7263a782",
        "name": "product-c",
        "num_tokens": 256,
        "is_seed": false
      }
    ]
  }
}

Modifying a Cluster

Example: Drain (http://www.datastax.com/documentation/cassandra/2.0/cassandra/tools/toolsDrain.html)

PUT /instances/dfbbd9ca-b5e1-4028-adb7-f78643e17998/cluster
{
  "drain": {
    "id": "dfbbd9ca-b5e1-4028-adb7-f78643e17998"
  }
}

Remove an Instance

Request:

PUT /instances/dfbbd9ca-b5e1-4028-adb7-f78643e17998/cluster
{
  "promote": {
    "id": "dfbbd9ca-b5e1-4028-adb7-f78643e17998"
  }
}

Notes:

Note that 'promote' was used here vs. "remove/decomm_instance" because the action is akin to promoting a mysql/redis slave instance (see https://wiki.apache.org/cassandra/Operations#Removing_nodes_entirely)

Couchbase

Create Cluster

Create Initial Cluster

Request:

POST /instances
{
  "instance": {
    "name": "product-a",
    ...
    "datastore": {
      "type": "couchbase",
      "version": "2.5.1"
    },
    "cluster": {
      "cluster_name": "products",
      "join": false
    },
    ...
  }
}

Response:

{
  "instance": {
    "id": "dfbbd9ca-b5e1-4028-adb7-f78643e17998",
    ...
  }
}

Notes:

Couchbase does not support a "cluster name", but does require a xdcr-cluster-name when using XDCR.
Tip: Summary of operations at http://docs.couchbase.com/couchbase-manual-2.2/#xdcr-replicate-options

Add Instance to Cluster

Request:

POST /instances
{
  "instance": {
    "name": "product-b",
    ...
    "cluster": {
      "cluster_name": "products",
      "join": true
    },
    ...
  }
}

Response:

{
  "instance": {
    "status": "BUILD",
    "id": "061aaf4c-3a57-411e-9df9-2d0f813db859",
    ...
  }
}

Notes:

If 'join' is true, and there is no existing cluster for the tenant matching the 'cluster_name' value, the request will be failed.
In Couchbase, a new instance can join the cluster by referencing any existing instance, which is discoverable by trove via the cluster_name (unique per tenant)

Add Another Instance to Cluster

Request/Response omitted due to a lack of any special considerations required

Show Instance

Request/Response omitted due to a lack of any special considerations required

Show Cluster

Request/Response omitted due to a lack of any special considerations required

Modifying a Cluster

Example: Create Bucket

PUT /instances/dfbbd9ca-b5e1-4028-adb7-f78643e17998/cluster

{
  "create_bucket": {
    "bucket": "test_bucket",
    "bucket_type": "couchbase",
    "bucket_port": 11222,
    "bucket_ramsize": 200,
    "bucket_replica": 1
  }
}

Remove a Instance

Request:

PUT /instances/dfbbd9ca-b5e1-4028-adb7-f78643e17998/cluster

{
  "failover": {
    "ids": ["dfbbd9ca-b5e1-4028-adb7-f78643e17998"],
    "rebalance": true
  }
}

Notes:

As noted in the Couchbase documentation, removing an instance is akin to a failover and has extreme impliciations. For this reason, "promote" was not used (that, and "rebalance" is optional.)

Redis

Create Master

Request:

POST /instances
{
  "instance": {
    "name": "products",
    ...
    "datastore": {
      "type": "redis",
      "version": "2.8.6"
    },
    ...
  }
}

Response:

{
  "instance": {
    "id": "dfbbd9ca-b5e1-4028-adb7-f78643e17998",
    ...
  }
}

Notes:

cluster{} is not required for redis
note: 'join' field is not required for redis. also notice the lack of a 'cluster_name' of sorts (see 'Add Slave' for reasoning)

Add Slave

Request:

POST /instances
{
  "instance": {
    "name": "product-b",
    ...
    "datastore": {
      "type": "redis",
      "version": "2.8.6"
    },
    "cluster": {
      "slave_of": "dfbbd9ca-b5e1-4028-adb7-f78643e17998"
    },
    ...
  }
}

Response:

{
  "instance": {
    "status": "BUILD",
    "id": "061aaf4c-3a57-411e-9df9-2d0f813db859",
    ...
  }
}

Notes:

redis supports daisy-chaining slaves, therefore the 'slave_of' value needs to be a specific trove instance uuid vs. a manufactured 'cluster_name' of sorts.
whether it is a master or slave can be inferred from the presence (or lack thereof) of 'slave_of'.
note that 'slave_of' is also used for mysql (the datastore information will have to be used to determine the semantic difference)

Show Instance

Request:

GET /instances/dfbbd9ca-b5e1-4028-adb7-f78643e17998

Response:

{
  "instance": {
    ...
  }
}

Show Cluster

Request:

GET /instances/dfbbd9ca-b5e1-4028-adb7-f78643e17998/cluster

Response:

{
  "cluster": {
    "instances": [
      {
        "id": "dfbbd9ca-b5e1-4028-adb7-f78643e17998",
        "name": "products"
      },
      {
        "id": "061aaf4c-3a57-411e-9df9-2d0f813db859",
        "name": "product-b",
        "slave_of": "dfbbd9ca-b5e1-4028-adb7-f78643e17998"
      }
    ]
  }
}

Promote/Disconnect Slave

Request:

PUT /instances/dfbbd9ca-b5e1-4028-adb7-f78643e17998/cluster

{
  "promote": {
    "id": "dfbbd9ca-b5e1-4028-adb7-f78643e17998"
  }
}

Notes:

note that 'promote' is also used for mysql (the datastore information will have to be used to determine the semantic difference)

Redis Cluster

Create Cluster

Request:

POST /instances
{
  "instance": {
    "name": "product-a",
    ...
    "datastore": {
      "type": "redis",
      "version": "3.0.0-beta1"
    },
    "cluster": {
      "cluster_name": "products",
      "join": false
    },
    ...
  }
}

Response:

{
  "instance": {
    "id": "dfbbd9ca-b5e1-4028-adb7-f78643e17998",
    ...
  }
}

Notes:

Will require datastore-version (to 'manager' to cluster{}) validation that cluster_name is supported for redis (because redis 3.x supports clusters, whereas 2.x does not)
'cluster-node-timeout' should be handled by configuration by the deployer (outside the context of configuration-group).
Note: shares 'cluster_name' with Cassandra.

Add Another Master to Cluster

Request:

POST /instances
{
  "instance": {
    "name": "product-b",
    ...
    "datastore": {
      "type": "redis",
      "version": "3.0.0-beta1"
    },
    "cluster": {
      "cluster_name": "products",
      "join": true
    },
    ...
  }
}

Response:

{
  "instance": {
    "status": "BUILD",
    "id": "061aaf4c-3a57-411e-9df9-2d0f813db859",
    ...
  }
}

Add Slave to Cluster

Request:

POST /instances
{
  "instance": {
    "name": "product-c",
    ...
    "datastore": {
      "type": "redis",
      "version": "3.0.0-beta1"
    },
    "cluster": {
      "cluster_name": "products",
      "slave_of": "dfbbd9ca-b5e1-4028-adb7-f78643e17998"
    },
    ...
  }
}

Response:

{
  "instance": {
    "status": "BUILD",
    "id": "3a72ee87-cf3e-40f1-a1e1-fe8c7263a782",
    ...
  }
}

Notes:

Despite redis clusters supporting the ability to add a slave without designating the master, we will require it to avoid unoptimal geographical relationships.

Show Instance

Request:

GET /instances/dfbbd9ca-b5e1-4028-adb7-f78643e17998

Response:

{
  "instance": {
    ...
    "cluster": {
      "cluster_name": "products"
    },
    ...
  }
}

Show Cluster

Request:

GET /instances/dfbbd9ca-b5e1-4028-adb7-f78643e17998/cluster

Response:

{
  "cluster": {
    "name": "products",
    "instances": [
      {
        "id": "dfbbd9ca-b5e1-4028-adb7-f78643e17998",
        "name": "product-a"
      },
      {
        "id": "061aaf4c-3a57-411e-9df9-2d0f813db859",
        "name": "product-b"
      },
      {
        "id": "3a72ee87-cf3e-40f1-a1e1-fe8c7263a782",
        "name": "product-c",
        "slave_of": "dfbbd9ca-b5e1-4028-adb7-f78643e17998"
      }
    ]
  }
}

Promote/Disconnect Slave

Work in Progress

Summary

Summary of cluster{} Fields

Field	Datastore(s)
slave_of	mysql, redis, redis-cluster
read_only	mysql
instance_type	mongodb
num_instances	mongodb, cassandra, couchbase, redis-cluster
cluster_name	mongodb, cassandra, couchbase, redis-cluster
priority	mongodb
hidden	mongodb
slaveDelay	mongodb
hostname	mongodb
votes	mongodb
num_tokens	cassandra
is_seed	cassandra
endpoint_snitch	cassandra
auto_bootstrap	cassandra
join	mongodb, cassandra, couchbase, redis-cluster

Questions:

Is this an acceptable number of added fields, is it confusing, and is the growth of fields as we add more datastores sustainable?

Summary of Route Changes

New Routes:

GET /instances/<id>/cluster

{
  "cluster": {
    "instances": [
      {
        "id": "<id>",
        "name": "<name>",
        <datastore-specific cluster fields>
      },
      <repeat>
    ]
  }
}

PUT /instances/<id>/cluster

{
  "<action>": {
    <action-specific fields, with one or more including the targeted id(s)>
  }
}

Open Questions

How to handle additional filtering requirements for instance-list (need to likely filter by cluster_name, so how to handle mysql/redis in these scenarios).

Discussion Points

Replica-Set Member Type/Attribute Updates

Option #1: PATCH /instances/:id/cluster

Request:

PATCH /instances/dfbbd9ca-b5e1-4028-adb7-f78643e17998/cluster

{
  "instances": [
    {
      "id": "dfbbd9ca-b5e1-4028-adb7-f78643e17998",
      "priority": 2
    },
    {
      "id": "061aaf4c-3a57-411e-9df9-2d0f813db859",
      "priority": 1
    },
    {
      "id": "3a72ee87-cf3e-40f1-a1e1-fe8c7263a782",
      "priority": 0.5
    }
  ]
}

Response:

{
  "cluster": {
    "name": "products",
    "instances": [
      {
        "id": "dfbbd9ca-b5e1-4028-adb7-f78643e17998",
        "name": "product-a",
        "instance_type": "member"
      },
      {
        "id": "061aaf4c-3a57-411e-9df9-2d0f813db859",
        "name": "product-b",
        "instance_type": "member"
      },
      {
        "id": "3a72ee87-cf3e-40f1-a1e1-fe8c7263a782",
        "name": "product-c",
        "instance_type": "member"
      }
    ]
  }
}

Notes:

An HTTP PATCH vs. PUT because the omission of a field should not be an indication to drop/delete it.
All modified fields in a request will be changed transactionally in a single rs.reconfig().
It should now be clear why 'priority', 'hidden' and 'slaveDelay' are in cluster{} vs. a configuration-group: when a configuration-group is changed, an event is immediately triggered to update any attached trove instances. Therefore, if you have a heterogeneous mixture of configuration-groups in a replica-set, there is no way to coordinate a consolidated rs.reconfig().
Downside: cluster{} may have fields returned on a GET that you cannot change in a PATCH/PUT; re-worded, the granularity of what is permissible to change in a PATCH becomes complicated to check and validate.
TBD on what should be returned in the cluster{} on a GET /instance/:id and GET /instance/:id/cluster. It's a question of whether we should persist anything beyond the 'instance_type' and 'cluster_name'. If say the 'priority' is stored, you introduce the possibility of drift from the truth, but can easily return it on a GET; if it's not stored, do we prompt MongoDB for the truth on a GET, or is that too computationally expensive?

Option #2: POST /instances/:id/cluster/action

Request:

   POST or PUT /instances/dfbbd9ca-b5e1-4028-adb7-f78643e17998/cluster/action
or POST or PUT /instances/dfbbd9ca-b5e1-4028-adb7-f78643e17998/cluster

{
  "update_member_attributes": {
    "instances": [
      {
        "id": "dfbbd9ca-b5e1-4028-adb7-f78643e17998",
        "priority": 2
      },
      {
        "id": "061aaf4c-3a57-411e-9df9-2d0f813db859",
        "priority": 1
      },
      {
        "id": "3a72ee87-cf3e-40f1-a1e1-fe8c7263a782",
        "priority": 0.5
      }
    ]
  }
}

Notes:

'update_member_attributes.members[]' elements will only permit 'priority', 'hidden', and 'slaveDelay' (possibly 'votes' and 'hostname' as mentioned earlier).
Due to the limited field-set, this approach is much more fine-grained than the PATCH approach in Option #1.

Decision: Option #2 is more fine-grained, easier to reason about, and less error-prone. As you'll see in later operations (like Remove a Member), is also more appropriate.

Remove a Member from a Replica-Set

Removing a member from a cluster/cluster is not the same as deleting one, therefore DELETE /instances/:id is not appropriate.

Option #1: PUT /instances/:id/cluster

Request:

PUT /instances/dfbbd9ca-b5e1-4028-adb7-f78643e17998/cluster

{
  "instances": [
    {
      "id": "dfbbd9ca-b5e1-4028-adb7-f78643e17998",
    },
    {
      "id": "061aaf4c-3a57-411e-9df9-2d0f813db859",
    }
  ]
}

Notes:

By omitting a member{} for id=3a72ee87-cf3e-40f1-a1e1-fe8c7263a782 in a PUT operation, this indicates the member should be removed from the replica-set.
It's possible that one might want to modify the 'priority', 'hidden', 'votes', etc. fields of the remaining members while dropping a member. So although the example above does not show it, mongodb{} can be included in a member to indicate other changes, *BUT*, since it's a PUT the expectation of what happens to omitted fields in mongodb{} becomes unclear.

Summary: Not very clean, mildly confusing, and very error-prone (nowhere is a "remove" action ever explicitly implied).

Option #2: POST /instances/:id/cluster/action

Request:

PUT /instances/dfbbd9ca-b5e1-4028-adb7-f78643e17998/cluster

{
  "remove_member": {
    "id": "dfbbd9ca-b5e1-4028-adb7-f78643e17998"
  }
}

Notes:

The 'remove_member' action is explicit here, vs. implicit as seen in the prior example.
'remove_member' has a strict set of fields that are supported, so there is no question as to what can be provided and what will be honored (as compared to Option #1).
Note: Could rename remove_member here to 'promote'!

Summary: Fairly clean, with no real drawbacks.

Option #3: POST /instances/:id/cluster/remove

POST /instances/dfbbd9ca-b5e1-4028-adb7-f78643e17998/cluster/remove

{
  "id": "dfbbd9ca-b5e1-4028-adb7-f78643e17998"
}

Notes:

Differs from Option #2 in that the action is in the URI vs. the payload (which is considered incorrect in REST due to it being a verb)
One drawback of this approach is that not every action will be supported across all datastores. So for example, a POST /instances/:id/cluster/changeoplogsize (http://docs.mongodb.org/manual/tutorial/change-oplog-size/) makes absolutely no sense to any datastore other than MongoDB.

Summary: At first glance it might look cleaner than Option #2 from a payload-perspective, but the URI discoverability and expansion is awful.

Option #4: POST /instances/:id/action

POST /instances/dfbbd9ca-b5e1-4028-adb7-f78643e17998/action

{
  "join": false
}

Notes:

Executed against the instance you wish to remove itself from the cluster, so providing the 'id' in the payload is unnecessary.
Drawback: There are actions that are replica-set-wide (or against a subset of the replica-set), meaning Option #1 or #2 or #3 would have to co-exist with this option anyway.
Drawback: Increase the number of ways to accomplish the same thing (could unjoin against /instances/:id/action, or against /instances/:id/cluster)

Summary: For this very specific example it looks great, but isn't expressive enough for other actions.

Option #5: POST /instances/:id/cluster/:id/action

POST /instances/dfbbd9ca-b5e1-4028-adb7-f78643e17998/cluster/dfbbd9ca-b5e1-4028-adb7-f78643e17998/action

{
  "remove": {}
}

Notes:

The /instances/:id is an arbitrary member in the replica-set, it doesn't matter which one; the cluster/:id is then a member of said replica-set that this action will be applied to.
Executed against the instance you wish to remove itself from the cluster, so providing the 'id' in the payload is unnecessary.
Needs More Thought: Could conceivably allow only specific operations here (like remove/unjoin), but not others that could be accomplished in a PATCH against /instances/:id/cluster (like 'priority', 'hidden', etc.)

Summary: Fairly clean with no real drawbacks.

@@ Line 56: / Line 56: @@
 ==== Create Slave ====
 <br>
+--[[User:Vipuls|vipuls]] ([[User talk:Vipuls|talk]]) 00:38, 25 April 2014 (UTC) I'm not a big fan of introducing the 'cluster' attribute to wrap what is a read-only slave.
 Request:
@@ Line 112: / Line 113: @@
 * Note: The master could have optionally included instance.cluster{} to provide a cluster_name to avoid it being the same as the master's name (see below for context)
 <br>
 ==== Show Instance ====
 <br>

Difference between revisions of "Trove/Replication-And-Clustering"

Revision as of 00:38, 25 April 2014

Contents

Summary/Viewpoints/Strategy

MySQL Master/Slave

Create Master

Create Slave

Show Instance

Show Cluster

Remove Replication (aka "Promote" to Standalone)

MongoDB

Create Single Instance Replica-Set

Create 3 Instance Replica-Set

Add Member to Replica-Set

Add Another Member to Replica-Set

Show Instance

Show Cluster

Add Arbiter

Add a Delayed Member

Modifying a Replica-Set

Remove a Member

MongoDB TokuMX

Cassandra

Create Cluster

Add Instance to Cluster

Add Another Instance to Cluster

Show Instance

Show Cluster

Modifying a Cluster

Remove an Instance

Couchbase

Create Cluster

Add Instance to Cluster

Add Another Instance to Cluster

Show Instance

Show Cluster

Modifying a Cluster

Remove a Instance

Redis

Create Master

Add Slave

Show Instance

Show Cluster

Promote/Disconnect Slave

Redis Cluster

Create Cluster

Add Another Master to Cluster

Add Slave to Cluster

Show Instance

Show Cluster

Promote/Disconnect Slave

Summary

Summary of cluster{} Fields

Summary of Route Changes

Open Questions

Discussion Points

Replica-Set Member Type/Attribute Updates

Remove a Member from a Replica-Set