Difference between revisions of "Trove/Specs/Trove-v1-MySQL-Replication"

Revision as of 21:49, 21 April 2014

Description

Providing support for the various replication use cases is critical for use of Trove in production. For the first phase implementation of Replication in Trove we will implement the functionality laid out in the Trove V1 Replication Blueprint

Use Case Summary

The following use cases will be addressed by this V1 implementation:

A. Read Replicas (Slaves)

The master can exist before the slave such that the master already contains data
N Slaves for one master
Slaves can be marked read-only (read-only will be default)
A slave can be detached from "replication set" to act as independent site
A pre-existing non-replication site can become the master of a new "replication set"
The health of a slave will be monitor-able

Design

Trove API

Create Slaves

POST /instances/{id}/replicate

{
    "count": 2,
    "read_only": "True"
}

Stop Replication

POST /instances/{id}/detach

{
    "empty body?"
}

Python-Troveclient

trove replicate <master instance> <slave count> --read-only=<boolean>

trove detach_replication <slave instance>

Taskmanager

The taskmanager will implement 2 API calls:

createReplication(master=<instance>, slave_count=<N>, read_only=true)
detachReplication(slave=<instance>)

taskmanager.createReplication

The Create Replication task will be performed with the following steps:

Execute getReplicationSnapshot() on the master site, receiving "master snapshot results metadata"
N times:
1. Create trove instance
2. execute createReplicationSlave() on new instance
3. Update instance metadata to add "replication_contract" section
delete replication snapshot from Swift

Trove GuestAgent

There will be 3 new methods added to the guestagent API:

getReplicationSnapshot()
attachReplicationSlave()
detachReplicationSlave()

Replication will be focused around a replication snapshot. This snapshot will contain the data necessary to set up a slave to replicate from the site which created the snapshot, typically a URI to the user's data set stored in Swift plus the metadata required to coordinate replication.

Each datastore implementation will need to implement these methods. The content of the image uploaded to swift is opaque to the taskmanager and higher components, so the guest agent is free to store whatever data it chooses, in whichever format is most appropriate. The content of the metadata is specific to the datastore, but will be represented as a JSON object.

Trove Guestagent - MySQL Datastore Implementation

getReplicationSnapshot()

The MySQL guestagent will use xtrabackup to create a backup of the user's data and upload it to Swift. The metadata will include a URI of the uploaded backup data, along with the site's binlog position and network information required to set up replication.

{
    "master": {
        "host": "192.168.0.1",
        "port": 3306
    },
    "dataset": {
        "datastore": "mysql",
        "datastore_version": "mysql-5.5",
        "dataset_size": 2,
        "snapshot_href": "http://..."
    },
    "binlog_position": <binlog position>
}

@@ Line 17: / Line 17: @@
 === Trove API ===
+==== Create Slaves ====
+'''POST''' /instances/{id}/replicate
+<pre>
+{
+    "count": 2,
+    "read_only": "True"
+}
+</pre>
+==== Stop Replication ====
+'''POST''' /instances/{id}/detach
+<pre>
+{
+    "empty body?"
+}
+</pre>
 === Python-Troveclient ===
+trove replicate <master instance> <slave count>  --read-only=<boolean>
+trove detach_replication <slave instance>
 === Taskmanager ===
@@ Line 24: / Line 46: @@
 The taskmanager will implement 2 API calls:
-* createReplication(master=<instance>, slave_count=<N>)
+* createReplication(master=<instance>, slave_count=<N>, read_only=true)
 * detachReplication(slave=<instance>)
-==== taskmanage.createReplication ====
+==== taskmanager.createReplication ====
 The Create Replication task will be performed with the following steps:
@@ Line 35: / Line 57: @@
 ## Create trove instance
 ## execute createReplicationSlave() on new instance
+## Update instance metadata to add "replication_contract" section
 # delete replication snapshot from Swift
@@ Line 42: / Line 65: @@
 * getReplicationSnapshot()
-* createReplicationSlave()
+* attachReplicationSlave()
 * detachReplicationSlave()
 Replication will be focused around a replication snapshot.  This snapshot will contain the data necessary to set up a slave to replicate from the site which created the snapshot, typically a URI to the user's data set stored in Swift plus the metadata required to coordinate replication.
-Each datastore implementation will need to implement these methods.   The content of the image uploaded to swift is opaque to the taskmanager and higher components, so the guest agent is free to store whatever data it chooses, in whichever format is most appropriate.  The content of the metadata is specific to the datastore, but will be stored as a JSON object.
+Each datastore implementation will need to implement these methods.   The content of the image uploaded to swift is opaque to the taskmanager and higher components, so the guest agent is free to store whatever data it chooses, in whichever format is most appropriate.  The content of the metadata is specific to the datastore, but will be represented as a JSON object.
 ==== Trove Guestagent - MySQL Datastore Implementation ====
@@ Line 55: / Line 78: @@
 The MySQL guestagent will use xtrabackup to create a backup of the user's data and upload it to Swift.  The metadata will include a URI of the uploaded backup data, along with the site's binlog position and network information required to set up replication.
+<pre>
 {
      "master": {
@@ Line 63: / Line 87: @@
          "datastore": "mysql",
          "datastore_version": "mysql-5.5",
-         "dataset_size": 2
+         "dataset_size": 2,
-         "snapshot_href": "http://...",
+         "snapshot_href": "http://..."
      },
      "binlog_position": <binlog position>
 }
+</pre>