Jump to: navigation, search

Trove-Replication-And-Clustering-API

Revision as of 23:12, 17 November 2014 by Amcrn (talk | contribs) (Note)

This Wiki Is Outdated

11/2014: This proposal has been superseded and is kept around for historical purposes. To see the true Clustering specification, visit https://blueprints.launchpad.net/trove/+spec/clustering

Instance Object Changes

Update: This is an out-of-date proposal that's kept around for historical purposes. To see the true Clustering specification, visit https://blueprints.launchpad.net/trove/+spec/clustering

Part of this proposal includes adding new properties to the json structure of an instance. The first would be to add a "nodes" property which would be list of refs back to the nova instance(s) that make up the trove instance.

The second would be a metadata storage that is of an open format to be used for storing critical data that is pertinent to the trove instance.

INSTANCES

Nodes

Because trove instances are a service type with an underlying nova resource(s) we can better display the nova resource(s) that make up the instance of a service type. When looking at a singular trove instance of something like redis or mysql the knowledge of the instance resource isn't all that important. Currently trove knows about the instance from a nova perspective but masks most of this information. This would look something like this:

GET /instances/{id}

{
    "instance": {
        "created": "2013-05-08T22:43:34",
        "hostname": "mysweetdb.example.com",
        "id": "b142d7fe-2174-4bb8-ba02-c2ffd49d2e33",
        "name": "json_instance",
        "status": "ACTIVE",
        "updated": "2013-05-08T22:43:34",
        "service_type": "mysql-5.1",
        "links": [
            {
                "href": "https://service/v1.0/1234/instances/b142d7fe-2174-4bb8-ba02-c2ffd49d2e33", 
                "rel": "self"
            }, 
            {
                "href": "https://service/instances/b142d7fe-2174-4bb8-ba02-c2ffd49d2e33", 
                "rel": "bookmark"
            }
        ],
        "nodes": [
        	{
                "id": "dcc5c518-73c7-4471-83e1-15fae67a98eb",
                "hostname": "master-n01.example.com",
                "name": "master-n01",
        		"flavor": {
            		"id": "1", 
            		"links": [...]
        		},
        		"links": [
            		{
                		"href": "https://service/v1.0/1234/instances/{instance_id}/nodes/dcc5c518-73c7-4471-83e1-15fae67a98eb", 
                		"rel": "self"
            		}, 
            		{
                		"href": "https://service/instances/{instance_id}/nodes/dcc5c518-73c7-4471-83e1-15fae67a98eb", 
                		"rel": "bookmark"
            		}
        		],
        		"volume": {
            		"size": 2,
            		"used": 0.16368598397821188
        		},
        	},
        ],
    }
}

Metadata

The metadata looks much like our previous proposal. It contains critical information to the configuration or topology of the cluster or replication agreement. This should look something like this:

This would be an example of a 3 node mult-master mysql replication contract GET /instances/{id}

{
    "instance": {
        "created": "2013-05-08T22:43:34",
        "id": "cfeccbf4-ac5b-494a-99b7-61593a6a71b0",
        "name": "multi-master-hotness",
        "status": "ACTIVE",
        "updated": "2013-05-08T22:43:34",
        "service_type": "mysql-5.5",
        "links": [
            {
                "href": "https://service/v1.0/1234/instances/cfeccbf4-ac5b-494a-99b7-61593a6a71b0", 
                "rel": "self"
            }, 
            {
                "href": "https://service/instances/cfeccbf4-ac5b-494a-99b7-61593a6a71b0", 
                "rel": "bookmark"
            }
        ],
        "nodes": [
        	{
        		"id": "dcc5c518-73c7-4471-83e1-15fae67a98eb",
        		"hostname": "master-n01.example.com",
                "name": "master-n01",
        		"flavor": {
            		"id": "1", 
            		"links": [...]
        		},
        		"links": [
            		{
                		"href": "https://service/v1.0/1234/instances/{instance_id}/nodes/dcc5c518-73c7-4471-83e1-15fae67a98eb", 
                		"rel": "self"
            		}, 
            		{
                		"href": "https://service/instances/{instance_id}/nodes/dcc5c518-73c7-4471-83e1-15fae67a98eb", 
                		"rel": "bookmark"
            		}
        		],
        		"volume": {
            		"size": 2,
            		"used": 0.16368598397821188
        		},
        	},
        	{
        		"id": "4530736b-c1ff-4c07-9a9f-fa9bca418157",
        		"hostname": "master-n02.example.com",
                "name": "master-n02",
        		"flavor": {
            		"id": "1", 
            		"links": [...]
        		},
        		"links": [
            		{
                		"href": "https://service/v1.0/1234/instances/{instance_id}/nodes/4530736b-c1ff-4c07-9a9f-fa9bca418157", 
                		"rel": "self"
            		}, 
            		{
                		"href": "https://service/instances/{instance_id}/nodes/4530736b-c1ff-4c07-9a9f-fa9bca418157", 
                		"rel": "bookmark"
            		}
        		],
        		"volume": {
            		"size": 2,
            		"used": 0.16368598397821188
        		},
        	},
        	{
        		"id": "736b9820-7e91-4775-84b0-78e71d60ce4c",
        		"hostname": "master-n03.example.com",
                "name": "master-n03",
        		"flavor": {
            		"id": "1", 
            		"links": [...]
        		},
        		"links": [
            		{
                		"href": "https://service/v1.0/1234/instances/{instance_id}/nodes/736b9820-7e91-4775-84b0-78e71d60ce4c", 
                		"rel": "self"
            		}, 
            		{
                		"href": "https://service/instances/{instance_id}/nodes/736b9820-7e91-4775-84b0-78e71d60ce4c", 
                		"rel": "bookmark"
            		}
        		],
        		"volume": {
            		"size": 2,
            		"used": 0.16368598397821188
        		},
        	},
        ],
        "metadata": {
        	"replication_contract": {
        		// We should use something more clear than just nova instance
        		// ids here but lets just use them for now for example. Maybe
        		// links? names? not sure what works best yet.
        		"dcc5c518-73c7-4471-83e1-15fae67a98eb": {
        			"replicates_to": [
        				"4530736b-c1ff-4c07-9a9f-fa9bca418157",
        			],
        			"replicates_from": [
        				"736b9820-7e91-4775-84b0-78e71d60ce4c",
        			],
        			"writeable": true,
        		},
        		"4530736b-c1ff-4c07-9a9f-fa9bca418157": {
        			"replicates_to": [
        				"736b9820-7e91-4775-84b0-78e71d60ce4c",
        			],
        			"replicates_from": [
        				"dcc5c518-73c7-4471-83e1-15fae67a98eb",
        			],
        			"writeable": true,
        		},
        		"736b9820-7e91-4775-84b0-78e71d60ce4c": {
        			"replicates_to": [
        				"dcc5c518-73c7-4471-83e1-15fae67a98eb",
        			],
        			"replicates_from": [
        				"4530736b-c1ff-4c07-9a9f-fa9bca418157",
        			],
        			"writeable": true
        		}
        	}
        }
    }
}

Other Examples

3 Node multi-master without nesting instances

Instance #1 (Master)

{
    "instance": {
        "created": "2013-05-08T22:43:34",
        "id": "cfeccbf4-ac5b-494a-99b7-61593a6a71b0",
        "name": "multi-master-hotness-n01",
        "status": "ACTIVE",
        "updated": "2013-05-08T22:43:34",
        "service_type": "mysql-5.5",
        "links": [
            {
                "href": "https://service/v1.0/1234/instances/cfeccbf4-ac5b-494a-99b7-61593a6a71b0", 
                "rel": "self"
            }, 
            {
                "href": "https://service/instances/cfeccbf4-ac5b-494a-99b7-61593a6a71b0", 
                "rel": "bookmark"
            }
        ],
        "nodes": [
        	{
        		"id": "dcc5c518-73c7-4471-83e1-15fae67a98eb",
        		"hostname": "master-n01.example.com",
                "name": "master-n01",
        		"flavor": {
            		"id": "1", 
            		"links": [...]
        		},
        		"links": [
            		{
                		"href": "https://service/v1.0/1234/instances/{instance_id}/nodes/dcc5c518-73c7-4471-83e1-15fae67a98eb", 
                		"rel": "self"
            		}, 
            		{
                		"href": "https://service/instances/{instance_id}/nodes/dcc5c518-73c7-4471-83e1-15fae67a98eb", 
                		"rel": "bookmark"
            		}
        		],
        		"volume": {
            		"size": 2,
            		"used": 0.16368598397821188
        		},
        	},
        ],
        "metadata": {
        	"replication_contract": {
        		// We should use something more clear than just nova instance
        		// ids here but lets just use them for now for example. Maybe
        		// links? names? not sure what works best yet.
        		"dcc5c518-73c7-4471-83e1-15fae67a98eb": {
        			"replicates_to": [
        				"4530736b-c1ff-4c07-9a9f-fa9bca418157",
        			],
        			"replicates_from": [
        				"736b9820-7e91-4775-84b0-78e71d60ce4c",
        			],
        			"writeable": true,
        		},
        	}
        }
    }
}

Instance #2 (Master)

{
    "instance": {
        "created": "2013-05-08T22:43:34",
        "id": "fdff9fa2-abe3-4536-94dc-887a1bba4274",
        "name": "multi-master-hotness-n02",
        "status": "ACTIVE",
        "updated": "2013-05-08T22:43:34",
        "service_type": "mysql-5.5",
        "links": [
            {
                "href": "https://service/v1.0/1234/instances/fdff9fa2-abe3-4536-94dc-887a1bba4274", 
                "rel": "self"
            }, 
            {
                "href": "https://service/instances/fdff9fa2-abe3-4536-94dc-887a1bba4274", 
                "rel": "bookmark"
            }
        ],
        "nodes": [
        	{
        		"id": "4530736b-c1ff-4c07-9a9f-fa9bca418157",
        		"hostname": "master-n02.example.com",
                "name": "master-n02",
        		"flavor": {
            		"id": "1", 
            		"links": [...]
        		},
        		"links": [
            		{
                		"href": "https://service/v1.0/1234/instances/{instance_id}/nodes/4530736b-c1ff-4c07-9a9f-fa9bca418157", 
                		"rel": "self"
            		}, 
            		{
                		"href": "https://service/instances/{instance_id}/nodes/4530736b-c1ff-4c07-9a9f-fa9bca418157", 
                		"rel": "bookmark"
            		}
        		],
        		"volume": {
            		"size": 2,
            		"used": 0.16368598397821188
        		},
        	},
        ],
        "metadata": {
        	"replication_contract": {
        		// We should use something more clear than just nova instance
        		// ids here but lets just use them for now for example. Maybe
        		// links? names? not sure what works best yet.
        		"4530736b-c1ff-4c07-9a9f-fa9bca418157": {
        			"replicates_to": [
        				"736b9820-7e91-4775-84b0-78e71d60ce4c",
        			],
        			"replicates_from": [
        				"dcc5c518-73c7-4471-83e1-15fae67a98eb",
        			],
        			"writeable": true,
        		},
        	}
        }
    }
}

Instance #3 (Master)

{
    "instance": {
        "created": "2013-05-08T22:43:34",
        "id": "e97cffc3-e91c-4526-987e-711fa557891a",
        "name": "multi-master-hotness-n03",
        "status": "ACTIVE",
        "updated": "2013-05-08T22:43:34",
        "service_type": "mysql-5.5",
        "links": [
            {
                "href": "https://service/v1.0/1234/instances/e97cffc3-e91c-4526-987e-711fa557891a", 
                "rel": "self"
            }, 
            {
                "href": "https://service/instances/e97cffc3-e91c-4526-987e-711fa557891a", 
                "rel": "bookmark"
            }
        ],
        "nodes": [
        	{
        		"id": "736b9820-7e91-4775-84b0-78e71d60ce4c",
        		"hostname": "master-n03.example.com",
                "name": "master-n03",
        		"flavor": {
            		"id": "1", 
            		"links": [...]
        		},
        		"links": [
            		{
                		"href": "https://service/v1.0/1234/instances/{instance_id}/nodes/736b9820-7e91-4775-84b0-78e71d60ce4c", 
                		"rel": "self"
            		}, 
            		{
                		"href": "https://service/instances/{instance_id}/nodes/736b9820-7e91-4775-84b0-78e71d60ce4c", 
                		"rel": "bookmark"
            		}
        		],
        		"volume": {
            		"size": 2,
            		"used": 0.16368598397821188
        		},
        	},
        ],
        "metadata": {
        	"replication_contract": {
        		// We should use something more clear than just nova instance
        		// ids here but lets just use them for now for example. Maybe
        		// links? names? not sure what works best yet.
        		"736b9820-7e91-4775-84b0-78e71d60ce4c": {
        			"replicates_to": [
        				"dcc5c518-73c7-4471-83e1-15fae67a98eb",
        			],
        			"replicates_from": [
        				"4530736b-c1ff-4c07-9a9f-fa9bca418157",
        			],
        			"writeable": true
        		}
        	}
        }
    }
}

Replication without nesting slave instances with their masters

Instance #1 (Master)

{
    "instance": {
        "created": "2013-05-08T22:43:34",
        "id": "cfeccbf4-ac5b-494a-99b7-61593a6a71b0",
        "name": "master-slave-sweetness-n01",
        "status": "ACTIVE",
        "updated": "2013-05-08T22:43:34",
        "service_type": "mysql-5.5",
        "links": [
            {
                "href": "https://service/v1.0/1234/instances/cfeccbf4-ac5b-494a-99b7-61593a6a71b0", 
                "rel": "self"
            }, 
            {
                "href": "https://service/instances/cfeccbf4-ac5b-494a-99b7-61593a6a71b0", 
                "rel": "bookmark"
            }
        ],
        "nodes": [
        	{
        		"id": "dcc5c518-73c7-4471-83e1-15fae67a98eb",
        		"hostname": "master-n01.example.com",
                "name": "master-n01",
        		"flavor": {
            		"id": "1", 
            		"links": [...]
        		},
        		"links": [
            		{
                		"href": "https://service/v1.0/1234/instances/{instance_id}/nodes/dcc5c518-73c7-4471-83e1-15fae67a98eb", 
                		"rel": "self"
            		}, 
            		{
                		"href": "https://service/instances/{instance_id}/nodes/dcc5c518-73c7-4471-83e1-15fae67a98eb", 
                		"rel": "bookmark"
            		}
        		],
        		"volume": {
            		"size": 2,
            		"used": 0.16368598397821188
        		},
        	},
        ],
        "metadata": {
        	"replication_contract": {
        		// We should use something more clear than just nova instance
        		// ids here but lets just use them for now for example. Maybe
        		// links? names? not sure what works best yet.
        		"dcc5c518-73c7-4471-83e1-15fae67a98eb": {
        			"replicates_to": [
        				"4530736b-c1ff-4c07-9a9f-fa9bca418157",
        			],
        			"writeable": true,
        		},
        	}
        }
    }
}

Instance #2 (Read-only Slave)

{
    "instance": {
        "created": "2013-05-08T22:43:34",
        "id": "fdff9fa2-abe3-4536-94dc-887a1bba4274",
        "name": "master-slave-sweetness-n02",
        "status": "ACTIVE",
        "updated": "2013-05-08T22:43:34",
        "service_type": "mysql-5.5",
        "links": [
            {
                "href": "https://service/v1.0/1234/instances/fdff9fa2-abe3-4536-94dc-887a1bba4274", 
                "rel": "self"
            }, 
            {
                "href": "https://service/instances/fdff9fa2-abe3-4536-94dc-887a1bba4274", 
                "rel": "bookmark"
            }
        ],
        "nodes": [
        	{
        		"id": "4530736b-c1ff-4c07-9a9f-fa9bca418157",
        		"hostname": "master-n02.example.com",
                "name": "master-n02",
        		"flavor": {
            		"id": "1", 
            		"links": [...]
        		},
        		"links": [
            		{
                		"href": "https://service/v1.0/1234/instances/{instance_id}/nodes/4530736b-c1ff-4c07-9a9f-fa9bca418157", 
                		"rel": "self"
            		}, 
            		{
                		"href": "https://service/instances/{instance_id}/nodes/4530736b-c1ff-4c07-9a9f-fa9bca418157", 
                		"rel": "bookmark"
            		}
        		],
        		"volume": {
            		"size": 2,
            		"used": 0.16368598397821188
        		},
        	},
        ],
        "metadata": {
        	"replication_contract": {
        		// We should use something more clear than just nova instance
        		// ids here but lets just use them for now for example. Maybe
        		// links? names? not sure what works best yet.
        		"4530736b-c1ff-4c07-9a9f-fa9bca418157": {
        			"replicates_from": [
        				"dcc5c518-73c7-4471-83e1-15fae67a98eb",
        			],
        			"writeable": false,
        		},
        	}
        }
    }
}

Clustering and Replication

A cluster is an independent service type.

  • This simplifies the configurations (ties to service_type)
  • The guest impl can be different for each service_type
  • We don't need cluster type and service_type when defining a cluster


Think: A instance is an instance is a cluster is an instance is NOT replication

CLUSTERING

Definition of cluster for purposes of Trove:
A cluster must provide

  • High Availablility
  • Fault Tolerance


A cluster object is represented as an instance with a metadata attribute containing node information. A node is a sub-resource of a cluster and has a limited number of actions that can be performed on them. A cluster is made of of equal sized nodes using all the same flavor/disk size.

Cluster Operations

Create cluster:

Create a cluster with flavor X and size Y. All nodes are of equal size.
POST /instances

{
	"name": "foobar",
	"flavor": "{flavor_id}",
	"service_type": "{service_type_id}",
	"volume": {
		"size": 50,
	},
	"metadata": {
		"nodes": 5,
		"valuable_cluster_info": {}
	}
}

Delete cluster:

  • Deletes all nodes in the cluster and the cluster itself.

DELETE /instances/{id} { "empty body?" }

Downsize a cluster (Delete nodes):

PATCH /instances/{id}

{
	"metadata": {
		"nodes": 3
	}
}

Or remove specific nodes from the cluster. e.g. You have a node acting up in the cluster and you want to pull it out PATCH /instances/{id}

{
	"nodes": [
		{"id": "32e3c3b8-8d71-4b76-b2be-27fec62bc302"},
		{"id": "3c6176fc-bb33-429f-a926-e8307c115752"},
	]
}

Restart a cluster:

POST /instances/{id}/restart

{
	"empty body?"
}

Resize a cluster:

POST /instances/{id}/resize

{
	"flavor": "flavor_id",
	"volume": {
		"size": 100
	}
}

Initialize cluster:

Clusters can get mucked up sometimes, you may want to keep a node in the cluster but wipe it's data and let the cluster technology repopulate it's data. POST /instance/{id}/initialize

{
	"nodes": [
		{"id": "32e3c3b8-8d71-4b76-b2be-27fec62bc302"},
		{"id": "3c6176fc-bb33-429f-a926-e8307c115752"},
	]
}

Rebalance/restripe cluster:

Changing the hashing algorithm on some cluster technologies may require a rebalance or restriping of the data. This is a maintenance method to support that. POST /instance/{id}/rebalance

{
	"nodes": [
		{"id": "32e3c3b8-8d71-4b76-b2be-27fec62bc302"},
		{"id": "3c6176fc-bb33-429f-a926-e8307c115752"},
	]
}

Node Operations:

Restart a node:

POST /instance/{id}/node/{id}/restart

{
	"empty body?"
}

Initialize a node:

Clusters can get mucked up sometimes, you may want to keep a node in the cluster but wipe it's data and let the cluster technology repopulate it's data. POST /instance/{id}/node/{id}/initialize

{
	"empty body?"
}

Rebalance/restripe cluster:

Changing the hashing algorithm on some cluster technologies may require a rebalance or restriping of the data. This is a maintenance method to support that. POST /instance/{id}/node/{id}/rebalance

{
	"empty body?"
}


REPLICATION

The implementation of replication varies from datastore type to datastore type. There has been some discussion around making a clear distinction of what Trove considers replication and what is considered clustering. When that distinction is made it will live at the beginning of this document. In the meantime we will work under the premise that replication means that the data from one datastore instance is replicated in part or in whole to another datastore instance of the same type with no inherent exclusivity implied. This may or may not be true for all datastores but it is definitely true for some so until we decide otherwise we'll assume we have to tailor to the lowest common denominator.

The replication contract will reside in the instance metadata and will have clearly defined verbiage that is allowed. Extraneous verbiage should be rejected when the call to create or modify is made. The validator should hook into the Instance class' create and future modify methods to validate the input of metadata before the instance is created.

Think of replication as a capability of some service types. Not every datastore type will have this capability and it should be configurable via the datastore types capabilities feature.

What you cannot do:

  • create a master and slave in one call

- one call per action means simplified workflow

Replication Contract

The replication contract is a structure contained within instance metadata that clearly defines a replication topology with regard to the queried instance's involvement in replication. An instance's replication contract should not contain information pertaining to parallel instances that it does not directly replicate to or from. e.g. You have a redis master server and it has two slaves. Each slave's replication contract would clearly indicate it is a slave of the master instance but would never have a reference to the other slave. This keeps the metadata simple and allows us to reduce the risk of having an orphaned entry in an instance's replication contract.

Replication Contract Verbiage

replicates_to

This term will be used in an instance that has slaves of itself. It is a clear indicator that while it may also be a slave, covered next, it is definitely a master of some.

replicates_from

This term will be used in an instance that is a slave of another instance. It is a clear indicator that while it may also be a master, covered previously, it is definitely a slave of another instance.

Example: You have an existing single instance with existing data. Your scaling needs now require you to add a slave. When creating the slave you would use the "replicates_from" keyword to indicate that this instance that is being created should be created with the understanding that it will become a slave of the instance indicated in the "replicates_from" list. A part of creating or satisfying the replication setup would be to populate the old running instance with a replication contract that indicates it "replicates_to" the new instance. This allows a user to programatically generate a replication topology solely from the api data by querying the api for each instance id listed in the replication contract.

writable

This term will be used in an instance to indicate whether it is intended to be used for writes. As replication is used commonly to scale read operations it is very common to have a read-only slave in many datastore types. It is beneficial to the user to be able to see this information when viewing the instance details.

Replication workflow:

  1. Create an instance
  2. Create an Nth instance with
  • Caveat: adding a replicated slave will alter data on the master instance and will use the agent and/or heat to create the replication.
{
	"metadata": {
            "replication_contract": {
                "replicates_from": [
			"<master_instance_id>"
		],
		// The writable flag should actually not be required but would populate based on
		// the underlying service type's replication setup.  It is an informative flag only
		// and doesn't have to block users.  It probably can be argued that if the underlying
                // datastore supports setting a slave as read only to prevent writes that this flag
                // should be honored.
		"writable": false 
	}
}


Replication Examples

Examples of what data returned back to the user when they query detailed instance info from the api with: GET /instances/<instance_id>

Master Instance

{
    "name": "blah",
    "id": "5c489b9d-96bb-4e15-a79d-412d88a0062e", etc...
    "metadata": {
        "replication_contract": {
            "replicates_to": [
                "7ee9664f-1261-4076-8abc-21af3a092132",
            ],
            "writable": true
        }
    }
}

Slave Instance (Read-Only)

{
    "name": "blah-slave",
    "id": "7ee9664f-1261-4076-8abc-21af3a092132", etc...
    "metadata": {
        "replication_contract": {
            "replicates_from": [
                "5c489b9d-96bb-4e15-a79d-412d88a0062e",
            ],
            "writable": false
        }
    }
}

Replication Operations

Create Replication:

POST /instance

{
	"name": "foobar",
	"flavor": "{flavor_id}",
	"service_type": "mysql-5.1",
	"size": 50,
	"metadata": {
		"replication_contract": {
			"replicates_from": [
				"32e3c3b8-8d71-4b76-b2be-27fec62bc302",
			]
		}
	}
}


New Wiki Page

Moving to a cleaner spec format per trove meetup. Duplicate data will live here from above until it's been all merged over.

    1. Description

> Describe the purpose of the enhancement

    1. Justification/Benefits

> What is the driving force behind this change?

> Does it allow for great flexibility? Stability? Security?

    1. Impacts
      1. Configuration

> Does this impact any configuration files? If so, which ones?

      1. Database

> Does this impact any existing tables? If so, which ones?

> Are the changes forward and backward compatible?

> Be sure to include the expected migration process

      1. Public API

> Does this change any API that an end-user has access to?

> Are there any exceptions in terms of consistency with other APIs?

      1. Internal API

> Does this change any internal messages between API and Task Manager or Task Manager to Guest

      1. Guest Agent

> Does this change behavior on the Guest Agent? If so, is it backwards compatible with API and Task Manager?

Comments: