Jump to: navigation, search

Trove-Replication-And-Clustering-API

Revision as of 21:42, 5 February 2014 by Imsplitbit (talk | contribs) (REPLICATION)

Instance Object Changes

Part of this proposal includes adding new properties to the json structure of an instance. The first would be to add a "nodes" property which would be list of refs back to the nova instance(s) that make up the trove instance.

The second would be a metadata storage that is of an open format to be used for storing critical data that is pertinent to the trove instance.

INSTANCES

Nodes

Because trove instances are a service type with an underlying nova resource(s) we can better display the nova resource(s) that make up the instance of a service type. When looking at a singular trove instance of something like redis or mysql the knowledge of the instance resource isn't all that important. Currently trove knows about the instance from a nova perspective but masks most of this information. This would look something like this:

GET /instances/{id}

{
    "instance": {
        "created": "2013-05-08T22:43:34",
        "hostname": "mysweetdb.example.com",
        "id": "b142d7fe-2174-4bb8-ba02-c2ffd49d2e33",
        "name": "json_instance",
        "status": "ACTIVE",
        "updated": "2013-05-08T22:43:34",
        "service_type": "mysql-5.1",
        "links": [
            {
                "href": "https://service/v1.0/1234/instances/b142d7fe-2174-4bb8-ba02-c2ffd49d2e33", 
                "rel": "self"
            }, 
            {
                "href": "https://service/instances/b142d7fe-2174-4bb8-ba02-c2ffd49d2e33", 
                "rel": "bookmark"
            }
        ],
        "nodes": [
        	{
                "id": "dcc5c518-73c7-4471-83e1-15fae67a98eb",
                "hostname": "master-n01.example.com",
                "name": "master-n01",
        		"flavor": {
            		"id": "1", 
            		"links": [...]
        		},
        		"links": [
            		{
                		"href": "https://service/v1.0/1234/instances/{instance_id}/nodes/dcc5c518-73c7-4471-83e1-15fae67a98eb", 
                		"rel": "self"
            		}, 
            		{
                		"href": "https://service/instances/{instance_id}/nodes/dcc5c518-73c7-4471-83e1-15fae67a98eb", 
                		"rel": "bookmark"
            		}
        		],
        		"volume": {
            		"size": 2,
            		"used": 0.16368598397821188
        		},
        	},
        ],
    }
}

Metadata

The metadata looks much like our previous proposal. It contains critical information to the configuration or topology of the cluster or replication agreement. This should look something like this:

This would be an example of a 3 node mult-master mysql replication contract GET /instances/{id}

{
    "instance": {
        "created": "2013-05-08T22:43:34",
        "id": "cfeccbf4-ac5b-494a-99b7-61593a6a71b0",
        "name": "multi-master-hotness",
        "status": "ACTIVE",
        "updated": "2013-05-08T22:43:34",
        "service_type": "mysql-5.5",
        "links": [
            {
                "href": "https://service/v1.0/1234/instances/cfeccbf4-ac5b-494a-99b7-61593a6a71b0", 
                "rel": "self"
            }, 
            {
                "href": "https://service/instances/cfeccbf4-ac5b-494a-99b7-61593a6a71b0", 
                "rel": "bookmark"
            }
        ],
        "nodes": [
        	{
        		"id": "dcc5c518-73c7-4471-83e1-15fae67a98eb",
        		"hostname": "master-n01.example.com",
                "name": "master-n01",
        		"flavor": {
            		"id": "1", 
            		"links": [...]
        		},
        		"links": [
            		{
                		"href": "https://service/v1.0/1234/instances/{instance_id}/nodes/dcc5c518-73c7-4471-83e1-15fae67a98eb", 
                		"rel": "self"
            		}, 
            		{
                		"href": "https://service/instances/{instance_id}/nodes/dcc5c518-73c7-4471-83e1-15fae67a98eb", 
                		"rel": "bookmark"
            		}
        		],
        		"volume": {
            		"size": 2,
            		"used": 0.16368598397821188
        		},
        	},
        	{
        		"id": "4530736b-c1ff-4c07-9a9f-fa9bca418157",
        		"hostname": "master-n02.example.com",
                "name": "master-n02",
        		"flavor": {
            		"id": "1", 
            		"links": [...]
        		},
        		"links": [
            		{
                		"href": "https://service/v1.0/1234/instances/{instance_id}/nodes/4530736b-c1ff-4c07-9a9f-fa9bca418157", 
                		"rel": "self"
            		}, 
            		{
                		"href": "https://service/instances/{instance_id}/nodes/4530736b-c1ff-4c07-9a9f-fa9bca418157", 
                		"rel": "bookmark"
            		}
        		],
        		"volume": {
            		"size": 2,
            		"used": 0.16368598397821188
        		},
        	},
        	{
        		"id": "736b9820-7e91-4775-84b0-78e71d60ce4c",
        		"hostname": "master-n03.example.com",
                "name": "master-n03",
        		"flavor": {
            		"id": "1", 
            		"links": [...]
        		},
        		"links": [
            		{
                		"href": "https://service/v1.0/1234/instances/{instance_id}/nodes/736b9820-7e91-4775-84b0-78e71d60ce4c", 
                		"rel": "self"
            		}, 
            		{
                		"href": "https://service/instances/{instance_id}/nodes/736b9820-7e91-4775-84b0-78e71d60ce4c", 
                		"rel": "bookmark"
            		}
        		],
        		"volume": {
            		"size": 2,
            		"used": 0.16368598397821188
        		},
        	},
        ],
        "metadata": {
        	"replication_contract": {
        		// We should use something more clear than just nova instance
        		// ids here but lets just use them for now for example. Maybe
        		// links? names? not sure what works best yet.
        		"dcc5c518-73c7-4471-83e1-15fae67a98eb": {
        			"replicates_to": [
        				"4530736b-c1ff-4c07-9a9f-fa9bca418157",
        			],
        			"replicates_from": [
        				"736b9820-7e91-4775-84b0-78e71d60ce4c",
        			],
        			"writeable": true,
        		},
        		"4530736b-c1ff-4c07-9a9f-fa9bca418157": {
        			"replicates_to": [
        				"736b9820-7e91-4775-84b0-78e71d60ce4c",
        			],
        			"replicates_from": [
        				"dcc5c518-73c7-4471-83e1-15fae67a98eb",
        			],
        			"writeable": true,
        		},
        		"736b9820-7e91-4775-84b0-78e71d60ce4c": {
        			"replicates_to": [
        				"dcc5c518-73c7-4471-83e1-15fae67a98eb",
        			],
        			"replicates_from": [
        				"4530736b-c1ff-4c07-9a9f-fa9bca418157",
        			],
        			"writeable": true
        		}
        	}
        }
    }
}

Other Examples

3 Node multi-master without nesting instances

Instance #1 (Master)

{
    "instance": {
        "created": "2013-05-08T22:43:34",
        "id": "cfeccbf4-ac5b-494a-99b7-61593a6a71b0",
        "name": "multi-master-hotness-n01",
        "status": "ACTIVE",
        "updated": "2013-05-08T22:43:34",
        "service_type": "mysql-5.5",
        "links": [
            {
                "href": "https://service/v1.0/1234/instances/cfeccbf4-ac5b-494a-99b7-61593a6a71b0", 
                "rel": "self"
            }, 
            {
                "href": "https://service/instances/cfeccbf4-ac5b-494a-99b7-61593a6a71b0", 
                "rel": "bookmark"
            }
        ],
        "nodes": [
        	{
        		"id": "dcc5c518-73c7-4471-83e1-15fae67a98eb",
        		"hostname": "master-n01.example.com",
                "name": "master-n01",
        		"flavor": {
            		"id": "1", 
            		"links": [...]
        		},
        		"links": [
            		{
                		"href": "https://service/v1.0/1234/instances/{instance_id}/nodes/dcc5c518-73c7-4471-83e1-15fae67a98eb", 
                		"rel": "self"
            		}, 
            		{
                		"href": "https://service/instances/{instance_id}/nodes/dcc5c518-73c7-4471-83e1-15fae67a98eb", 
                		"rel": "bookmark"
            		}
        		],
        		"volume": {
            		"size": 2,
            		"used": 0.16368598397821188
        		},
        	},
        ],
        "metadata": {
        	"replication_contract": {
        		// We should use something more clear than just nova instance
        		// ids here but lets just use them for now for example. Maybe
        		// links? names? not sure what works best yet.
        		"dcc5c518-73c7-4471-83e1-15fae67a98eb": {
        			"replicates_to": [
        				"4530736b-c1ff-4c07-9a9f-fa9bca418157",
        			],
        			"replicates_from": [
        				"736b9820-7e91-4775-84b0-78e71d60ce4c",
        			],
        			"writeable": true,
        		},
        	}
        }
    }
}

Instance #2 (Master)

{
    "instance": {
        "created": "2013-05-08T22:43:34",
        "id": "fdff9fa2-abe3-4536-94dc-887a1bba4274",
        "name": "multi-master-hotness-n02",
        "status": "ACTIVE",
        "updated": "2013-05-08T22:43:34",
        "service_type": "mysql-5.5",
        "links": [
            {
                "href": "https://service/v1.0/1234/instances/fdff9fa2-abe3-4536-94dc-887a1bba4274", 
                "rel": "self"
            }, 
            {
                "href": "https://service/instances/fdff9fa2-abe3-4536-94dc-887a1bba4274", 
                "rel": "bookmark"
            }
        ],
        "nodes": [
        	{
        		"id": "4530736b-c1ff-4c07-9a9f-fa9bca418157",
        		"hostname": "master-n02.example.com",
                "name": "master-n02",
        		"flavor": {
            		"id": "1", 
            		"links": [...]
        		},
        		"links": [
            		{
                		"href": "https://service/v1.0/1234/instances/{instance_id}/nodes/4530736b-c1ff-4c07-9a9f-fa9bca418157", 
                		"rel": "self"
            		}, 
            		{
                		"href": "https://service/instances/{instance_id}/nodes/4530736b-c1ff-4c07-9a9f-fa9bca418157", 
                		"rel": "bookmark"
            		}
        		],
        		"volume": {
            		"size": 2,
            		"used": 0.16368598397821188
        		},
        	},
        ],
        "metadata": {
        	"replication_contract": {
        		// We should use something more clear than just nova instance
        		// ids here but lets just use them for now for example. Maybe
        		// links? names? not sure what works best yet.
        		"4530736b-c1ff-4c07-9a9f-fa9bca418157": {
        			"replicates_to": [
        				"736b9820-7e91-4775-84b0-78e71d60ce4c",
        			],
        			"replicates_from": [
        				"dcc5c518-73c7-4471-83e1-15fae67a98eb",
        			],
        			"writeable": true,
        		},
        	}
        }
    }
}

Instance #3 (Master)

{
    "instance": {
        "created": "2013-05-08T22:43:34",
        "id": "e97cffc3-e91c-4526-987e-711fa557891a",
        "name": "multi-master-hotness-n03",
        "status": "ACTIVE",
        "updated": "2013-05-08T22:43:34",
        "service_type": "mysql-5.5",
        "links": [
            {
                "href": "https://service/v1.0/1234/instances/e97cffc3-e91c-4526-987e-711fa557891a", 
                "rel": "self"
            }, 
            {
                "href": "https://service/instances/e97cffc3-e91c-4526-987e-711fa557891a", 
                "rel": "bookmark"
            }
        ],
        "nodes": [
        	{
        		"id": "736b9820-7e91-4775-84b0-78e71d60ce4c",
        		"hostname": "master-n03.example.com",
                "name": "master-n03",
        		"flavor": {
            		"id": "1", 
            		"links": [...]
        		},
        		"links": [
            		{
                		"href": "https://service/v1.0/1234/instances/{instance_id}/nodes/736b9820-7e91-4775-84b0-78e71d60ce4c", 
                		"rel": "self"
            		}, 
            		{
                		"href": "https://service/instances/{instance_id}/nodes/736b9820-7e91-4775-84b0-78e71d60ce4c", 
                		"rel": "bookmark"
            		}
        		],
        		"volume": {
            		"size": 2,
            		"used": 0.16368598397821188
        		},
        	},
        ],
        "metadata": {
        	"replication_contract": {
        		// We should use something more clear than just nova instance
        		// ids here but lets just use them for now for example. Maybe
        		// links? names? not sure what works best yet.
        		"736b9820-7e91-4775-84b0-78e71d60ce4c": {
        			"replicates_to": [
        				"dcc5c518-73c7-4471-83e1-15fae67a98eb",
        			],
        			"replicates_from": [
        				"4530736b-c1ff-4c07-9a9f-fa9bca418157",
        			],
        			"writeable": true
        		}
        	}
        }
    }
}

Replication without nesting slave instances with their masters

Instance #1 (Master)

{
    "instance": {
        "created": "2013-05-08T22:43:34",
        "id": "cfeccbf4-ac5b-494a-99b7-61593a6a71b0",
        "name": "master-slave-sweetness-n01",
        "status": "ACTIVE",
        "updated": "2013-05-08T22:43:34",
        "service_type": "mysql-5.5",
        "links": [
            {
                "href": "https://service/v1.0/1234/instances/cfeccbf4-ac5b-494a-99b7-61593a6a71b0", 
                "rel": "self"
            }, 
            {
                "href": "https://service/instances/cfeccbf4-ac5b-494a-99b7-61593a6a71b0", 
                "rel": "bookmark"
            }
        ],
        "nodes": [
        	{
        		"id": "dcc5c518-73c7-4471-83e1-15fae67a98eb",
        		"hostname": "master-n01.example.com",
                "name": "master-n01",
        		"flavor": {
            		"id": "1", 
            		"links": [...]
        		},
        		"links": [
            		{
                		"href": "https://service/v1.0/1234/instances/{instance_id}/nodes/dcc5c518-73c7-4471-83e1-15fae67a98eb", 
                		"rel": "self"
            		}, 
            		{
                		"href": "https://service/instances/{instance_id}/nodes/dcc5c518-73c7-4471-83e1-15fae67a98eb", 
                		"rel": "bookmark"
            		}
        		],
        		"volume": {
            		"size": 2,
            		"used": 0.16368598397821188
        		},
        	},
        ],
        "metadata": {
        	"replication_contract": {
        		// We should use something more clear than just nova instance
        		// ids here but lets just use them for now for example. Maybe
        		// links? names? not sure what works best yet.
        		"dcc5c518-73c7-4471-83e1-15fae67a98eb": {
        			"replicates_to": [
        				"4530736b-c1ff-4c07-9a9f-fa9bca418157",
        			],
        			"writeable": true,
        		},
        	}
        }
    }
}

Instance #2 (Read-only Slave)

{
    "instance": {
        "created": "2013-05-08T22:43:34",
        "id": "fdff9fa2-abe3-4536-94dc-887a1bba4274",
        "name": "master-slave-sweetness-n02",
        "status": "ACTIVE",
        "updated": "2013-05-08T22:43:34",
        "service_type": "mysql-5.5",
        "links": [
            {
                "href": "https://service/v1.0/1234/instances/fdff9fa2-abe3-4536-94dc-887a1bba4274", 
                "rel": "self"
            }, 
            {
                "href": "https://service/instances/fdff9fa2-abe3-4536-94dc-887a1bba4274", 
                "rel": "bookmark"
            }
        ],
        "nodes": [
        	{
        		"id": "4530736b-c1ff-4c07-9a9f-fa9bca418157",
        		"hostname": "master-n02.example.com",
                "name": "master-n02",
        		"flavor": {
            		"id": "1", 
            		"links": [...]
        		},
        		"links": [
            		{
                		"href": "https://service/v1.0/1234/instances/{instance_id}/nodes/4530736b-c1ff-4c07-9a9f-fa9bca418157", 
                		"rel": "self"
            		}, 
            		{
                		"href": "https://service/instances/{instance_id}/nodes/4530736b-c1ff-4c07-9a9f-fa9bca418157", 
                		"rel": "bookmark"
            		}
        		],
        		"volume": {
            		"size": 2,
            		"used": 0.16368598397821188
        		},
        	},
        ],
        "metadata": {
        	"replication_contract": {
        		// We should use something more clear than just nova instance
        		// ids here but lets just use them for now for example. Maybe
        		// links? names? not sure what works best yet.
        		"4530736b-c1ff-4c07-9a9f-fa9bca418157": {
        			"replicates_from": [
        				"dcc5c518-73c7-4471-83e1-15fae67a98eb",
        			],
        			"writeable": false,
        		},
        	}
        }
    }
}

Clustering and Replication

A cluster is an independent service type.

  • This simplifies the configurations (ties to service_type)
  • The guest impl can be different for each service_type
  • We don't need cluster type and service_type when defining a cluster


Think: A instance is an instance is a cluster is an instance is NOT replication

CLUSTERING

Definition of cluster for purposes of Trove:
A cluster must provide

  • High Availablility
  • Fault Tolerance


A cluster object is represented as an instance with a metadata attribute containing node information. A node is a sub-resource of a cluster and has a limited number of actions that can be performed on them. A cluster is made of of equal sized nodes using all the same flavor/disk size.

Cluster Operations

Create cluster:

Create a cluster with flavor X and size Y. All nodes are of equal size.
POST /instances

{
	"name": "foobar",
	"flavor": "{flavor_id}",
	"service_type": "{service_type_id}",
	"volume": {
		"size": 50,
	},
	"metadata": {
		"nodes": 5,
		"valuable_cluster_info": {}
	}
}

Delete cluster:

  • Deletes all nodes in the cluster and the cluster itself.

DELETE /instances/{id} { "empty body?" }

Downsize a cluster (Delete nodes):

PATCH /instances/{id}

{
	"metadata": {
		"nodes": 3
	}
}

Or remove specific nodes from the cluster. e.g. You have a node acting up in the cluster and you want to pull it out PATCH /instances/{id}

{
	"nodes": [
		{"id": "32e3c3b8-8d71-4b76-b2be-27fec62bc302"},
		{"id": "3c6176fc-bb33-429f-a926-e8307c115752"},
	]
}

Restart a cluster:

POST /instances/{id}/restart

{
	"empty body?"
}

Resize a cluster:

POST /instances/{id}/resize

{
	"flavor": "flavor_id",
	"volume": {
		"size": 100
	}
}

Initialize cluster:

Clusters can get mucked up sometimes, you may want to keep a node in the cluster but wipe it's data and let the cluster technology repopulate it's data. POST /instance/{id}/initialize

{
	"nodes": [
		{"id": "32e3c3b8-8d71-4b76-b2be-27fec62bc302"},
		{"id": "3c6176fc-bb33-429f-a926-e8307c115752"},
	]
}

Rebalance/restripe cluster:

Changing the hashing algorithm on some cluster technologies may require a rebalance or restriping of the data. This is a maintenance method to support that. POST /instance/{id}/rebalance

{
	"nodes": [
		{"id": "32e3c3b8-8d71-4b76-b2be-27fec62bc302"},
		{"id": "3c6176fc-bb33-429f-a926-e8307c115752"},
	]
}

Node Operations:

Restart a node:

POST /instance/{id}/node/{id}/restart

{
	"empty body?"
}

Initialize a node:

Clusters can get mucked up sometimes, you may want to keep a node in the cluster but wipe it's data and let the cluster technology repopulate it's data. POST /instance/{id}/node/{id}/initialize

{
	"empty body?"
}

Rebalance/restripe cluster:

Changing the hashing algorithm on some cluster technologies may require a rebalance or restriping of the data. This is a maintenance method to support that. POST /instance/{id}/node/{id}/rebalance

{
	"empty body?"
}


REPLICATION

Think of replication as a capability of some service types. Not every service type will have this capability and it should be configurable via the service types capabilities feature.

What you cannot do:

  • create a master and slave in one call

- one call per action means simplified workflow


Replication workflow:

  1. Create an instance
  2. Create an Nth instance with
  • Caveat: adding a replicated slave will alter data on the master instance and will use the agent to create the replication.
{
	"metadata": {
		"replicates_from": [
			"{master_instance_id}"
		],
		// The writeable flag should actually not be required but would populate based on
		// the underlying service type's replication setup.  It is an informative flag only
		// and shouldn't block users.
		"writeable": false 
	}
}


Replication Examples

Examples of what data returned back to the user when they query detailed instance info from the api with: GET /instances/<instance_id>

Master Instance

{
    "name": "blah",
    "id": "5c489b9d-96bb-4e15-a79d-412d88a0062e", etc...
    "metadata": {
        "replication_contract": {
            "replicates_to": [
                "7ee9664f-1261-4076-8abc-21af3a092132",
            ],
            "writeable": true
        }
    }
}

Slave Instance (Read-Only)

{
    "name": "blah-slave",
    "id": "7ee9664f-1261-4076-8abc-21af3a092132", etc...
    "metadata": {
        "replication_contract": {
            "replicates_from": [
                "5c489b9d-96bb-4e15-a79d-412d88a0062e",
            ],
            "writeable": true
        }
    }
}

Replication Operations

Create Replication:

POST /instance

{
	"name": "foobar",
	"flavor": "{flavor_id}",
	"service_type": "mysql-5.1",
	"size": 50,
	"metadata": {
		"replication_contract": {
			"replicates_from": [
				"32e3c3b8-8d71-4b76-b2be-27fec62bc302",
			]
		}
	}
}

Comments: