Jump to: navigation, search

Difference between revisions of "Trove-Replication-And-Clustering-API"

(POST /clusters/restore/{backups_id}/)
Line 1: Line 1:
== List cluster types: ==
+
***************** WORK IN PROGRESS ********************
Returned json:<br/>
+
A cluster is an independant service type.
 +
- This simplifies the configurations (ties to service_type)
 +
- The guest impl can be different for each service_type
 +
- We don't need cluster type and service_type when defining a cluster
  
===== GET /clustertypes =====
+
Think: A instance is an instance is a cluster is an instance is *NOT* replication
<pre>
+
 
{
+
CLUSTERING
    "clusterTypes": [
 
        {
 
            "id": "7782954c-ebec-42f0-894b-d3601016a91e",
 
            "links": [
 
                {
 
                    "href": "https://service/v1.0/1234/clustertypes/7782954c-ebec-42f0-894b-d3601016a91e",
 
                    "rel": "self"
 
                },
 
                {
 
                    "href": "https://service/clustertypes/7782954c-ebec-42f0-894b-d3601016a91e",
 
                    "rel": "self"
 
                }
 
            ],
 
            "name": "Master/Slave Replication",
 
            "type": "master-slave"
 
        }
 
    ]
 
}
 
</pre>
 
  
== Create Replication Set: (No previous db instance, fresh) ==
+
Definition of cluster for purposes of Trove:
This will create 1 master with 2 replicas.<br/>
+
A cluster must provide
 +
- High Availablility
 +
- Fault Tolerance
  
===== POST /clusters =====
+
A cluster object is represented as an instance with a metadata attribute
  <pre>
+
containing node information. A node is a sub-resource of a cluster and
{
+
has a limited number of actions that can be performed on them.
    "cluster": {
 
        "nodes": 3,
 
        "flavorRef": "https://service/v1.0/1234/flavors/1",
 
        "name": "replication_set_1",
 
        "volume": {
 
            "size": 2
 
        },
 
        "clusterConfig": {
 
            "type": "https://service/v1.0/1234/clustertypes/7782954c-ebec-42f0-894b-d3601016a91e"
 
        }
 
    }
 
}
 
</pre>
 
'''Errors:'''<br>
 
ClusterTypeNotFound: ClusterType '''{id}''' not found<br>
 
MaxNumNodes: Node count '''{nodes}''' exceeds maximum '''{conf max}'''<br>
 
  
== Create Replication Set: (Previous db instance) ==
+
Create cluster:
Create a replication set with primaryNode as the model for instance flavor and volume size and it will server as the "Master" node if the cluster type is master/slave or equivalent<br/>
+
Create a cluster with flavor X and size Y.  All nodes are of equal size.
  
===== POST /clusters =====
+
POST /instance
'''Optional attributes: ''' flavorRef
 
<pre>
 
 
{
 
{
    "cluster": {
+
"name": "foobar",
        "nodes": 3,
+
"flavor": "{flavor_id}",
        "name": "replication_set_1",
+
"service_type": "{service_type_id}",
        "flavorRef": "https://service/v1.0/1234/flavors/1",
+
"size": 50,
        "clusterConfig": {
+
"metadata": {
            "type": "https://service/v1.0/1234/clustertypes/7782954c-ebec-42f0-894b-d3601016a91e",
+
"nodes": 5,
            "primaryNode": "https://service/v1.0/1234/instances/d3ca44b6-1ba2-4862-b16d-1b2cd115944d"
+
}
        },
 
    }
 
 
}
 
}
</pre>
 
'''Errors:'''<br>
 
ClusterTypeNotFound: ClusterType '''{id}''' not found<br>
 
MaxNumNodes: Node count '''{nodes}''' exceeds maximum '''{conf max}'''<br>
 
PrimaryNodeNotFound: Primary node '''{id}''' not found<br>
 
  
== List cluster or replication sets: ==
 
===== GET /clusters =====
 
Returned JSON object<br/>
 
<pre>
 
{
 
    "clusters": [
 
        {
 
            "clusterConfig": {
 
                "type": "https://service/v1.0/1234/clustertypes/7782954c-ebec-42f0-894b-d3601016a91e",
 
            },
 
            "id": "a66cab3a-041b-4c17-82e7-7ca5333a18f3",
 
            "nodes": [
 
                {"id": "https://service/v1.0/1234/instances/fbd14327-230c-495e-b6d9-cb593ccd4cbb"},
 
                {"id": "https://service/v1.0/1234/instances/7a369ce2-0cfb-4bcd-b342-370a4ae55c09"},
 
                {"id": "https://service/v1.0/1234/instances/f7e68ccf-2da6-4bb8-a75b-afe0dd3a0080"}
 
            ]
 
        }
 
    ]
 
}
 
</pre>
 
== List cluster or replication set details: ==
 
===== GET /clusters/{cluster_id} =====
 
<pre>
 
{
 
    "cluster": {
 
        "clusterConfig": {
 
            "type": "https://service/v1.0/1234/clustertypes/7782954c-ebec-42f0-894b-d3601016a91e",
 
        },
 
        "id": "a66cab3a-041b-4c17-82e7-7ca5333a18f3",
 
        "nodes": [
 
            {
 
                "flavor": {
 
                    "id": "1",
 
                    "links": [
 
                        {
 
                            "href": "https://service/v1.0/1234/flavors/1",
 
                            "rel": "self"
 
                        },
 
                        {
 
                            "href": "https://service/flavors/1",
 
                            "rel": "bookmark"
 
                        }
 
                    ]
 
                },
 
                "id": "fbd14327-230c-495e-b6d9-cb593ccd4cbb",
 
                "links": [
 
                    {
 
                        "href": "https://service/v1.0/1234/instances/fbd14327-230c-495e-b6d9-cb593ccd4cbb",
 
                        "rel": "self"
 
                    },
 
                    {
 
                        "href": "https://service/instances/fbd14327-230c-495e-b6d9-cb593ccd4cbb",
 
                        "rel": "bookmark"
 
                    }
 
                ],
 
                "name": "master_instance",
 
                "status": "ACTIVE",
 
                "volume": {
 
                    "size": 20
 
                },
 
            },
 
            {
 
                "flavor": {
 
                    "id": "1",
 
                    "links": [
 
                        {
 
                            "href": "https://service/v1.0/1234/flavors/1",
 
                            "rel": "self"
 
                        },
 
                        {
 
                            "href": "https://service/flavors/1",
 
                            "rel": "bookmark"
 
                        }
 
                    ]
 
                },
 
                "id": "7a369ce2-0cfb-4bcd-b342-370a4ae55c09",
 
                "links": [
 
                    {
 
                        "href": "https://service/v1.0/1234/instances/7a369ce2-0cfb-4bcd-b342-370a4ae55c09",
 
                        "rel": "self"
 
                    },
 
                    {
 
                        "href": "https://service/instances/7a369ce2-0cfb-4bcd-b342-370a4ae55c09",
 
                        "rel": "bookmark"
 
                    }
 
                ],
 
                "name": "master_instance",
 
                "status": "ACTIVE",
 
                "volume": {
 
                    "size": 20
 
                },
 
            },
 
            {
 
                "flavor": {
 
                    "id": "1",
 
                    "links": [
 
                        {
 
                            "href": "https://service/v1.0/1234/flavors/1",
 
                            "rel": "self"
 
                        },
 
                        {
 
                            "href": "https://service/flavors/1",
 
                            "rel": "bookmark"
 
                        }
 
                    ]
 
                },
 
                "id": "f7e68ccf-2da6-4bb8-a75b-afe0dd3a0080",
 
                "links": [
 
                    {
 
                        "href": "https://service/v1.0/1234/instances/f7e68ccf-2da6-4bb8-a75b-afe0dd3a0080",
 
                        "rel": "self"
 
                    },
 
                    {
 
                        "href": "https://service/instances/f7e68ccf-2da6-4bb8-a75b-afe0dd3a0080",
 
                        "rel": "bookmark"
 
                    }
 
                ],
 
                "name": "master_instance",
 
                "status": "ACTIVE",
 
                "volume": {
 
                    "size": 20
 
                },
 
            },
 
        ]
 
    }
 
}
 
</pre>
 
== Drop node from cluster: ==
 
List the nodes to delete, allow for more than one node to be specified.
 
===== DELETE /clusters/{cluster_id}/nodes =====
 
<pre>
 
{
 
    "cluster": {
 
        "id": "a66cab3a-041b-4c17-82e7-7ca5333a18f3",
 
        "nodes": [
 
            /* The proposal here would be to allow multiple
 
            instances to be removed if the clusterType
 
            supports it. Think "I have a master and 2 slaves
 
            for christmas traffic but no longer need the
 
            slaves." */
 
            {"id": "https://service/v1.0/1234/instances/7a369ce2-0cfb-4bcd-b342-370a4ae55c09"},
 
            {"id": "https://service/v1.0/1234/instances/f7e68ccf-2da6-4bb8-a75b-afe0dd3a0080"}
 
        ]
 
    }
 
}
 
</pre>
 
Or, only one at a time? Or both? <br/>
 
<pre>
 
{
 
    "cluster": {
 
        "id": "a66cab3a-041b-4c17-82e7-7ca5333a18f3",
 
        "node": {
 
            "id": "f7e68ccf-2da6-4bb8-a75b-afe0dd3a0080"
 
        }
 
    }
 
}
 
</pre>
 
Or just reduce the number of nodes and let the system decide?
 
or just allow either "node" or "nodes" to give users more ways to
 
control their cluster. <br/>
 
<pre>
 
{
 
    "cluster": {
 
        "id": "a66cab3a-041b-4c17-82e7-7ca5333a18f3",
 
        "nodes": 2
 
    }
 
}
 
</pre>
 
'''Errors:'''<br>
 
MinNumNodes: Node count '''{nodes}''' below the minimum required '''{conf max}'''<br>
 
  
== Add node to cluster/replication: ==
+
Delete cluster:
===== PUT /clusters/{cluster_id}/nodes =====
+
Deletes all nodes in the cluster.
Add nodes by just increasing the node count? <br/>
+
DELETE /instance/{id}
<pre>
 
{
 
    /* adding nodes to the cluster, it is assumed that
 
    you are bringing an uncreated instance into the
 
    the cluster of servers. If nodes are of different
 
    sizes then default to the size of the master/primary node */
 
    "cluster": {
 
        "id": "a66cab3a-041b-4c17-82e7-7ca5333a18f3",
 
        "nodes": 5,
 
        "clusterConfig": {
 
            "type": "https://service/v1.0/1234/clustertypes/7782954c-ebec-42f0-894b-d3601016a91e",
 
        }
 
  
    }
+
Downsize a cluster (Delete nodes):
}
+
PATCH /instance/{id}
</pre>
 
<pre>
 
 
{
 
{
    /* Add nodes to the cluster by giving specs on
+
"metadata" {
    the node to add */
+
"nodes": 3
    "cluster": {
+
}
        "id": "a66cab3a-041b-4c17-82e7-7ca5333a18f3",
 
        "clusterConfig": {
 
            "type": "https://service/v1.0/1234/clustertypes/7782954c-ebec-42f0-894b-d3601016a91e",
 
        },
 
        "node": {
 
            "flavorRef": "https://service/v1.0/1234/flavors/1",
 
            "volume": {
 
                "size": 40
 
            }
 
        }
 
    }
 
 
}
 
}
</pre>
 
'''Errors:'''<br>
 
ClusterTypeNotFound: ClusterType '''{id}''' not found<br>
 
MaxNumNodes: Node count '''{nodes}''' exceeds maximum '''{conf max}'''<br>
 
  
== Promote a slave node to master: ==
+
Restart a cluster:
===== PUT /clusters/{cluster_id} =====
+
POST /instance/{id}/restart
<pre>
 
{
 
    "cluster": {
 
        "clusterConfig": {
 
            "type": "https://service/v1.0/1234/clustertypes/7782954c-ebec-42f0-894b-d3601016a91e",
 
        },
 
        "id": "a66cab3a-041b-4c17-82e7-7ca5333a18f3",
 
        "node": {
 
            "flavor": {
 
                "id": "1",
 
                "links": [
 
                    {
 
                        "href": "https://service/v1.0/1234/flavors/1",
 
                        "rel": "self"
 
                    },
 
                    {
 
                        "href": "https://service/flavors/1",
 
                        "rel": "bookmark"
 
                    }
 
                ]
 
            },
 
            "id": "f7e68ccf-2da6-4bb8-a75b-afe0dd3a0080",
 
            "links": [
 
                {
 
                    "href": "https://service/v1.0/1234/instances/f7e68ccf-2da6-4bb8-a75b-afe0dd3a0080",
 
                    "rel": "self"
 
                },
 
                {
 
                    "href": "https://service/instances/f7e68ccf-2da6-4bb8-a75b-afe0dd3a0080",
 
                    "rel": "bookmark"
 
                }
 
            ],
 
            "name": "master_instance",
 
            "status": "ACTIVE",
 
            "volume": {
 
                "size": 20
 
            },
 
        }
 
    }
 
}
 
</pre>
 
'''Errors:'''<br>
 
ClusterTypeNotFound: ClusterType '''{id}''' not found<br>
 
PrimaryNodeNotFound: Primary node '''{id}''' not found<br>
 
InvalidRole: Role '''{role}''' is not valid<br>
 
  
== Restart a cluster (All nodes): ==
+
Restart a cluster node
===== PUT /clusters/{cluster_id}/restart =====
+
POST /instance/{id}/node/{id}/restart
<pre>
 
{
 
    "cluster": {
 
        "clusterConfig": {
 
            "type": "https://service/v1.0/1234/clustertypes/7782954c-ebec-42f0-894b-d3601016a91e",
 
        },
 
        "id": "a66cab3a-041b-4c17-82e7-7ca5333a18f3",
 
    }
 
}
 
</pre>
 
'''Errors:'''<br>
 
ClusterTypeNotFound: ClusterType '''{id}''' not found<br>
 
ClusterNotFound: Cluster '''{id}''' not found<br>
 
  
== Restart an instance of a cluster: ==
+
Resize a cluster:
===== PUT /clusters/{cluster_id}/nodes/{node_id}/restart =====
+
POST /instance/{id}/resize
<pre>
 
{
 
    "cluster": {
 
        "clusterConfig": {
 
            "type": "https://service/v1.0/1234/clustertypes/7782954c-ebec-42f0-894b-d3601016a91e",
 
        },
 
        "id": "a66cab3a-041b-4c17-82e7-7ca5333a18f3",
 
    },
 
    "node": {
 
        "id": "f7e68ccf-2da6-4bb8-a75b-afe0dd3a0080",
 
    }
 
}
 
</pre>
 
'''Errors:'''<br>
 
ClusterTypeNotFound: ClusterType '''{id}''' not found<br>
 
ClusterNotFound: Cluster '''{id}''' not found<br>
 
NodeNotFound: Node '''{id}''' not found<br>
 
  
== Resize an instance of a cluster: ==
+
Initialize cluster:
===== PUT /clusters/{cluster_id}/nodes/{node_id}/resize =====
+
POST /instance/{id}/initialize
<pre>
 
{
 
    "cluster": {
 
        "clusterConfig": {
 
            "type": "https://service/v1.0/1234/clustertypes/7782954c-ebec-42f0-894b-d3601016a91e",
 
        },
 
        "id": "a66cab3a-041b-4c17-82e7-7ca5333a18f3",
 
    },
 
    "node": {
 
        "id": "f7e68ccf-2da6-4bb8-a75b-afe0dd3a0080",
 
        "flavor": {
 
            "id": "2",
 
            "links": [
 
                {
 
                    "href": "https://service/v1.0/1234/flavors/2",
 
                    "rel": "self"
 
                },
 
                {
 
                    "href": "https://service/flavors/2",
 
                    "rel": "bookmark"
 
                }
 
            ],
 
        },
 
        "volume": {
 
            "size": 40
 
        },
 
    }
 
}
 
</pre>
 
'''Errors:'''<br>
 
ClusterTypeNotFound: ClusterType '''{id}''' not found<br>
 
ClusterNotFound: Cluster '''{id}''' not found<br>
 
NodeNotFound: Node '''{id}''' not found<br>
 
  
 +
Rebalance/restripe cluster:
 +
POST /instance/{id}/rebalance
  
== List backups ==
+
Node Operations:
===== GET /clusters/{cluster_id}/backups =====
+
Restart a node:
 +
PATCH /instance/{id}/node/{id}/restart
  
<pre>
+
Initialize a node:
{
+
This only where applicable...
  "backups": [
+
PATCH /instance/{id}/node/{id}/initialize
                {
 
                    "id": "a66cab3a-041b-4c17-82e7-7ca5333a18f3"
 
                    "name": "testback-backup"
 
                    "description":"Backup description"
 
                    "created" : "1980-03-22T00:00:00UTC"
 
                  },
 
                {
 
                      "id": "a66cab3a-041b-4c17-82e7-7ca5333a18f5"
 
                    "name": "new-year-backup"
 
                    "description":"Backup description"
 
                    "created" : "1980-03-22T00:00:00UTC"
 
                  }
 
    ]
 
}
 
  
</pre>
 
  
'''Errors:'''<br>
 
ClusterNotFound: Cluster '''{id}''' not found<br>
 
  
== Create Cluster Backup ==
 
=====POST /clusters/{cluster_id}/backups=====
 
<pre>
 
{
 
  "backup": [
 
                {
 
                    "id": a66cab3a-041b-4c17-82e7-7ca5333a18f5
 
                    "name": "testback-backup"
 
                    "description": [
 
                                            {
 
                                                "clusterType": 7782954c-ebec-42f0-894b-d3601016a91e
 
                                                "nodes": 3
 
                                                "node_backups": [ (list of node backups) ]
 
                                                "service_engine": "cassandra"
 
                                                "service_version": "1.2.8"
 
                                            }
 
                                        ]
 
                  }
 
          ]
 
}
 
</pre>
 
  
'''Errors:'''<br>
+
REPLICATION
ClusterNotFound: Cluster '''{id}''' not found<br>
 
  
 +
We think of replication as a capability of some service types.  Not every
 +
service type will have this capability and it should be configurable via
 +
the service types capabilities feature.
  
== Point-to-Restore Cluster Backup ==
+
What you cannot do:
=====POST /clusters/restore/{backups_id}/=====
+
- create a master and slave in one call
<pre>
+
- one call per action means simplified workflow
{
 
  "backup": [
 
                {
 
                    "id": a66cab3a-041b-4c17-82e7-7ca5333a18f5
 
                    "nodes": 2(optional, this parameter could be passed to restore cluster with less nodes than in backup stores)
 
                    "clusterType": 7782954c-ebec-42f0-894b-d3601016a91e
 
                  }
 
          ]
 
}
 
</pre>
 
  
'''Errors:'''<br>
 
ClusterBackupNotFound: Backup '''{id}''' not found<br>
 
  
== Boot from backup ==
+
Replication workflow:
===== PUT /clusters/boot/{backups_id}/ =====
+
1.  Create an instance
<pre>
+
2. Create an Nth instance with
{
+
- metadata {"master_instance": "id from 1", "replication_type": "typeid from replication_types api"}
          "id": a66cab3a-041b-4c17-82e7-7ca5333a18f5
+
- Caveat: adding a replicated slave will alter data on the master instance and will use the agent
}
+
to create the replication.
</pre>
 
''' *** backup stores clusters configuration, so it recreates whole new cluster with the same configuration and data *** ''' <br>
 
  
'''Errors:'''<br>
+
Create Replication:
ClusterBackupNotFound: Backup '''{id}''' not found<br>
 
  
== Delete backup ==
+
POST /instance
===== DELETE /clusters/{cluster_id}/backups =====
 
<pre>
 
 
{
 
{
    backups: [
+
"name": "foobar",
        { "id": "a66cab3a-041b-4c17-82e7-7ca5333a18f5"},
+
"flavor": "flavor_id",
        { "id": "a66cab3a-041b-4c17-82e7-7ca5333a183"},
+
"service_type": "mysql-5.1",
    ]
+
"size": 50,
 +
"metadata": {
 +
"replication_type": "replication_type_id",
 +
"replication_data": {
 +
"master_instance": "instance_uuid"
 +
}
 +
}
 
}
 
}
</pre>
 
'''Errors:'''<br>
 
ClusterNotFound: Cluster '''{id}''' not found<br>
 
BackupNotFound: Backup '''{id}''' not found<br>
 
 
= COMMENTS =
 
I think imsplitbit is the bee's knees.
 

Revision as of 17:39, 7 October 2013

                                  • WORK IN PROGRESS ********************

A cluster is an independant service type. - This simplifies the configurations (ties to service_type) - The guest impl can be different for each service_type - We don't need cluster type and service_type when defining a cluster

Think: A instance is an instance is a cluster is an instance is *NOT* replication

CLUSTERING

Definition of cluster for purposes of Trove: A cluster must provide - High Availablility - Fault Tolerance

A cluster object is represented as an instance with a metadata attribute containing node information. A node is a sub-resource of a cluster and has a limited number of actions that can be performed on them.

Create cluster: Create a cluster with flavor X and size Y. All nodes are of equal size.

POST /instance { "name": "foobar", "flavor": "{flavor_id}", "service_type": "{service_type_id}", "size": 50, "metadata": { "nodes": 5, } }


Delete cluster: Deletes all nodes in the cluster. DELETE /instance/{id}

Downsize a cluster (Delete nodes): PATCH /instance/{id} { "metadata" { "nodes": 3 } }

Restart a cluster: POST /instance/{id}/restart

Restart a cluster node POST /instance/{id}/node/{id}/restart

Resize a cluster: POST /instance/{id}/resize

Initialize cluster: POST /instance/{id}/initialize

Rebalance/restripe cluster: POST /instance/{id}/rebalance

Node Operations: Restart a node: PATCH /instance/{id}/node/{id}/restart

Initialize a node: This only where applicable... PATCH /instance/{id}/node/{id}/initialize



REPLICATION

We think of replication as a capability of some service types. Not every service type will have this capability and it should be configurable via the service types capabilities feature.

What you cannot do: - create a master and slave in one call - one call per action means simplified workflow


Replication workflow: 1. Create an instance 2. Create an Nth instance with - metadata {"master_instance": "id from 1", "replication_type": "typeid from replication_types api"} - Caveat: adding a replicated slave will alter data on the master instance and will use the agent to create the replication.

Create Replication:

POST /instance { "name": "foobar", "flavor": "flavor_id", "service_type": "mysql-5.1", "size": 50, "metadata": { "replication_type": "replication_type_id", "replication_data": { "master_instance": "instance_uuid" } } }