Trove/Replication-And-Clustering-With-Nodes-5

Example: Cassandra

To illustrate the approach, Cassandra is used in the examples below. The eccentricities of each Datastore will be explained in their own sections.

Create Cluster

Request:

POST /clusters
{
  "cluster": {
    "name": "products",
    "datastore": {
      "type": "cassandra",
      "version": "2.0.6"
    },
    "instances": [
      {
        "flavorRef": "7",
        "volume": {
          "size": 1
        },
        "configuration": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
        "region": "west"
      },
      {
        "flavorRef": "7",
        "volume": {
          "size": 1
        },
        "configuration": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
        "region": "east"
      },
      {
        "flavorRef": "7",
        "volume": {
          "size": 1
        },
        "configuration": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
        "region": "eu"
      }
    ]
  }
}

Response:

Add Me

Future (TBD): Defaults

POST /clusters
{
  "cluster": {
    "name": "products",
    "datastore": {
      "type": "cassandra",
      "version": "2.0.6"
    },
    "defaults": {
      "flavorRef": "7"
      ...
    }

Show Cluster

Request:

GET /clusters/dfbbd9ca-b5e1-4028-adb7-f78643e17998

Response:

{
  "cluster": {
    "status": "ACTIVE",
    "id": "dfbbd9ca-b5e1-4028-adb7-f78643e17998",
    "name": "products",
    "created": "2014-04-25T20:19:23",
    "updated": "2014-04-25T20:19:23",
    "links": [{...}],
    "datastore": {
      "type": "cassandra",
      "version": "2.0.6"
    },
    "instances": [
      {
        "id": "416b0b16-ba55-4302-bbd3-ff566032e1c1",
        "flavor": {
          "id": "7",
          "links": [{...}]
        },
        "configuration": {
          "id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
          "links": [{...}]
        },
        "volume": {
          "size": 1
        },
        "region": "west"
      },
      {
        "id": "7f52e4f9-3fa6-4238-ac08-1ce15197329a"
        "flavor": {
          "id": "7",
          "links": [{...}]
        },
        "configuration": {
          "id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
          "links": [{...}]
        },
        "volume": {
          "size": 1
        },
        "region": "east"
      },
      {
        "id": "ff9d680c-fde3-49c6-a84e-76173b6df39d"
        "flavor": {
          "id": "7",
          "links": [{...}]
        },
        "configuration": {
          "id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
          "links": [{...}]
        },
        "volume": {
          "size": 1
        },
        "region": "eu"
      }
    ]
  }
}

Show Instance

Request:

GET /clusters/dfbbd9ca-b5e1-4028-adb7-f78643e17998/instances/416b0b16-ba55-4302-bbd3-ff566032e1c1

Response:

{
  "instance": {
    "status": "ACTIVE",
    "id": "416b0b16-ba55-4302-bbd3-ff566032e1c1",
    "name": "products-1",
    "created": "2014-04-25T20:19:23",
    "updated": "2014-04-25T20:19:23",
    "links": [{...}],
    "ip": ["10.0.0.1"],
    "configuration": {
      "id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
      "links": [{...}],
    },
    "flavor": {
      "id": "7",
      "links": [{...}],
    },
    "volume": {
      "size": 1,
      "used": 0.17
    }
  }
}

Add Instance(s)

Request:

POST /clusters/dfbbd9ca-b5e1-4028-adb7-f78643e17998/instances
{
  "instances": [
    {
      "flavorRef": "7",
      "volume": {
        "size": 1
      },
      "configuration": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
      "region": "west"
    },
    {
      "flavorRef": "7",
      "volume": {
        "size": 1
      },
      "configuration": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
      "region": "west"
    }
  ]
}

Response:

HTTP 202 (Empty Body)

Replace Instance

Request:

POST /clusters/dfbbd9ca-b5e1-4028-adb7-f78643e17998/action

{
  "replace_instance": {
    "id": "7f52e4f9-3fa6-4238-ac08-1ce15197329a"
  }
}

Response:

HTTP 202 (Empty Body)

Notes:

• http://www.datastax.com/documentation/cassandra/2.0/cassandra/operations/ops_replace_live_node.html
• http://www.datastax.com/documentation/cassandra/2.0/cassandra/operations/ops_replace_seed_node.html
• http://www.datastax.com/documentation/cassandra/2.0/cassandra/operations/ops_replace_node_t.html

Remove Instance

Request:

DELETE /clusters/dfbbd9ca-b5e1-4028-adb7-f78643e17998/instances/7f52e4f9-3fa6-4238-ac08-1ce15197329a

Response:

HTTP 202 (Empty Body)

Notes:

• http://www.datastax.com/documentation/cassandra/2.0/cassandra/operations/ops_remove_node_t.html

Example: MongoDB

Create Cluster

Request:

POST /clusters
{
  "cluster": {
    "name": "products",
    "datastore": {
      "type": "mongodb",
      "version": "2.4.10"
    },
    "instances": [
      {
        "flavorRef": "7",
        "volume": {
          "size": 1
        },
        "configuration": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
        "region": "west"
      },
      {
        "flavorRef": "7",
        "volume": {
          "size": 1
        },
        "configuration": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
        "region": "west"
      },
      {
        "flavorRef": "7",
        "volume": {
          "size": 1
        },
        "configuration": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
        "region": "west"
      },
      {
        "flavorRef": "7",
        "volume": {
          "size": 1
        },
        "configuration": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
        "region": "east"
      },
      {
        "flavorRef": "7",
        "volume": {
          "size": 1
        },
        "configuration": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
        "region": "east"
      }
    ]
  }
}

Response:

Add Me

Show Cluster

Request:

GET /clusters/dfbbd9ca-b5e1-4028-adb7-f78643e17998

Response:

{
  "cluster": {
    "status": "ACTIVE",
    "id": "dfbbd9ca-b5e1-4028-adb7-f78643e17998",
    "name": "products",
    "created": "2014-04-25T20:19:23",
    "updated": "2014-04-25T20:19:23",
    "links": [{...}],
    "datastore": {
      "type": "mongodb",
      "version": "2.4.10"
    },
    "instances": [
      {
        "id": "416b0b16-ba55-4302-bbd3-ff566032e1c1",
        "flavor": {
          "id": "7",
          "links": [{...}]
        },
        "configuration": {
          "id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
          "links": [{...}]
        },
        "volume": {
          "size": 1
        },
        "region": "west"
      },
      {
        "id": "965ef811-7c1d-47fc-89f2-a89dfdd23ef2"
        "flavor": {
          "id": "7",
          "links": [{...}]
        },
        "configuration": {
          "id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
          "links": [{...}]
        },
        "volume": {
          "size": 1
        },
        "region": "west"
      },
      {
        "id": "3642f41c-e8ad-4164-a089-3891bf7f2d2b"
        "flavor": {
          "id": "7",
          "links": [{...}]
        },
        "configuration": {
          "id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
          "links": [{...}]
        },
        "volume": {
          "size": 1
        },
        "region": "west"
      },
      {
        "id": "7f52e4f9-3fa6-4238-ac08-1ce15197329a"
        "flavor": {
          "id": "7",
          "links": [{...}]
        },
        "configuration": {
          "id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
          "links": [{...}]
        },
        "volume": {
          "size": 1
        },
        "region": "east"
      },
      {
        "id": "ff9d680c-fde3-49c6-a84e-76173b6df39d"
        "flavor": {
          "id": "7",
          "links": [{...}]
        },
        "configuration": {
          "id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
          "links": [{...}]
        },
        "volume": {
          "size": 1
        },
        "region": "east"
      }
    ]
  }
}

Show Instance

Request:

GET /clusters/dfbbd9ca-b5e1-4028-adb7-f78643e17998/instances/416b0b16-ba55-4302-bbd3-ff566032e1c1

Response:

{
  "instance": {
    "status": "ACTIVE",
    "id": "416b0b16-ba55-4302-bbd3-ff566032e1c1",
    "name": "products-1",
    "created": "2014-04-25T20:19:23",
    "updated": "2014-04-25T20:19:23",
    "links": [{...}],
    "ip": ["10.0.0.1"],
    "configuration": {
      "id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
      "links": [{...}],
    },
    "flavor": {
      "id": "7",
      "links": [{...}],
    },
    "volume": {
      "size": 1,
      "used": 0.17
    }
  }
}

Create Arbiter(s)

Request:

POST /clusters/dfbbd9ca-b5e1-4028-adb7-f78643e17998/instances

{
  "instances": [
    {
      "flavorRef": "7",
      "volume": {
        "size": 1
      },
      "configuration": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
      "region": "eu",
      "type": "arbiter"
    },
    {
      "flavorRef": "7",
      "volume": {
        "size": 1
      },
      "configuration": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
      "region": "eu",
      "type": "arbiter"
    }
  ]
}

Response:

HTTP 202 (Empty Body)

Show Cluster (After Arbiters)

Request:

GET /clusters/dfbbd9ca-b5e1-4028-adb7-f78643e17998

Response:

{
  "cluster": {
    "status": "ACTIVE",
    "id": "dfbbd9ca-b5e1-4028-adb7-f78643e17998",
    "name": "products",
    "created": "2014-04-25T20:19:23",
    "updated": "2014-04-25T20:19:23",
    "links": [{...}],
    "datastore": {
      "type": "mongodb",
      "version": "2.4.10"
    },
    "instances": [
      {
        "id": "416b0b16-ba55-4302-bbd3-ff566032e1c1",
        "flavor": {
          "id": "7",
          "links": [{...}]
        },
        "configuration": {
          "id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
          "links": [{...}]
        },
        "volume": {
          "size": 1
        },
        "region": "west"
      },
      {
        "id": "965ef811-7c1d-47fc-89f2-a89dfdd23ef2"
        "flavor": {
          "id": "7",
          "links": [{...}]
        },
        "configuration": {
          "id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
          "links": [{...}]
        },
        "volume": {
          "size": 1
        },
        "region": "west"
      },
      {
        "id": "3642f41c-e8ad-4164-a089-3891bf7f2d2b"
        "flavor": {
          "id": "7",
          "links": [{...}]
        },
        "configuration": {
          "id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
          "links": [{...}]
        },
        "volume": {
          "size": 1
        },
        "region": "west"
      },
      {
        "id": "7f52e4f9-3fa6-4238-ac08-1ce15197329a"
        "flavor": {
          "id": "7",
          "links": [{...}]
        },
        "configuration": {
          "id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
          "links": [{...}]
        },
        "volume": {
          "size": 1
        },
        "region": "east"
      },
      {
        "id": "ff9d680c-fde3-49c6-a84e-76173b6df39d"
        "flavor": {
          "id": "7",
          "links": [{...}]
        },
        "configuration": {
          "id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
          "links": [{...}]
        },
        "volume": {
          "size": 1
        },
        "region": "east"
      },
      {
        "id": "77032c55-4496-4e35-8c0d-6cd1c18e1a9c"
        "flavor": {
          "id": "7",
          "links": [{...}]
        },
        "configuration": {
          "id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
          "links": [{...}]
        },
        "volume": {
          "size": 1
        },
        "region": "eu",
        "type": "arbiter"
      },
      {
        "id": "1fd054ed-221f-4c99-8d17-570bcff4c1d2"
        "flavor": {
          "id": "7",
          "links": [{...}]
        },
        "configuration": {
          "id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
          "links": [{...}]
        },
        "volume": {
          "size": 1
        },
        "region": "eu",
        "type": "arbiter"
      }
    ]
  }
}

Example: MySQL

Create Master

Request:

POST /instances
{
  "instance": {
    "name": "products",
    "datastore": {
      "type": "mysql",
      "version": "5.5"
    },
    "configuration": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
    "flavorRef": "7",
    "volume": {
      "size": 1
    }
  }
}

Response:

{
  "instance": {
    "status": "BUILD",
    "id": "dfbbd9ca-b5e1-4028-adb7-f78643e17998",
    "name": "products",
    "created": "2014-04-25T20:19:23",
    "updated": "2014-04-25T20:19:23",
    "links": [{...}],
    "datastore": {
      "type": "mysql",
      "version": "5.5"
    },
    "configuration": {
      "id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
      "links": [{...}],
    },
    "flavor": {
      "id": "7",
      "links": [{...}],
    },
    "volume": {
      "size": 1
    }
  }
}

Create Slave

Request:

POST /instances
{
  "instance": {
    "name": "products-slave",
    "datastore": {
      "type": "mysql",
      "version": "5.5"
    },
    "configuration": "fc318e00-3a6f-4f93-af99-146b44912188",
    "flavorRef": "7",
    "volume": {
      "size": 1
    },
    "slave": {
      "of": "dfbbd9ca-b5e1-4028-adb7-f78643e17998",
      "read_only": true
    }
  }
}

Response:

{
  "instance": {
    "status": "BUILD",
    "id": "061aaf4c-3a57-411e-9df9-2d0f813db859",
    "name": "products",
    "created": "2014-04-25T20:19:23",
    "updated": "2014-04-25T20:19:23",
    "links": [{...}],
    "datastore": {
      "type": "mysql",
      "version": "5.5"
    },
    "configuration": {
      "id": "fc318e00-3a6f-4f93-af99-146b44912188",
      "links": [{...}],
    },
    "flavor": {
      "id": "7",
      "links": [{...}],
    },
    "volume": {
      "size": 1
    },
    "slave": {
      "of": "dfbbd9ca-b5e1-4028-adb7-f78643e17998",
      "read_only": true
    }
  }
}

Show Master

Request:

GET /instances/dfbbd9ca-b5e1-4028-adb7-f78643e17998

Response:

{
  "instance": {
    "status": "ACTIVE",
    "id": "dfbbd9ca-b5e1-4028-adb7-f78643e17998",
    "name": "products",
    "created": "2014-04-25T20:19:23",
    "updated": "2014-04-25T20:19:23",
    "links": [{...}],
    "datastore": {
      "type": "mysql",
      "version": "5.5"
    },
    "configuration": {
      "id": "b9c8a3f8-7ace-4aea-9908-7b555586d7b6",
      "links": [{...}],
    },
    "flavor": {
      "id": "7",
      "links": [{...}],
    },
    "volume": {
      "size": 1
    },
    "slave": {
      "list": [
        {"id": "061aaf4c-3a57-411e-9df9-2d0f813db859"}
      ]
    }
  }
}

Show Slave

Request:

GET /instances/061aaf4c-3a57-411e-9df9-2d0f813db859

Response:

{
  "instance": {
    "status": "ACTIVE",
    "id": "061aaf4c-3a57-411e-9df9-2d0f813db859",
    "name": "products",
    "created": "2014-04-25T20:19:23",
    "updated": "2014-04-25T20:19:23",
    "links": [{...}],
    "datastore": {
      "type": "mysql",
      "version": "5.5"
    },
    "configuration": {
      "id": "fc318e00-3a6f-4f93-af99-146b44912188",
      "links": [{...}],
    },
    "flavor": {
      "id": "7",
      "links": [{...}],
    },
    "volume": {
      "size": 1
    },
    "slave": {
      "of": "dfbbd9ca-b5e1-4028-adb7-f78643e17998",
      "read_only": true
    }
  }
}

Detach Slave

Request:

POST /instances/061aaf4c-3a57-411e-9df9-2d0f813db859/action

{
  "detach": {}
}

Response:

HTTP 202 (Empty Body)

Delete Master

Request:

DELETE /instances/dfbbd9ca-b5e1-4028-adb7-f78643e17998

Response:

HTTP 202 (Empty Body)

Notes:

• How to handle situation in which a slave is attached to a master, and the user attempts to delete the master?

Delete Slave

Request:

DELETE /instances/061aaf4c-3a57-411e-9df9-2d0f813db859

Response:

HTTP 202 (Empty Body)

Data Model Changes

clusters Table

Create a new 'clusters' Table:

CREATE TABLE "clusters" (
  "id" varchar(36) NOT NULL,
  "created" datetime DEFAULT NULL,
  "updated" datetime DEFAULT NULL,
  "name" varchar(255) DEFAULT NULL,
  "task_id" int(11) DEFAULT NULL,
  "task_description" varchar(32) DEFAULT NULL,
  "task_start_time" datetime DEFAULT NULL,
  "tenant_id" varchar(36) DEFAULT NULL,
  "server_status" varchar(64) DEFAULT NULL,
  "datastore_version_id" varchar(36) NOT NULL,
  "deleted" tinyint(1) DEFAULT NULL,
  "deleted_at" datetime DEFAULT NULL,
  PRIMARY KEY ("id"),
  KEY "datastore_version_id" ("datastore_version_id"),
  KEY "clusters_tenant_id" ("tenant_id"),
  KEY "clusters_deleted" ("deleted"),
  CONSTRAINT "clusters_ibfk_1" FOREIGN KEY ("datastore_version_id") REFERENCES "datastore_versions" ("id")
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

Notes:

• todo: task_* related columns were simply copy/pasted from instances; could diverge
• todo: change 'default null' to 'not null' whenever possible

Alter Instances Table

Add slave_of Column to Instances Table (+ Constraint + Index):

ALTER TABLE instances ADD COLUMN slave_of VARCHAR(36) DEFAULT NULL;
KEY "slave_of" ("slave_of"),
CONSTRAINT "instances_ibfk_3" FOREIGN KEY ("slave_of") REFERENCES "instances" ("id")

Add cluster_id Column to Instances Table (+ Constraint + Index):

ALTER TABLE instances ADD COLUMN cluster_id VARCHAR(36) DEFAULT NULL;
KEY "cluster_id" ("cluster_id"),
CONSTRAINT "instances_ibfk_4" FOREIGN KEY ("cluster_id") REFERENCES "clusters" ("id")

TaskManager

• add cluster_id to /etc/guest_info (if it's an instance in a cluster). guest_id remains as-is.
• for-loop create each instance.
• poll until all instances are active.
• for each instance: use trove/nova to get ip/hostname
• for couchbase:
  • send ip/hostname list via rpc cast to guest
• for cassandra:
  • send seed ip list via rpc cast to guest seed instances, one by one (polling on REBOOT => ACTIVE), then to rest of instances.
• for mongodb:
  • send ip/hostname list via rpc cast to guest that is the db.isMaster()

Guest

• update heartbeat payload ( heartbeat(guest_id, payload, sent) ) from {"service_status": "<status>"} to {"service_status": "<status>", "cluster_id": "<cluster-id>"}
• add method to each datastore guest manager for handling ip/hostname list

Conductor

• update heartbeat logic to update the clusters table (cluster status) if an instance's status changes the overall state of the cluster

Capabilities

A capability might be supported for a datastore-version for standalone instances, but not for clusters. Therefore, the capability tables must be amended to include a cluster-enabled flag.

ALTER TABLE capabilities ADD COLUMN enabled_cluster TINYINT(1) DEFAULT NULL;

ALTER TABLE capability_overrides ADD COLUMN enabled_cluster TINYINT(1) DEFAULT NULL;

The following capabilities should have enabled_cluster set to false for the first iteration of clusters:

• backup-create + list-instance
• configuration-attach + detach + instances
• resize-<all>
• database-<all>
• root-<all>
• secgroup-<all>
• user-<all>

Configuration Groups

Introduce read_only and hidden Parameters

Need to introduce two additional attributes for configuration group parameters: read_only and hidden.

• read_only fields include cluster_name, num_tokens, seed_provider, seeds, endpoint_snitch (cassandra) + replSet (mongodb) + server_id, log_bin (mysql).
• depending on the provider, some of the read_only fields should also be hidden from the user on a configuration-show.
• once read_only + hidden are available, a parallel effort should move configuration-default to configurations-show if a configuration-group is attached.
• amcrn (talk) 20:43, 8 May 2014 (UTC): update: mysql master/slave will not be required to do this because the overrides.cnf functionality handles this nicely. to be determined as to how clustering will handle this. it could be this, or it could be a copy of the original conf, or a mixture thereof.

Auto-Create and Attach

• cassandra & mongodb need to have configuration-groups automatically created and attached to each instance (for cluster_name, replset, etc.) during provisioning.
• unique configuration-group per instance.
• auto-created+attached configuration-groups need to not be detachable from the instance.
• dependency: configuration-group support for mongodb + cassandra
• amcrn (talk) 20:43, 8 May 2014 (UTC): update: mysql master/slave will not be required to do this because the overrides.cnf functionality handles this nicely. to be determined as to how clustering will handle this. it could be this, or it could be a copy of the original conf, or a mixture thereof.

python-troveclient

Notes:

• Discussed options are documented at https://gist.github.com/amcrn/c4ae2210e9d9864c21fd . An option was chosen on 05/21 and is described below.

Create-Cluster Help

$ trove help create-cluster

usage: trove create-cluster <name> <datastore> <datastore_version>
                            [--instance <instance>]

Creates a new cluster.

Positional arguments:
  <name>                Name of the cluster
  <datastore>           A datastore name or UUID
  <datastore_version>   A datastore version name or UUID

Optional arguments:

  --instance <flavor=flavor-id,disk=disk-size,config-group=group-uuid,type=type-name>
                        Create an instance for the cluster. Specify multiple
                        times to create multiple instances.

Create Cluster Request

$ trove create-cluster products mongodb "2.4.10"
    --instance flavor=12,disk=100,config-group=e0a6a893-1e41-4a05-8252-3c2cb934dd79 \
    --instance flavor=12,disk=100,config-group=e0a6a893-1e41-4a05-8252-3c2cb934dd79 \
    --instance flavor=12,disk=100,config-group=e0a6a893-1e41-4a05-8252-3c2cb934dd79 \
    --instance flavor=12,disk=100,config-group=e0a6a893-1e41-4a05-8252-3c2cb934dd79 \
    --instance flavor=12,disk=100,config-group=e0a6a893-1e41-4a05-8252-3c2cb934dd79 \
    --instance flavor=7,disk=1,config-group=b9c8a3f8-7ace-4aea-9908-7b555586d7b6,type=arbiter \
    --instance flavor=7,disk=1,config-group=b9c8a3f8-7ace-4aea-9908-7b555586d7b6,type=arbiter

Create Cluster Response

TBD