Jump to: navigation, search

Trove/Configurations

Overview

The MySQL configuration management capability will allow users of Trove to override the default MySQL configuration settings provided by the operator of the Trove service. This is implemented by leveraging MySQL’s includedir directive, which is already pre-defined to point to /etc/mysql/conf.d. The configuration files stored (or symlinked) into this directory will be interpreted by MySQL at startup and values with supersede the any value defined in the my.cnf file or MySQL’s default directives.

To implement this management capability, a database instance may be optionally associated to a configuration when the instance is created or through associating a configuration to the instance after it has been created.

A configuration is a collection of key / value pairs where the valid key/values are defined in the MySQL manual at http://dev.mysql.com/doc/refman/5.1/en/server-system-variables.html. Some directives are capable of being applied dynamically, while other directives require a server restart to take effect.

When a configuration is modified either by way of adding a new key/value pair or modifying an existing key/value pair, the service will update the overrides.cnf file for every instance that is associated to the configuration and attempt to dynamically update running MySQL instances if the value is allowed to be modified dynamically.

The MySQL configuration management feature will integrate with the quotas capability defined in the following blueprint that has been approved https://blueprints.launchpad.net/trove/+spec/quotas. There should be two quotas established (1) maximum number of configuration groups per tenant and (2) maximum number of values per configuration group.

--hubcap (talk) 14:49, 21 March 2013 (UTC)We should create an independent BP For this thats dependent on this BP.

The system allows for the operator to configure the validation rules for key/value pairs that can be assigned to a configuration group. These rules can be retrieved by the user from the API through the configuration-parameters resource as described in the API documentation below. These validation rules are defined in the configuration-validation.conf file.

Service Options

Mysql

A list of the current configuration options for mysql can be found here.

Cassandra

A list of the current configuration options for cassandra can be found here

API Resources

Two new resources, configurations and configuration-parameters will be exposed as part of the Trove API.

The configurations resource will allow users to create a new configuration group, which contains a user-defined name, description, and set of key value pairs that will be written to the overrides.cnf file for MySQL.

The configurations resource provides the following data:

  • id – A system-defined UUID assigned to the configuration group when it is created for the first time.
  • name – A user-defined name for the configuration group.
  • description – A user-defined description for the configuration group (optional).
  • values – A list of key/value pairs assigned to the configuration group.
  • instances - A list of instances (if applicable) associated to the configuration group.


The configuration-parameters resource is used to provide the user with metadata that includes the following information:

  • name – Name of the key.
  • description – A description of the option/setting.
  • default - The default value of the key as supplied by MySQL. The service provider may choose to override certain MySQL defaults based on the size of the instance or the configuration of the infrastructure. The default will represent the MySQL default.
  • configurable – a boolean value to indicate whether or not the service provider allows for the value to be tuned. True indicates that a configuration group is permitted to override the value.
  • dynamic – a boolean value to indicate whether a value can be applied or updated while MySQL is running.
  • type – Identifies what type of data is expected for the value of the key (Boolean, String, Integer)


Optionally, the following attributes will be provided in the response for values that are expected to be integers:

  • minimum - The minimum value permitted by the service.
  • maximum – The maximum value permitted by the service.

API Details: Create / Modify / List Instance

To implement this capability, the create/modify/list instance operations will be extended in a manner that does not break the existing 1.0 contract. These operations will permit a user to associate a configuration group to a new or existing instance and determine what (if any) configuration group has been associated to a running instance through the existing operations.

Verb Resource Description
GET /instances/{id}/configuration 'New Operation – Allow user to see the "default" mysql configuration settings from template applied to the instance.
POST /instances Existing operation – extended to allow user to include configuration group identifier in message body
PUT /instances/{id} New Operation – This operation does not currently exist and will be created to allow a user to associate an existing instance to a configuration group. This operation can be extended in the future to allow changes to other mutable properties, like the instance name.
GET /instances/{id} "Existing operation" - extended to provide the user with configuration group id/name if the running instance has been associated to a configuration group.

Extensions to the message body are colored red configurationRef is an optional argument. Standard response codes / validation is expected in all operations.

--hubcap (talk) 15:35, 21 March 2013 (UTC)should this just be the config uuid?

Get an Instance Default Configuration (GET)

This is the default configurations that are applied to the instance according the the flavor assigned.

Response

{
   "instance": {
       "configuration": {
           "basedir": "/usr",
           "connect_timeout": "15",
           "datadir": "/var/lib/mysql",
           "default_storage_engine": "innodb",
           "innodb_buffer_pool_size": "150M",
           "innodb_data_file_path": "ibdata1:10M:autoextend",
           "innodb_file_per_table": "1",
           "innodb_log_buffer_size": "25M",
           "innodb_log_file_size": "50M",
           "innodb_log_files_in_group": "2",
           "join_buffer_size": "1M",
           "key_buffer_size": "50M",
           "local-infile": "0",
           "max_allowed_packet": "1M",
           "max_connections": "100",
           "max_heap_table_size": "16M",
           "max_user_connections": "100",
           "myisam-recover": "BACKUP",
           "open_files_limit": "512",
           "pid_file": "/var/run/mysqld/mysqld.pid",
           "port": "3306",
           "query_cache_limit": "1M",
           "query_cache_size": "8M",
           "query_cache_type": "1",
           "read_buffer_size": "512K",
           "read_rnd_buffer_size": "512K",
           "server_id": "206237194",
           "skip-external-locking": null,
           "sort_buffer_size": "1M",
           "table_definition_cache": "256",
           "table_open_cache": "256",
           "thread_cache_size": "4",
           "thread_stack": "192K",
           "tmp_table_size": "16M",
           "tmpdir": "/var/tmp",
           "user": "mysql",
           "wait_timeout": "120"
       }
   }

}

Create an Instance (POST)

Request

{
   "instance": {
       "name": "mysql_instance", 
       "flavorRef": "https://endpoint/v1.0/1234/flavors/1", 
       "volume": {
           "size": 2
       }
       "configuration": "12345678-1111-2222-3333-444444444444", 
   }
}

Response

{
   "instance": {
       "created": "2012-01-25T21:53:09Z", 
       "flavor": {
           "id": "1", 
           "links": [
               {
                   "href": "https://endpoint/v1.0/1234/flavors/1", 
                   "rel": "self"
               }, 
               {
                   "href": "https://endpoint/flavors/1", 
                   "rel": "bookmark"
               }
           ]
       },
       "configuration": {
          "id": "12345678-1111-2222-3333-444444444444",
          "name": "MySQL Tuned Config",
          "links": [
              {
                  "href": "https://endpoint/v1.0/1234/configurations/12345678-1111-2222-3333-444444444444", 
                  "rel": "self"
              }, 
              {
                  "href": "https://endpoint/configurations/12345678-1111-2222-3333-444444444444", 
                  "rel": "bookmark"
              }
          ]
      },
       "hostname": "e09ad9a3f73309469cf1f43d11e79549caf9acf2.hostname", 
       "id": "dea5a2f7-3ec7-4496-adab-0abb5a42d635", 
       "links": [
           {
               "href": "https://endpoint/v1.0/1234/instances/dea5a2f7-3ec7-4496-adab-0abb5a42d635", 
               "rel": "self"
           }, 
           {
               "href": "https://endpoint/instances/dea5a2f7-3ec7-4496-adab-0abb5a42d635", 
               "rel": "bookmark"
           }
       ], 
       "name": "json_rack_instance", 
       "status": "BUILD", 
       "updated": "2012-01-25T21:53:10Z", 
       "volume": {
           "size": 2
       }
   }
}

Update an Instance (PUT)

Request

{
   "instance": {
       "configuration": "12345678-1111-2222-3333-444444444444", 
   }
}

Get an Instance (GET)

Response {

   "instance": {
       "created": "2012-01-25T21:53:09Z", 
       "flavor": {
           "id": "1", 
           "links": [
               {
                   "href": "https://endpoint/v1.0/1234/flavors/1", 
                   "rel": "self"
               }, 
               {
                   "href": "https://endpoint/flavors/1", 
                   "rel": "bookmark"
               }
           ]
       },
       "configuration": {
           "id": "12345678-1111-2222-3333-444444444444",
           "name": "MySQL Tuned Config",
           "links": [
               {
                   "href": "https://endpoint/v1.0/1234/configurations/12345678-1111-2222-3333-444444444444", 
                   "rel": "self"
               }, 
               {
                   "href": "https://endpoint/configurations/12345678-1111-2222-3333-444444444444", 
                   "rel": "bookmark"
               }
           ]
       },
       "hostname": "e09ad9a3f73309469cf1f43d11e79549caf9acf2.hostname", 
       "id": "dea5a2f7-3ec7-4496-adab-0abb5a42d635", 
       "links": [
           {
               "href": "https://endpoint/v1.0/1234/instances/dea5a2f7-3ec7-4496-adab-0abb5a42d635", 
               "rel": "self"
           }, 
           {
               "href": "https://endpoint/instances/dea5a2f7-3ec7-4496-adab-0abb5a42d635", 
               "rel": "bookmark"
           }
       ], 
       "name": "json_rack_instance", 
       "status": "BUILD", 
       "updated": "2012-01-25T21:53:10Z", 
       "volume": {
           "size": 2
       }
   }
}

API Details: Configuration

Verb Resource Description
GET /configurations Returns of list of defined configuration groups for the tenant.
GET /configurations/{id} Returns the specified configuration group.
GET /configurations/{id}/instances Returns a list of instances that are associated with the configuration
POST /configurations Creates a new configuration group based on the parameters supplied in the request body.
PATCH /configurations/{id} Update some of the key/value pairs associated with the configuration group.
PUT /configurations/{id} Replace all the key/value pairs associated with the configuration group.
DELETE /configurations/{id} Delete a configuration group. No message body is expected in the request.

--hubcap (talk) 15:34, 21 March 2013 (UTC)Id prefer /configurations --jrodom 03:02, 12 April 2013 (UTC) Agreed, this was a bad c/p. fixed.

Get Configuration (GET)

Request

(no message body)

Response

{
   "configurations": {
       "id": "12345678-1111-2222-3333-444444444444",
       "name": "Enhanced Performance Settings",
       "description": "Performance Enhancing Settings",
       "values": {
           "ft_min_word_len": "2",
           "key_buffer_size": "300M"
       },
       "instances": [
           {
               "id": "dea5a2f7-3ec7-4496-adab-0abb5a42d635",
               "name": "json_rack_instance"
           }
       ]
   }
}

Get Configurations (GET)

Request

(no message body)

Response

{
   "configurations": [
       {
           "id": "12345678-1111-2222-3333-444444444444",
           "name": "Enhanced Performance Settings",
           "description": "Performance Enhancing Settings"
       },
       {
           "id": "11111111-2222-3333-4444-444444444444",
           "name": "Another great configuration group",
           "description": "Super Speed MySQL"
       }
   ]
}

Create a Configuration (POST)

When creating a configuration, the caller must supply a name and at least one value. The description is optional.

Request

{ 
  "configuration" : {
     "name" : "Enhanced Performance Settings",
     "description" : "Performance Enhancing Settings",
     "values" : { 
         "ft_min_word_len" : "2",
         "key_buffer_size" : "300M"
       }
   } 
}

Response

{ 
  "configuration" : {
     “id” : “12345678-1111-2222-3333-444444444444”,
     "name" : "Enhanced Performance Settings",
     "description" : "Performance Enhancing Settings",
     "values" : { 
        "ft_min_word_len" : "2",
         "key_buffer_size" : "300M"
       }
   } 
}

Update a Configuration Parameter (GET)

When updating a configuration, the user must supply the id along with any attributes that require updates. When updating the values array, the caller should supply a complete list of values, which will replace the existing array in its entirety.

Request

{ 
  "configuration" : {
     "values" : { 
         "join_buffer_size": 1048576
       }
   } 
}


Response

(No response body)

Update all Configuration Parameters (PUT)

When updating a configuration, the user must supply the id along with any attributes that require updates. When updating the values array, the caller should supply a complete list of values, which will replace the existing array in its entirety.

Request

{ 
  "configuration" : {
     "values" : { 
         "key_buffer_size" : 102400
       }
   } 
}


Response

{ 
  "configuration" : {
     “id” : “12345678-1111-2222-3333-444444444444”,
     "name" : "Enhanced Performance Settings",
     "description" : "Performance Enhancing Settings",
     "values" : { 
         "key_buffer_size" : 102400
       }
   } 
}

API Details: Configuration Parameters

Configuration Parameters are stored in the database Trove/Configurations#Datastore_Configuration_Parameters. Defined here are the api URI's that allow a user to access the configurable parameters for a datastore version. There will be a separate management call that allows you to import the datastore configuration parameters into the database.Trove/Configurations#Management_Commands

Verb Resource Description
GET /datastores/{datastore}/versions/{version}/parameters Supplies a list of configuration keys that may be configured on the system.
GET /datastores/{datastore}/versions/{version}/parameters/{key} Get the details of a configuration key that may be configured on the system.

Request

(no message body)

Response

{
  "parameters":[
     {
        "name":"autocommit",
        "description":"Enable or disable MySQLs autocommit feature",
        "default":"true",
        "configurable":"true",
        "dynamic":"true",
        "type":"boolean"
     }
  ]
}

Database Schema

Two new entities will be created in the trove database: configuration_group and configuration_item. These entities will store the configuration group data that is supplied by the user.

Configuration Group (configuration_group)

This table will contain the id, name, description, and tenant id the group belongs to.

Name Data Type Length Nullable Details
id VARCHAR 36 False Primary Key, Generated UUID
name VARCHAR 64 False -
description VARCHAR 256 True -
tenant_id VARCHAR 36 False -

Configuration Item (configuration_item)

This table maintains a many-to-one relationship to configuration_group via the foreign key reference to configuration_group_id

Name Data Type Length Nullable Details
configuration_group_id VARCHAR 36 False Foreign Key reference to configuration_group.id
key VARCHAR 128 False -
value VARCHAR 128 False -

Datastore Configuration Parameters

This table contains the list of all the Configuration Parameters and associates them with a specific datastore version. Trove/Configurations#API_Details:_Configuration_Parameters

Name Data Type Length Nullable Details
id VARCHAR 36 False Primary Key, Generated UUID
name VARCHAR 256 False -
datastore_version_id VARCHAR 36 False -
restart_required BOOLEAN 1 False Default=True
max_size VARCHAR 40 True -
min_size VARCHAR 40 True -
data_type VARCHAR 256 False -
deleted BOOLEAN 1 False Default=False
deleted_at DATETIME - True -

Management Commands

The trove-manage cmd will have a way of importing a json file with a list of configuration parameters to make it easier to load in a ton of parameters for each datastore version.

trove-manage load_datastore_config_parameters [datastore_version_id] [full path to the json file]