Trove/Configurations

Overview
The MySQL configuration management capability will allow users of Trove to override the default MySQL configuration settings provided by the operator of the Trove service. This is implemented by leveraging MySQL’s includedir directive, which is already pre-defined to point to /etc/mysql/conf.d. The configuration files stored (or symlinked) into this directory will be interpreted by MySQL at startup and values with supersede the any value defined in the my.cnf file or MySQL’s default directives.

To implement this management capability, a database instance may be optionally associated to a configuration when the instance is created or through associating a configuration to the instance after it has been created.

A configuration is a collection of key / value pairs where the valid key/values are defined in the MySQL manual at http://dev.mysql.com/doc/refman/5.1/en/server-system-variables.html. Some directives are capable of being applied dynamically, while other directives require a server restart to take effect.

When a configuration is modified either by way of adding a new key/value pair or modifying an existing key/value pair, the service will update the overrides.cnf file for every instance that is associated to the configuration and attempt to dynamically update running MySQL instances if the value is allowed to be modified dynamically.

The MySQL configuration management feature will integrate with the quotas capability defined in the following blueprint that has been approved https://blueprints.launchpad.net/trove/+spec/quotas. There should be two quotas established (1) maximum number of configuration groups per tenant and (2) maximum number of values per configuration group.

--hubcap (talk) 14:49, 21 March 2013 (UTC)We should create an independent BP For this thats dependent on this BP.

The system allows for the operator to configure the validation rules for key/value pairs that can be assigned to a configuration group. These rules can be retrieved by the user from the API through the configuration-parameters resource as described in the API documentation below. These validation rules are defined in the configuration-validation.conf file.

Mysql
A list of the current configuration options for mysql can be found here.

Cassandra
A list of the current configuration options for cassandra can be found here

API Resources
Two new resources, configurations and configuration-parameters will be exposed as part of the Trove API.

The configurations resource will allow users to create a new configuration group, which contains a user-defined name, description, and set of key value pairs that will be written to the overrides.cnf file for MySQL.

The configurations resource provides the following data:


 * id – A system-defined UUID assigned to the configuration group when it is created for the first time.
 * name – A user-defined name for the configuration group.
 * description – A user-defined description for the configuration group (optional).
 * values – A list of key/value pairs assigned to the configuration group.
 * instances - A list of instances (if applicable) associated to the configuration group.

The configuration-parameters resource is used to provide the user with metadata that includes the following information:


 * name – Name of the key.
 * description – A description of the option/setting.
 * default - The default value of the key as supplied by MySQL.  The service provider may choose to override certain MySQL defaults based on the size of the instance or the configuration of the infrastructure.  The default will represent the MySQL default.
 * configurable – a boolean value to indicate whether or not the service provider allows for the value to be tuned. True indicates that a configuration group is permitted to override the value.
 * dynamic – a boolean value to indicate whether a value can be applied or updated while MySQL is running.
 * type – Identifies what type of data is expected for the value of the key (Boolean, String, Integer)

Optionally, the following attributes will be provided in the response for values that are expected to be integers:


 * minimum - The minimum value permitted by the service.
 * maximum – The maximum value permitted by the service.

API Details: Create / Modify / List Instance
To implement this capability, the create/modify/list instance operations will be extended in a manner that does not break the existing 1.0 contract. These operations will permit a user to associate a configuration group to a new or existing instance and determine what (if any) configuration group has been associated to a running instance through the existing operations.

''Extensions to the message body are colored red configurationRef is an optional argument. Standard response codes / validation is expected in all operations.''

--hubcap (talk) 15:35, 21 March 2013 (UTC)should this just be the config uuid?

Get an Instance Default Configuration (GET)
This is the default configurations that are applied to the instance according the the flavor assigned.

Response {   "instance": { "configuration": { "basedir": "/usr", "connect_timeout": "15", "datadir": "/var/lib/mysql", "default_storage_engine": "innodb", "innodb_buffer_pool_size": "150M", "innodb_data_file_path": "ibdata1:10M:autoextend", "innodb_file_per_table": "1", "innodb_log_buffer_size": "25M", "innodb_log_file_size": "50M", "innodb_log_files_in_group": "2", "join_buffer_size": "1M", "key_buffer_size": "50M", "local-infile": "0", "max_allowed_packet": "1M", "max_connections": "100", "max_heap_table_size": "16M", "max_user_connections": "100", "myisam-recover": "BACKUP", "open_files_limit": "512", "pid_file": "/var/run/mysqld/mysqld.pid", "port": "3306", "query_cache_limit": "1M", "query_cache_size": "8M", "query_cache_type": "1", "read_buffer_size": "512K", "read_rnd_buffer_size": "512K", "server_id": "206237194", "skip-external-locking": null, "sort_buffer_size": "1M", "table_definition_cache": "256", "table_open_cache": "256", "thread_cache_size": "4", "thread_stack": "192K", "tmp_table_size": "16M", "tmpdir": "/var/tmp", "user": "mysql", "wait_timeout": "120" }   } }

Create an Instance (POST)
Request {   "instance": { "name": "mysql_instance", "flavorRef": "https://endpoint/v1.0/1234/flavors/1", "volume": { "size": 2 }       "configuration": "12345678-1111-2222-3333-444444444444", } } Response {   "instance": { "created": "2012-01-25T21:53:09Z", "flavor": { "id": "1", "links": [ {                   "href": "https://endpoint/v1.0/1234/flavors/1", "rel": "self" },                {                    "href": "https://endpoint/flavors/1", "rel": "bookmark" }           ]        },        "configuration": { "id": "12345678-1111-2222-3333-444444444444", "name": "MySQL Tuned Config", "links": [ {                  "href": "https://endpoint/v1.0/1234/configurations/12345678-1111-2222-3333-444444444444", "rel": "self" },               {                   "href": "https://endpoint/configurations/12345678-1111-2222-3333-444444444444", "rel": "bookmark" }          ]       },         "hostname": "e09ad9a3f73309469cf1f43d11e79549caf9acf2.hostname", "id": "dea5a2f7-3ec7-4496-adab-0abb5a42d635", "links": [ {               "href": "https://endpoint/v1.0/1234/instances/dea5a2f7-3ec7-4496-adab-0abb5a42d635", "rel": "self" },            {                "href": "https://endpoint/instances/dea5a2f7-3ec7-4496-adab-0abb5a42d635", "rel": "bookmark" }       ],         "name": "json_rack_instance", "status": "BUILD", "updated": "2012-01-25T21:53:10Z", "volume": { "size": 2 }   } }

Update an Instance (PUT)
Request {   "instance": { "configuration": "12345678-1111-2222-3333-444444444444", } }

Get an Instance (GET)
Response {   "instance": { "created": "2012-01-25T21:53:09Z", "flavor": { "id": "1", "links": [ {                   "href": "https://endpoint/v1.0/1234/flavors/1", "rel": "self" },                {                    "href": "https://endpoint/flavors/1", "rel": "bookmark" }           ]        },        "configuration": { "id": "12345678-1111-2222-3333-444444444444", "name": "MySQL Tuned Config", "links": [ {                   "href": "https://endpoint/v1.0/1234/configurations/12345678-1111-2222-3333-444444444444", "rel": "self" },                {                    "href": "https://endpoint/configurations/12345678-1111-2222-3333-444444444444", "rel": "bookmark" }           ]        },         "hostname": "e09ad9a3f73309469cf1f43d11e79549caf9acf2.hostname", "id": "dea5a2f7-3ec7-4496-adab-0abb5a42d635", "links": [ {               "href": "https://endpoint/v1.0/1234/instances/dea5a2f7-3ec7-4496-adab-0abb5a42d635", "rel": "self" },            {                "href": "https://endpoint/instances/dea5a2f7-3ec7-4496-adab-0abb5a42d635", "rel": "bookmark" }       ],         "name": "json_rack_instance", "status": "BUILD", "updated": "2012-01-25T21:53:10Z", "volume": { "size": 2 }   } }

API Details: Configuration
--hubcap (talk) 15:34, 21 March 2013 (UTC)Id prefer /configurations --jrodom 03:02, 12 April 2013 (UTC) Agreed, this was a bad c/p. fixed.

Get Configuration (GET)
Request

(no message body)

Response {   "configurations": { "id": "12345678-1111-2222-3333-444444444444", "name": "Enhanced Performance Settings", "description": "Performance Enhancing Settings", "values": { "ft_min_word_len": "2", "key_buffer_size": "300M" },       "instances": [ {               "id": "dea5a2f7-3ec7-4496-adab-0abb5a42d635", "name": "json_rack_instance" }       ]    } }

Get Configurations (GET)
Request

(no message body)

Response {   "configurations": [ {           "id": "12345678-1111-2222-3333-444444444444", "name": "Enhanced Performance Settings", "description": "Performance Enhancing Settings" },       {            "id": "11111111-2222-3333-4444-444444444444", "name": "Another great configuration group", "description": "Super Speed MySQL" }   ] }

Create a Configuration (POST)
When creating a configuration, the caller must supply a name and at least one value. The description is optional.

Request {   "configuration" : { "name" : "Enhanced Performance Settings", "description" : "Performance Enhancing Settings", "values" : { "ft_min_word_len" : "2", "key_buffer_size" : "300M" }   }  }

Response {   "configuration" : { “id” : “12345678-1111-2222-3333-444444444444”, "name" : "Enhanced Performance Settings", "description" : "Performance Enhancing Settings", "values" : { "ft_min_word_len" : "2", "key_buffer_size" : "300M" }   }  }

Update a Configuration Parameter (GET)
When updating a configuration, the user must supply the id along with any attributes that require updates. When updating the values array, the caller should supply a complete list of values, which will replace the existing array in its entirety.

Request {   "configuration" : { "values" : { "join_buffer_size": 1048576 }   }  }

Response

(No response body)

Update all Configuration Parameters (PUT)
When updating a configuration, the user must supply the id along with any attributes that require updates. When updating the values array, the caller should supply a complete list of values, which will replace the existing array in its entirety.

Request {   "configuration" : { "values" : { "key_buffer_size" : 102400 }   }  }

Response {   "configuration" : { “id” : “12345678-1111-2222-3333-444444444444”, "name" : "Enhanced Performance Settings", "description" : "Performance Enhancing Settings", "values" : { "key_buffer_size" : 102400 }   }  }

API Details: Configuration Parameters
Configuration Parameters are stored in the database Trove/Configurations. Defined here are the api URI's that allow a user to access the configurable parameters for a datastore version. There will be a separate management call that allows you to import the datastore configuration parameters into the database.Trove/Configurations

Request

(no message body)

Response {  "parameters":[ {        "name":"autocommit", "description":"Enable or disable MySQLs autocommit feature", "default":"true", "configurable":"true", "dynamic":"true", "type":"boolean" }  ] }

Database Schema
Two new entities will be created in the trove database: configuration_group and configuration_item. These entities will store the configuration group data that is supplied by the user.

Configuration Group (configuration_group)
This table will contain the id, name, description, and tenant id the group belongs to.

Configuration Item (configuration_item)
This table maintains a many-to-one relationship to configuration_group via the foreign key reference to configuration_group_id

Datastore Configuration Parameters
This table contains the list of all the Configuration Parameters and associates them with a specific datastore version. Trove/Configurations

Management Commands
The trove-manage cmd will have a way of importing a json file with a list of configuration parameters to make it easier to load in a ton of parameters for each datastore version.

trove-manage load_datastore_config_parameters [datastore_version_id] [full path to the json file]