Jump to: navigation, search

Difference between revisions of "Trove/ConfigurableDBPlugins"

Line 4: Line 4:
 
The current trove sqlalchemy implementation contains the plumbing and support necessary to allow consumers to "plug-in" their own database mappers which in turn can define their own ORM mappings, schema, etc.. The hook point for such extensions in python can be found in trove.db.sqlalchemy.api.py in the configure_db() method which looks like this:
 
The current trove sqlalchemy implementation contains the plumbing and support necessary to allow consumers to "plug-in" their own database mappers which in turn can define their own ORM mappings, schema, etc.. The hook point for such extensions in python can be found in trove.db.sqlalchemy.api.py in the configure_db() method which looks like this:
  
def configure_db(options, *plugins):
+
def configure_db(options, *plugins):
 
     session.configure_db(options)
 
     session.configure_db(options)
 
     configure_db_for_plugins(options, *plugins)
 
     configure_db_for_plugins(options, *plugins)
Line 12: Line 12:
 
For example you could define this simple custom DB plugin / mapper (fictional):
 
For example you could define this simple custom DB plugin / mapper (fictional):
  
class Mapper(object):
+
class Mapper(object):
 
 
 
     def map(self, engine):
 
     def map(self, engine):
 
         meta = MetaData()
 
         meta = MetaData()
Line 23: Line 22:
  
  
class DBPlugins(object):
+
class DBPlugins(object):
 
 
 
     def __init__(self):
 
     def __init__(self):
 
         self.mapper = Mapper()
 
         self.mapper = Mapper()
Line 31: Line 29:
 
which can then be added to trove's ORM using:
 
which can then be added to trove's ORM using:
  
from trove.db import get_db_api
+
from trove.db import get_db_api
get_db_api().configure_db(CONF, DBPlugins())
+
get_db_api().configure_db(CONF, DBPlugins())
  
  
Line 39: Line 37:
 
Whats being propose here is:
 
Whats being propose here is:
 
(a) Support a comma list property on CONF.DEFAULT in the trove conf files. e.g.
 
(a) Support a comma list property on CONF.DEFAULT in the trove conf files. e.g.
[DEFAULT]
+
[DEFAULT]
db_plugins = org.foo.bar.sqlalchemy.BarPlugins,org.yadda.sqlalchemy.MyPlugins
+
db_plugins = org.foo.bar.sqlalchemy.BarPlugins,org.yadda.sqlalchemy.MyPlugins
  
 
(b) Update the common.py entry point (see link above) to load each of the CONF.db_plugins as an object and pass them to configure_db() in the common.py (linked above).
 
(b) Update the common.py entry point (see link above) to load each of the CONF.db_plugins as an object and pass them to configure_db() in the common.py (linked above).
Line 46: Line 44:
 
a-b above would permit consumers to plug into troves ORM.
 
a-b above would permit consumers to plug into troves ORM.
  
 +
 +
Obviously at this point you are probably wondering about migration and munking up troves tables / schema. However the way I've done this in a PoC allows the db plugin schema and migration impl to exist in parallel to trove's. Here's an overview of how that would look from a consumer perspective wanting to plug into trove's db:
 +
- Consumer creates their own migrate_repo artifacts (separate from troves) including:
 +
- migrate.cnf with own version_table and repository_id
 +
- schema.py if needed
 +
- migrate version py files.. for example com.foo.dbaas.db.migate_repo.versions.001_myplugin.py
 +
- Consumer defines their plugin mapper. for example: com.foo.dbaas.db.plugin.py (see my previous description on what the mapper does)
 +
- Consumer adds their plugin to the trove.conf: db_plugins = com.foo.dbaas.db.plugin.DBPlugins
 +
- Consumer uses trove-manage to create their schema (1-time action upon install). e.g. trove-manage --config-file=/etc/trove/trove.conf db_sync --repo_path=/path/to/com/foo/dbaas/db/plugin/migrate_repo
 +
- Now when trove is started it imports the plugin object from CONF.db_plugins and passes that to configure_db() method... everything just works now
 +
 +
Note in the above, trove-manage already supports pointing to a different migrate_repo path.
 +
 +
As you can see in the above, the consumer manages their own db schema, migration, etc. and it just resides happily in trove's db next to the trove proper schema.
  
 
== Justification/Benefits ==
 
== Justification/Benefits ==

Revision as of 12:42, 3 June 2014

Description

From a vendor / consumer perspective, there are likely cases where consumers want to leverage the existing trove sqlachemy DB models and framework abstractions for their own extensions / features within the trove framework. For example a consumer may want to develop custom in-house (proprietary) add-ons which use persistence for trove which either they do not wish to contribute upstream, or they want to build out a PoC in-house before upstreaming. In such cases a more rapid time to value and lower risk investment can be achieved by leveraging the existing trove db framework and having the ability to plug-in their own schema / migration / etc..

The current trove sqlalchemy implementation contains the plumbing and support necessary to allow consumers to "plug-in" their own database mappers which in turn can define their own ORM mappings, schema, etc.. The hook point for such extensions in python can be found in trove.db.sqlalchemy.api.py in the configure_db() method which looks like this:

def configure_db(options, *plugins):
   session.configure_db(options)
   configure_db_for_plugins(options, *plugins)

Here any number of "plugins" can be passed to the configure_db() function allowing consumers to plug into trove's sqlalchemy ORM engine. In the current impl, the plugin is just a python object which containers a 'mapper' attribute which defines the map(self, engine) method which of course defines ORM mappings atop troves sqlalchemy engine.

For example you could define this simple custom DB plugin / mapper (fictional):

class Mapper(object):
   def map(self, engine):
       meta = MetaData()
       meta.bind = engine
       if mappers.mapping_exists(my_models.Person):
           return
       orm.mapper(my_models.Person,
                  Table('person', meta, autoload=True))


class DBPlugins(object):
   def __init__(self):
       self.mapper = Mapper()


which can then be added to trove's ORM using:

from trove.db import get_db_api
get_db_api().configure_db(CONF, DBPlugins())


This is all fine, and everything shown above exists already in trove to date. However what's missing is the ability for consumers to pass in their 'plugins' via the main entry points. Instead the entry points do not permit passing any plugins to def configure_db(options, *plugins). See: https://github.com/openstack/trove/blob/master/trove/cmd/common.py#L52

Whats being propose here is: (a) Support a comma list property on CONF.DEFAULT in the trove conf files. e.g.

[DEFAULT]
db_plugins = org.foo.bar.sqlalchemy.BarPlugins,org.yadda.sqlalchemy.MyPlugins

(b) Update the common.py entry point (see link above) to load each of the CONF.db_plugins as an object and pass them to configure_db() in the common.py (linked above).

a-b above would permit consumers to plug into troves ORM.


Obviously at this point you are probably wondering about migration and munking up troves tables / schema. However the way I've done this in a PoC allows the db plugin schema and migration impl to exist in parallel to trove's. Here's an overview of how that would look from a consumer perspective wanting to plug into trove's db: - Consumer creates their own migrate_repo artifacts (separate from troves) including:

- migrate.cnf with own version_table and repository_id
- schema.py if needed
- migrate version py files.. for example com.foo.dbaas.db.migate_repo.versions.001_myplugin.py

- Consumer defines their plugin mapper. for example: com.foo.dbaas.db.plugin.py (see my previous description on what the mapper does) - Consumer adds their plugin to the trove.conf: db_plugins = com.foo.dbaas.db.plugin.DBPlugins - Consumer uses trove-manage to create their schema (1-time action upon install). e.g. trove-manage --config-file=/etc/trove/trove.conf db_sync --repo_path=/path/to/com/foo/dbaas/db/plugin/migrate_repo - Now when trove is started it imports the plugin object from CONF.db_plugins and passes that to configure_db() method... everything just works now

Note in the above, trove-manage already supports pointing to a different migrate_repo path.

As you can see in the above, the consumer manages their own db schema, migration, etc. and it just resides happily in trove's db next to the trove proper schema.

Justification/Benefits

  • What is the driving force behind this change?
  • Does it allow for great flexibility? Stability? Security?

Impacts

Configuration

  • Does this impact any configuration files? If so, which ones?

Database

  • Does this impact any existing tables? If so, which ones?
  • Are the changes forward and backward compatible?
  • Be sure to include the expected migration process

Public API

  • Does this change any API that an end-user has access to?
  • Are there any exceptions in terms of consistency with other APIs?

CLI interface

  • How the command will look like?
  • Does it extends the already existed command interfaces ?

ReST Part

  • Which HTTP methods added ?
  • Which routes were added/modified/extended?
  • How does the Request body look like?
  • How does the Response object look like?

Internal API

  • Does this change any internal messages between API and Task Manager or Task Manager to Guest

RPC API description

  • Method name.
  • Method parameters.
  • Message type (cast/call).

Guest Agent

  • Does this change behavior on the Guest Agent? If so, is it backwards compatible with API and Task Manager?