Jump to: navigation, search

Difference between revisions of "Trove/ConfigurableDBPlugins"

Line 46: Line 46:
  
 
Obviously at this point you are probably wondering about migration and munking up troves tables / schema. However the way I've done this in a PoC allows the db plugin schema and migration impl to exist in parallel to trove's. Here's an overview of how that would look from a consumer perspective wanting to plug into trove's db:
 
Obviously at this point you are probably wondering about migration and munking up troves tables / schema. However the way I've done this in a PoC allows the db plugin schema and migration impl to exist in parallel to trove's. Here's an overview of how that would look from a consumer perspective wanting to plug into trove's db:
- Consumer creates their own migrate_repo artifacts (separate from troves) including:
+
* Consumer creates their own migrate_repo artifacts (separate from troves) including:
- migrate.cnf with own version_table and repository_id
+
** migrate.cnf with own version_table and repository_id
- schema.py if needed
+
** schema.py if needed
- migrate version py files.. for example com.foo.dbaas.db.migate_repo.versions.001_myplugin.py
+
** migrate version py files.. for example com.foo.dbaas.db.migate_repo.versions.001_myplugin.py
- Consumer defines their plugin mapper. for example: com.foo.dbaas.db.plugin.py (see my previous description on what the mapper does)
+
* Consumer defines their plugin mapper. for example: com.foo.dbaas.db.plugin.py (see my previous description on what the mapper does)
- Consumer adds their plugin to the trove.conf: db_plugins = com.foo.dbaas.db.plugin.DBPlugins
+
* Consumer adds their plugin to the trove.conf: db_plugins = com.foo.dbaas.db.plugin.DBPlugins
- Consumer uses trove-manage to create their schema (1-time action upon install). e.g. trove-manage --config-file=/etc/trove/trove.conf db_sync --repo_path=/path/to/com/foo/dbaas/db/plugin/migrate_repo
+
* Consumer uses trove-manage to create their schema (1-time action upon install). e.g. trove-manage --config-file=/etc/trove/trove.conf db_sync --repo_path=/path/to/com/foo/dbaas/db/plugin/migrate_repo
- Now when trove is started it imports the plugin object from CONF.db_plugins and passes that to configure_db() method... everything just works now
+
* Now when trove is started it imports the plugin object from CONF.db_plugins and passes that to configure_db() method... everything just works now
  
 
Note in the above, trove-manage already supports pointing to a different migrate_repo path.  
 
Note in the above, trove-manage already supports pointing to a different migrate_repo path.  
Line 60: Line 60:
  
 
== Justification/Benefits ==
 
== Justification/Benefits ==
* What is the driving force behind this change
+
* Allows consumers / vendors to streamline development or prototypes for custom non-standard "add-ons" to trove which may or may not be upstreamed at some future point by allowing them to leverage the existing (proven) trove db models / abstractions /etc. without having to change trove proper code.
* Does it allow for great flexibility? Stability? Security?
+
* Exposes existing plug-point in trove code which is currently severed at the cmd main entry point (common.py).
 +
 
  
 
== Impacts ==
 
== Impacts ==
 +
Very minimal impacts including:
 +
* A small change to trove.cmd.common.py to load any CONF.db_plugins class objects and them pass them down on get_db_api().configure_db(conf)
 +
* A new conf comma list property
  
 
=== Configuration ===  
 
=== Configuration ===  
* Does this impact any configuration files? If so, which ones?
+
* Proposing a new comma list property be supported on all controller (non-agent) based services: CONF.db_plugins
 +
* By default the proper is empty list ([])
  
 
=== Database ===
 
=== Database ===
* Does this impact any existing tables?  If so, which ones?
+
* No changes to trove proper tables / schema
* Are the changes forward and backward compatible?
 
* Be sure to include the expected migration process
 
  
 
=== Public API ===
 
=== Public API ===
* Does this change any API that an end-user has access to?
+
* No changes to public API
* Are there any exceptions in terms of consistency with other APIs?
 
  
 
==== CLI interface====
 
==== CLI interface====
* How the command will look like?
+
* No new CLI opts needed
* Does it extends the already existed command interfaces ?
+
* Can leverage existing support in trove-manage to specify a repo-path (already works with trove today)
  
 
==== ReST Part ====
 
==== ReST Part ====
* Which HTTP methods added ?
+
* No new REST APIs
* Which routes were added/modified/extended?
 
* How does the Request body look like?
 
* How does the Response object look like?
 
  
 
=== Internal API ===
 
=== Internal API ===
* Does this change any internal messages between API and Task Manager or Task Manager to Guest
+
* No changes to internal APIs
 +
 
 
==== RPC API description====
 
==== RPC API description====
* Method name.
+
* No changes to RPC
* Method parameters.
 
* Message type (cast/call).
 
  
 
=== Guest Agent ===
 
=== Guest Agent ===
* Does this change behavior on the Guest Agent? If so, is it backwards compatible with API and Task Manager?
+
* No guest agent changes

Revision as of 12:50, 3 June 2014

Description

From a vendor / consumer perspective, there are likely cases where consumers want to leverage the existing trove sqlachemy DB models and framework abstractions for their own extensions / features within the trove framework. For example a consumer may want to develop custom in-house (proprietary) add-ons which use persistence for trove which either they do not wish to contribute upstream, or they want to build out a PoC in-house before upstreaming. In such cases a more rapid time to value and lower risk investment can be achieved by leveraging the existing trove db framework and having the ability to plug-in their own schema / migration / etc..

The current trove sqlalchemy implementation contains the plumbing and support necessary to allow consumers to "plug-in" their own database mappers which in turn can define their own ORM mappings, schema, etc.. The hook point for such extensions in python can be found in trove.db.sqlalchemy.api.py in the configure_db() method which looks like this:

def configure_db(options, *plugins):
   session.configure_db(options)
   configure_db_for_plugins(options, *plugins)

Here any number of "plugins" can be passed to the configure_db() function allowing consumers to plug into trove's sqlalchemy ORM engine. In the current impl, the plugin is just a python object which containers a 'mapper' attribute which defines the map(self, engine) method which of course defines ORM mappings atop troves sqlalchemy engine.

For example you could define this simple custom DB plugin / mapper (fictional):

class Mapper(object):
   def map(self, engine):
       meta = MetaData()
       meta.bind = engine
       if mappers.mapping_exists(my_models.Person):
           return
       orm.mapper(my_models.Person,
                  Table('person', meta, autoload=True))


class DBPlugins(object):
   def __init__(self):
       self.mapper = Mapper()


which can then be added to trove's ORM using:

from trove.db import get_db_api
get_db_api().configure_db(CONF, DBPlugins())


This is all fine, and everything shown above exists already in trove to date. However what's missing is the ability for consumers to pass in their 'plugins' via the main entry points. Instead the entry points do not permit passing any plugins to def configure_db(options, *plugins). See: https://github.com/openstack/trove/blob/master/trove/cmd/common.py#L52

Whats being propose here is: (a) Support a comma list property on CONF.DEFAULT in the trove conf files. e.g.

[DEFAULT]
db_plugins = org.foo.bar.sqlalchemy.BarPlugins,org.yadda.sqlalchemy.MyPlugins

(b) Update the common.py entry point (see link above) to load each of the CONF.db_plugins as an object and pass them to configure_db() in the common.py (linked above).

a-b above would permit consumers to plug into troves ORM.


Obviously at this point you are probably wondering about migration and munking up troves tables / schema. However the way I've done this in a PoC allows the db plugin schema and migration impl to exist in parallel to trove's. Here's an overview of how that would look from a consumer perspective wanting to plug into trove's db:

  • Consumer creates their own migrate_repo artifacts (separate from troves) including:
    • migrate.cnf with own version_table and repository_id
    • schema.py if needed
    • migrate version py files.. for example com.foo.dbaas.db.migate_repo.versions.001_myplugin.py
  • Consumer defines their plugin mapper. for example: com.foo.dbaas.db.plugin.py (see my previous description on what the mapper does)
  • Consumer adds their plugin to the trove.conf: db_plugins = com.foo.dbaas.db.plugin.DBPlugins
  • Consumer uses trove-manage to create their schema (1-time action upon install). e.g. trove-manage --config-file=/etc/trove/trove.conf db_sync --repo_path=/path/to/com/foo/dbaas/db/plugin/migrate_repo
  • Now when trove is started it imports the plugin object from CONF.db_plugins and passes that to configure_db() method... everything just works now

Note in the above, trove-manage already supports pointing to a different migrate_repo path.

As you can see in the above, the consumer manages their own db schema, migration, etc. and it just resides happily in trove's db next to the trove proper schema.

Justification/Benefits

  • Allows consumers / vendors to streamline development or prototypes for custom non-standard "add-ons" to trove which may or may not be upstreamed at some future point by allowing them to leverage the existing (proven) trove db models / abstractions /etc. without having to change trove proper code.
  • Exposes existing plug-point in trove code which is currently severed at the cmd main entry point (common.py).


Impacts

Very minimal impacts including:

  • A small change to trove.cmd.common.py to load any CONF.db_plugins class objects and them pass them down on get_db_api().configure_db(conf)
  • A new conf comma list property

Configuration

  • Proposing a new comma list property be supported on all controller (non-agent) based services: CONF.db_plugins
  • By default the proper is empty list ([])

Database

  • No changes to trove proper tables / schema

Public API

  • No changes to public API

CLI interface

  • No new CLI opts needed
  • Can leverage existing support in trove-manage to specify a repo-path (already works with trove today)

ReST Part

  • No new REST APIs

Internal API

  • No changes to internal APIs

RPC API description

  • No changes to RPC

Guest Agent

  • No guest agent changes