Jump to: navigation, search

Trove/ConfigurableDBPlugins

< Trove
Revision as of 12:50, 3 June 2014 by Boden (talk | contribs)

Description

From a vendor / consumer perspective, there are likely cases where consumers want to leverage the existing trove sqlachemy DB models and framework abstractions for their own extensions / features within the trove framework. For example a consumer may want to develop custom in-house (proprietary) add-ons which use persistence for trove which either they do not wish to contribute upstream, or they want to build out a PoC in-house before upstreaming. In such cases a more rapid time to value and lower risk investment can be achieved by leveraging the existing trove db framework and having the ability to plug-in their own schema / migration / etc..

The current trove sqlalchemy implementation contains the plumbing and support necessary to allow consumers to "plug-in" their own database mappers which in turn can define their own ORM mappings, schema, etc.. The hook point for such extensions in python can be found in trove.db.sqlalchemy.api.py in the configure_db() method which looks like this:

def configure_db(options, *plugins):
   session.configure_db(options)
   configure_db_for_plugins(options, *plugins)

Here any number of "plugins" can be passed to the configure_db() function allowing consumers to plug into trove's sqlalchemy ORM engine. In the current impl, the plugin is just a python object which containers a 'mapper' attribute which defines the map(self, engine) method which of course defines ORM mappings atop troves sqlalchemy engine.

For example you could define this simple custom DB plugin / mapper (fictional):

class Mapper(object):
   def map(self, engine):
       meta = MetaData()
       meta.bind = engine
       if mappers.mapping_exists(my_models.Person):
           return
       orm.mapper(my_models.Person,
                  Table('person', meta, autoload=True))


class DBPlugins(object):
   def __init__(self):
       self.mapper = Mapper()


which can then be added to trove's ORM using:

from trove.db import get_db_api
get_db_api().configure_db(CONF, DBPlugins())


This is all fine, and everything shown above exists already in trove to date. However what's missing is the ability for consumers to pass in their 'plugins' via the main entry points. Instead the entry points do not permit passing any plugins to def configure_db(options, *plugins). See: https://github.com/openstack/trove/blob/master/trove/cmd/common.py#L52

Whats being propose here is: (a) Support a comma list property on CONF.DEFAULT in the trove conf files. e.g.

[DEFAULT]
db_plugins = org.foo.bar.sqlalchemy.BarPlugins,org.yadda.sqlalchemy.MyPlugins

(b) Update the common.py entry point (see link above) to load each of the CONF.db_plugins as an object and pass them to configure_db() in the common.py (linked above).

a-b above would permit consumers to plug into troves ORM.


Obviously at this point you are probably wondering about migration and munking up troves tables / schema. However the way I've done this in a PoC allows the db plugin schema and migration impl to exist in parallel to trove's. Here's an overview of how that would look from a consumer perspective wanting to plug into trove's db:

  • Consumer creates their own migrate_repo artifacts (separate from troves) including:
    • migrate.cnf with own version_table and repository_id
    • schema.py if needed
    • migrate version py files.. for example com.foo.dbaas.db.migate_repo.versions.001_myplugin.py
  • Consumer defines their plugin mapper. for example: com.foo.dbaas.db.plugin.py (see my previous description on what the mapper does)
  • Consumer adds their plugin to the trove.conf: db_plugins = com.foo.dbaas.db.plugin.DBPlugins
  • Consumer uses trove-manage to create their schema (1-time action upon install). e.g. trove-manage --config-file=/etc/trove/trove.conf db_sync --repo_path=/path/to/com/foo/dbaas/db/plugin/migrate_repo
  • Now when trove is started it imports the plugin object from CONF.db_plugins and passes that to configure_db() method... everything just works now

Note in the above, trove-manage already supports pointing to a different migrate_repo path.

As you can see in the above, the consumer manages their own db schema, migration, etc. and it just resides happily in trove's db next to the trove proper schema.

Justification/Benefits

  • Allows consumers / vendors to streamline development or prototypes for custom non-standard "add-ons" to trove which may or may not be upstreamed at some future point by allowing them to leverage the existing (proven) trove db models / abstractions /etc. without having to change trove proper code.
  • Exposes existing plug-point in trove code which is currently severed at the cmd main entry point (common.py).


Impacts

Very minimal impacts including:

  • A small change to trove.cmd.common.py to load any CONF.db_plugins class objects and them pass them down on get_db_api().configure_db(conf)
  • A new conf comma list property

Configuration

  • Proposing a new comma list property be supported on all controller (non-agent) based services: CONF.db_plugins
  • By default the proper is empty list ([])

Database

  • No changes to trove proper tables / schema

Public API

  • No changes to public API

CLI interface

  • No new CLI opts needed
  • Can leverage existing support in trove-manage to specify a repo-path (already works with trove today)

ReST Part

  • No new REST APIs

Internal API

  • No changes to internal APIs

RPC API description

  • No changes to RPC

Guest Agent

  • No guest agent changes