Jump to: navigation, search

Difference between revisions of "Trove/DynamicExtensionLoading"

(Created page with "== Description == The current trove impl for loading API extensions is based on a single path which is specified in the trove conf file and then searched at trove-api start-up...")
 
 
(2 intermediate revisions by the same user not shown)
Line 1: Line 1:
 
== Description ==
 
== Description ==
The current trove impl for loading API extensions is based on a single path which is specified in the trove conf file and then searched at trove-api start-up time to find and load extensions. While this approach is sufficient for loading trove proper extensions it becomes more painful for consumers who wish to package and / or consume trove extensions not shipped with trove proper. With the current approach, a consumer must copy his / her extension class (a concrete impl of trove.common.extensions.ExtensionDescriptor) into the path specified by the api_extensions_path conf property in trove.conf in order for it to be picked up the the trove-api extension loading. The act of copying py files into another modules install path is a bad practice and this is not a very robust / flexible approach.
+
The current trove impl for loading API extensions is based on a single path which is specified in the trove conf file and then searched at trove-api start-up time to find and load extensions. While this approach is sufficient for loading trove proper extensions it becomes more painful for consumers who wish to package and / or consume trove extensions not shipped with trove proper. With the current approach, a consumer must copy his / her extension class (a concrete impl of trove.common.extensions.ExtensionDescriptor [1]) into the path specified by the api_extensions_path conf property[2] in trove.conf in order for it to be picked up the the trove-api extension loading [3]. The act of copying py files into another modules install path is a bad practice and this is not a very robust / flexible approach.
  
 
What's proposed here to move to a more dynamic means to discover and load extensions using the stevedore package: http://stevedore.readthedocs.org/en/latest/index.html
 
What's proposed here to move to a more dynamic means to discover and load extensions using the stevedore package: http://stevedore.readthedocs.org/en/latest/index.html
 +
 +
* [1] https://github.com/openstack/trove/blob/master/trove/openstack/common/extensions.py#L35
 +
* [2] https://github.com/openstack/trove/blob/master/trove/common/cfg.py#L40
 +
* [3] https://github.com/openstack/trove/blob/master/trove/openstack/common/extensions.py#L407
 +
  
 
== Justification/Benefits ==
 
== Justification/Benefits ==
* What is the driving force behind this change? 
+
A more dynamic / robust means to discover and load trove extensions will provide greater ease of use and flexibility for trove consumers as well as for trove proper. Moreover using stevedore to discover and load extensions will move trove proper more in line with how other core projects do things (e.g. nova and co). In particular:
* Does it allow for great flexibility? Stability? Security?
+
* Consumers can package and distribute trove extensions in separate python packages and the trove framework will automatically discover and load them without any changes to trove code or conf.
 +
* Trove proper development will not be constrained to only defining extensions in the fixed path / package of trove.extensions.routes
 +
* Use of stevedore will be more in line with with other core projects do and put trove in a closer position to updating our extension handling (https://blueprints.launchpad.net/trove/+spec/extensions-update).
  
 
== Impacts ==
 
== Impacts ==
 +
A simple PoC has been coded up and can be viewed here: https://github.com/bodenr/trove/commit/fa06e1d96e6a49a2a54057e8feb8e624edeaf728
 +
 +
As shown in the PoC minimal code changes are necessary to trove proper hence minimal risk from a code churn POV. The major impacts are as follows:
 +
* The api_extensions_path conf property is no longer needed.
 +
* Extensions are defined in the setup.cfg of python projects under a well defined extension key.
 +
* As per the PoC the base Extension class (ExtensionDescriptor) is now a "proper" abstract class (using py package abc) which provides stronger runtime checking... However this change is not required per say.
 +
* The extension loading code no longer needs to check a path and load init python classes -- this is done via stevedore.
 +
 +
However a potential impact which is not shown in the PoC is -- how do we handle upgrade for extensions. The concern here is that consumers may have extensions using the old (current) means whereupon they copied them into the extension path, but with the new approach these extensions need to be put into a well defined section of a setup.cfg. There are a few options for this consideration:
 +
* Doc the change assuming not many consumers have custom extensions.
 +
* Retain a legacy path in the code whereupon in addition to using the stevedore method to discover plugins we also still check the old-school path and only load plugins which were not loaded with stevedore... This provides backwards compatibility.
 +
* Have a simple CLI or code in the "upgrade" path which writes out a custom setup.cfg to capture any non-standard extensions and effectively migrate them to the new approach.
 +
* etc..
  
 
=== Configuration ===  
 
=== Configuration ===  
* Does this impact any configuration files? If so, which ones?
+
* trove.conf will no longer need the api_extensions_path unless we keep it for backwards compat.
 +
* The setup.cfg will have a new section to define the trove proper extensions (see PoC for example)
  
 
=== Database ===
 
=== Database ===
* Does this impact any existing tables?  If so, which ones?
+
No DB changes.
* Are the changes forward and backward compatible?
 
* Be sure to include the expected migration process
 
  
 
=== Public API ===
 
=== Public API ===
* Does this change any API that an end-user has access to?
+
No public API changes.
* Are there any exceptions in terms of consistency with other APIs?
 
  
 
==== CLI interface====
 
==== CLI interface====
* How the command will look like?
+
No CLI changes.
* Does it extends the already existed command interfaces ?
 
  
 
==== ReST Part ====
 
==== ReST Part ====
* Which HTTP methods added ?
+
No REST API changes.
* Which routes were added/modified/extended?
 
* How does the Request body look like?
 
* How does the Response object look like?
 
  
 
=== Internal API ===
 
=== Internal API ===
* Does this change any internal messages between API and Task Manager or Task Manager to Guest
+
Nothing new other than whats described in the above sections w/r/t to internal impl of extension discovery and loading with stevedore.
 +
 
 +
 
 
==== RPC API description====
 
==== RPC API description====
* Method name.
+
No change.
* Method parameters.
 
* Message type (cast/call).
 
  
 
=== Guest Agent ===
 
=== Guest Agent ===
* Does this change behavior on the Guest Agent? If so, is it backwards compatible with API and Task Manager?
+
No change.

Latest revision as of 17:52, 25 July 2014

Description

The current trove impl for loading API extensions is based on a single path which is specified in the trove conf file and then searched at trove-api start-up time to find and load extensions. While this approach is sufficient for loading trove proper extensions it becomes more painful for consumers who wish to package and / or consume trove extensions not shipped with trove proper. With the current approach, a consumer must copy his / her extension class (a concrete impl of trove.common.extensions.ExtensionDescriptor [1]) into the path specified by the api_extensions_path conf property[2] in trove.conf in order for it to be picked up the the trove-api extension loading [3]. The act of copying py files into another modules install path is a bad practice and this is not a very robust / flexible approach.

What's proposed here to move to a more dynamic means to discover and load extensions using the stevedore package: http://stevedore.readthedocs.org/en/latest/index.html


Justification/Benefits

A more dynamic / robust means to discover and load trove extensions will provide greater ease of use and flexibility for trove consumers as well as for trove proper. Moreover using stevedore to discover and load extensions will move trove proper more in line with how other core projects do things (e.g. nova and co). In particular:

  • Consumers can package and distribute trove extensions in separate python packages and the trove framework will automatically discover and load them without any changes to trove code or conf.
  • Trove proper development will not be constrained to only defining extensions in the fixed path / package of trove.extensions.routes
  • Use of stevedore will be more in line with with other core projects do and put trove in a closer position to updating our extension handling (https://blueprints.launchpad.net/trove/+spec/extensions-update).

Impacts

A simple PoC has been coded up and can be viewed here: https://github.com/bodenr/trove/commit/fa06e1d96e6a49a2a54057e8feb8e624edeaf728

As shown in the PoC minimal code changes are necessary to trove proper hence minimal risk from a code churn POV. The major impacts are as follows:

  • The api_extensions_path conf property is no longer needed.
  • Extensions are defined in the setup.cfg of python projects under a well defined extension key.
  • As per the PoC the base Extension class (ExtensionDescriptor) is now a "proper" abstract class (using py package abc) which provides stronger runtime checking... However this change is not required per say.
  • The extension loading code no longer needs to check a path and load init python classes -- this is done via stevedore.

However a potential impact which is not shown in the PoC is -- how do we handle upgrade for extensions. The concern here is that consumers may have extensions using the old (current) means whereupon they copied them into the extension path, but with the new approach these extensions need to be put into a well defined section of a setup.cfg. There are a few options for this consideration:

  • Doc the change assuming not many consumers have custom extensions.
  • Retain a legacy path in the code whereupon in addition to using the stevedore method to discover plugins we also still check the old-school path and only load plugins which were not loaded with stevedore... This provides backwards compatibility.
  • Have a simple CLI or code in the "upgrade" path which writes out a custom setup.cfg to capture any non-standard extensions and effectively migrate them to the new approach.
  • etc..

Configuration

  • trove.conf will no longer need the api_extensions_path unless we keep it for backwards compat.
  • The setup.cfg will have a new section to define the trove proper extensions (see PoC for example)

Database

No DB changes.

Public API

No public API changes.

CLI interface

No CLI changes.

ReST Part

No REST API changes.

Internal API

Nothing new other than whats described in the above sections w/r/t to internal impl of extension discovery and loading with stevedore.


RPC API description

No change.

Guest Agent

No change.