Jump to: navigation, search

Difference between revisions of "Barbican/Discussion-Plugin-Design"

m (Added schedule/batch processing section.)
(Added plugin source code organization section.)
Line 33: Line 33:
 
== Scheduled and Batch Processing ==
 
== Scheduled and Batch Processing ==
  
The discussion so far has focused on synchronous and asynchronous invocation of plugins for a given workflow or order instance, but some processes might require batch processing across multiple instances. For example, SSL certification processing may involve requesting status from a CA on a scheduled basis. This status might be a batch of multiple SSL certificate orders at once, so Barbican would need to iterate through these order statuses and individually invoke plugin tasks for them. The plugin might provide a batch method that Barbican Core could invoke on a scheduled basis, with a callback function passed in that the plugin calls for each order instance seen in the batch.
+
The discussion so far has focused on synchronous and asynchronous invocation of plugins for a given workflow or order instance, but some processes might require batch processing across multiple instances. For example, SSL certification processing may involve requesting status from a CA on a scheduled basis. This status might be a batch of multiple SSL certificate orders at once, so Barbican would need to iterate through these order statuses and individually invoke plugin tasks for them. The plugin might provide a batch method that Barbican Core could invoke on a scheduled basis, with a callback function passed in that the plugin calls for each order instance seen in the batch. Barbican Core could then enqueue plugin RPC tasks for each one for the worker nodes to process.
 +
 
 +
== Plugin Source Code Organization ==
 +
 
 +
The term 'Barbican Core' refers to code found in the [https://github.com/stackforge/barbican stackforge/barbican repository] and includes logic supported the left hand side of the figures above, and an abstract base class defining the interactions to the plugins in the middle of the diagrams. Core should also always include simple example and standalone plugin implementations that are enabled out-of-the-box on local installations. They shouldn't require network access to function and demonstrate, should be well unit-tested and should provide a good example to developers of new plugins.
 +
 
 +
Beyond these simple default plugins however, it is not as obvious how to manage specific plugin implementations' source code. On one hand it is convenient to bundle with Core source code for specific plugin implementations that are likely to be used for production Barbican installations. For example, Barbican Core does currently include PKCS11 and Dogtag based crypto plugins. On the other hand, these plugins usually have dependencies on libraries that are not part of the OpenStack global requirements, and therefore have to accommodate out-of-the-box deployments that don't have those dependencies installed. Hence thorough unit testing is more difficult (via patching) and code logic is a bit more complicated to deal with missing imports.
 +
 
 +
Another option is to create separate git repositories for the plugin implementation source files, with a dependency on the Barbican Core source base such as to extend abstract plugin contracts. This approach would simplify the Barbican Core code base, but would require integrating multiple repositories for testing purposes. It would also require mechanisms to extend Barbican to include these external dependencies at package time. This is explored in the next section.

Revision as of 23:47, 4 May 2014

Contents

This page explores design concepts for Barbican plugin interfaces. Barbican currently uses plugins to interface with cryptographic resources such as hardware security modules (HSMs). This page also discusses how the plugin approach could accommodate the planned addition of SSL certificate generation and management to the orders resource.

Overview

The following figure depicts a generic plugin dataflow within Barbican. Note the separation of 'core' Barbican functionality (available in the main Barbican repository and representing work done on behalf of plugins) from 'plugin' functionality to perform some type of work, which might include interaction with external services. The source code for these 'plugins' may or may not be available in the Barbican code base.

Generic Plugin Data Flow

Focusing on Barbican Core, a plugin must be selectable from more than one potential implementing plugin based on some criteria, such as first one to support a feature. Plugin selection is discussed in a later section.

Barbican must then provide inputs to the plugin to do its work. If the plugin is stateful across multiple calls to the plugin, then Barbican should store this state on the plugin's behalf, keying this data to an flow instance such as a specific order process. Note that Barbican may also pass an 'inversion of control' (IoC) component into the plugin, which would allow the plugin to interact with Barbican services (such as event generation) without knowledge of how Barbican implements these services.

When the plugin is invoked, the plugin performs its work, which may include interacting with an external service. For synchronous work flows (such as Barbican API processing), these service calls should be made as fast as possible since the response back to the client will be blocked until they complete.

Once a plugin returns, Barbican Core can persist the results. State can also be persisted into the Barbican Core data store if required for follow on plugin calls (such as extended workflow processing of a given SSL certificate). Barbican Core could also support if a plugin needs to be called again on a scheduled basis.

Asynchronous Order Processing Plugins

The Overview section detailed Barbican plugin flows. This section adds more detail for asynchronous order process flows, especially for SSL certification generation involving interacting with a certification authority (CA). The following figure depicts asynchronous processing by the Barbican Core worker process, invoked via RPC calls from the oslo.messaging queue service.

Asynchronous Order Processing Plugin Data Flow

For SSL certification generation, more than one vendor plugin may be available such as for Dogtag or Symantec, hence the order's details should include which vendor to use for the selection criteria, or else Barbican should support specifying a default vendor/plugin to use.

Barbican then retrieves any state associated with a given order instance, probably via order meta data information stored along with the order record. The plugin could define this status as key/value pairs for example. Since the same plugin may be called multiple times for the same order instance, this persisted state might include a state machine state name that directs which business logic to use within the vendor plugin. If the order instance needs to 'link' to an external system's order reference (such as for Symantec) this could be stored in the meta data as well (as determined by the plugin).

Next Barbican Core must provide IoC components to allow plugins to perform system interaction (such as database updates and event notification) without them directly accessing these critical core components. As depicted in the figure, one IoC handler could present specific methods such as 'notify_ssl_cert_is_ready()' which are handled by Barbican Core as simple log messages for out-of-the-box deployments, or else as CADF messages sent via oslo incubator or Ceilometer for external systems to consume. Another IoC handler could 'wrap' data model operations such as 'generate_private_key()' which Barbican Core would implement as a generate/encrypt/store operation in the crypto package.

The order processing plugin can be invoked, perhaps routing flow based on the previous state information, such as for state machine processing for SSL certification processing. The plugin might respond with a status that the Barbican Core logic could use to determine what to do with the plugin next...for example, 'Done' might indicate order processing is completed, 'Continue' might mean persist plugin state with the order for a future plugin call (say via invoked scheduled batch update from the CA), and 'Retry' might mean call the plugin again at a future time to retry an operation.

Scheduled and Batch Processing

The discussion so far has focused on synchronous and asynchronous invocation of plugins for a given workflow or order instance, but some processes might require batch processing across multiple instances. For example, SSL certification processing may involve requesting status from a CA on a scheduled basis. This status might be a batch of multiple SSL certificate orders at once, so Barbican would need to iterate through these order statuses and individually invoke plugin tasks for them. The plugin might provide a batch method that Barbican Core could invoke on a scheduled basis, with a callback function passed in that the plugin calls for each order instance seen in the batch. Barbican Core could then enqueue plugin RPC tasks for each one for the worker nodes to process.

Plugin Source Code Organization

The term 'Barbican Core' refers to code found in the stackforge/barbican repository and includes logic supported the left hand side of the figures above, and an abstract base class defining the interactions to the plugins in the middle of the diagrams. Core should also always include simple example and standalone plugin implementations that are enabled out-of-the-box on local installations. They shouldn't require network access to function and demonstrate, should be well unit-tested and should provide a good example to developers of new plugins.

Beyond these simple default plugins however, it is not as obvious how to manage specific plugin implementations' source code. On one hand it is convenient to bundle with Core source code for specific plugin implementations that are likely to be used for production Barbican installations. For example, Barbican Core does currently include PKCS11 and Dogtag based crypto plugins. On the other hand, these plugins usually have dependencies on libraries that are not part of the OpenStack global requirements, and therefore have to accommodate out-of-the-box deployments that don't have those dependencies installed. Hence thorough unit testing is more difficult (via patching) and code logic is a bit more complicated to deal with missing imports.

Another option is to create separate git repositories for the plugin implementation source files, with a dependency on the Barbican Core source base such as to extend abstract plugin contracts. This approach would simplify the Barbican Core code base, but would require integrating multiple repositories for testing purposes. It would also require mechanisms to extend Barbican to include these external dependencies at package time. This is explored in the next section.