Barbican

2015-02-06T19:18:06Z

Woodster: /* Discussions / Etherpads */

Barbican is a ReST API designed for the secure storage, provisioning and management of secrets. It is aimed at being useful for all environments, including large ephemeral Clouds. All documentation and work can be found on either Launchpad or Github at the following locations:

* [https://github.com/openstack/barbican Source Code]
* [https://launchpad.net/barbican Launchpad]
* [https://github.com/cloudkeep/barbican/wiki Wiki]
* [https://github.com/cloudkeep/barbican/wiki/Gerrit-Review-Process Contributing]

== Core Team ==

* [PTL] Douglas Mendizábal (redrobot)
* Ade Lee (alee)
* Chad Lung (chadlung)
* Donald Stufft (dstufft)
* Jarret Raim (jraim)
* John Vrbanac (jvrbanac)
* John Wood (woodster)
* Juan Antonio Osorio Robles (jaosorior)
* Nathan Reller (rellerreller)
* Paul Kehrer (reaperhulk)
* Steve Heyman (hockeynut)
* Steven Gonzales (codekobe)

== Resources ==

{| border="1" cellpadding="2" cellspacing="0"
| Meetings
| [[Meetings/Barbican]]
|-
| IRC
| #openstack-barbican on Freenode
|-
| Milestones
| https://launchpad.net/barbican/+milestones
|-
| Integration
| [[Barbican/Integration]]
|-
| Planning
| http://git.openstack.org/cgit/openstack/barbican-specs
|-
| Mailing List
| [http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev OpenStack dev] list, prefix with [barbican]
|-
| Bug tracker
| https://bugs.launchpad.net/barbican
|-
| Blueprints
| [[Barbican/Blueprints]]
|}

== Discussions / Etherpads ==
* [https://etherpad.openstack.org/p/barbican-L-design-sessions Barbican "L" Design Sessions Etherpad]
* [[Barbican/Kilo]]
* [https://etherpad.openstack.org/p/barbican-juno-meetup Barbican Juno Midcycle Meetup]
* [https://etherpad.openstack.org/p/barbican-juno-roadmap Barbican Juno Roadmap]
* [https://etherpad.openstack.org/p/barbican-events Barbican Events (Atlanta Summit 2014)]
* [https://etherpad.openstack.org/p/barbican-plugins Barbican Plugins (Atlanta Summit 2014)]
* [https://etherpad.openstack.org/p/barbican-asym Barbican Asymmetric Support (Atlanta Summit 2014)]
* [https://etherpad.openstack.org/p/barbican-kite Kite (Atlanta Summit 2014)]
* [https://etherpad.openstack.org/p/secret-store Secret Store (Atlanta Summit 2014)]
* [[Barbican/Certmonger]]
* [[Barbican/Discussion-Plugin-Design]]

== Getting Started ==

* [https://github.com/cloudkeep/barbican/wiki/Barbican-Getting-Started-Guide Getting Started]
* [https://wiki.openstack.org/wiki/BarbicanDevStack Running Barbican via DevStack]
* [https://github.com/cloudkeep-ops/barbican-vagrant-zero/blob/master/README.md Setting up a local Barbican Cluster using Vagrant]
* [https://wiki.openstack.org/wiki/Barbican/Barbican-IDE-Debugging Barbican debugging with IDE]

== Automation Details ==

* [https://wiki.openstack.org/wiki/BarbicanAutomation Barbican Automation (Chef)]

== Technical Details ==

* [https://github.com/cloudkeep/barbican/wiki/Application-Programming-Interface API Documentation]
* [https://github.com/cloudkeep/barbican/wiki/Architecture Architecture]

== Presentations ==

* [http://www.openstack.org/summit/openstack-summit-hong-kong-2013/session-videos/presentation/barbican-1-0-open-source-key-management-for-openstack Havanna Summit in Hong Kong], [https://speakerdeck.com/jraim/barbican-1-dot-0 Presentation]
* [http://www.openstack.org/summit/portland-2013/session-videos/presentation/cloud-keep-openstack-key-management-as-a-service Grizzly Summit in Portand], [https://speakerdeck.com/jraim/secret-as-a-service-barbican Presentation]
* [https://speakerdeck.com/jraim/cloudifying-key-management Cloudifying Key Management] from BSides Austin 2013

[[category: KeyManagementService]]

Barbican

2015-02-06T19:17:52Z

Woodster: /* Discussions / Etherpads */

Barbican is a ReST API designed for the secure storage, provisioning and management of secrets. It is aimed at being useful for all environments, including large ephemeral Clouds. All documentation and work can be found on either Launchpad or Github at the following locations:

* [https://github.com/openstack/barbican Source Code]
* [https://launchpad.net/barbican Launchpad]
* [https://github.com/cloudkeep/barbican/wiki Wiki]
* [https://github.com/cloudkeep/barbican/wiki/Gerrit-Review-Process Contributing]

== Core Team ==

* [PTL] Douglas Mendizábal (redrobot)
* Ade Lee (alee)
* Chad Lung (chadlung)
* Donald Stufft (dstufft)
* Jarret Raim (jraim)
* John Vrbanac (jvrbanac)
* John Wood (woodster)
* Juan Antonio Osorio Robles (jaosorior)
* Nathan Reller (rellerreller)
* Paul Kehrer (reaperhulk)
* Steve Heyman (hockeynut)
* Steven Gonzales (codekobe)

== Resources ==

{| border="1" cellpadding="2" cellspacing="0"
| Meetings
| [[Meetings/Barbican]]
|-
| IRC
| #openstack-barbican on Freenode
|-
| Milestones
| https://launchpad.net/barbican/+milestones
|-
| Integration
| [[Barbican/Integration]]
|-
| Planning
| http://git.openstack.org/cgit/openstack/barbican-specs
|-
| Mailing List
| [http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev OpenStack dev] list, prefix with [barbican]
|-
| Bug tracker
| https://bugs.launchpad.net/barbican
|-
| Blueprints
| [[Barbican/Blueprints]]
|}

== Discussions / Etherpads ==
* [https://etherpad.openstack.org/p/barbican-L-design-sessions Barbican "L" Design Sessions Etherpad]
* [[Barbican/Kilo] Kilo Planning Wiki]
* [https://etherpad.openstack.org/p/barbican-juno-meetup Barbican Juno Midcycle Meetup]
* [https://etherpad.openstack.org/p/barbican-juno-roadmap Barbican Juno Roadmap]
* [https://etherpad.openstack.org/p/barbican-events Barbican Events (Atlanta Summit 2014)]
* [https://etherpad.openstack.org/p/barbican-plugins Barbican Plugins (Atlanta Summit 2014)]
* [https://etherpad.openstack.org/p/barbican-asym Barbican Asymmetric Support (Atlanta Summit 2014)]
* [https://etherpad.openstack.org/p/barbican-kite Kite (Atlanta Summit 2014)]
* [https://etherpad.openstack.org/p/secret-store Secret Store (Atlanta Summit 2014)]
* [[Barbican/Certmonger]]
* [[Barbican/Discussion-Plugin-Design]]

== Getting Started ==

* [https://github.com/cloudkeep/barbican/wiki/Barbican-Getting-Started-Guide Getting Started]
* [https://wiki.openstack.org/wiki/BarbicanDevStack Running Barbican via DevStack]
* [https://github.com/cloudkeep-ops/barbican-vagrant-zero/blob/master/README.md Setting up a local Barbican Cluster using Vagrant]
* [https://wiki.openstack.org/wiki/Barbican/Barbican-IDE-Debugging Barbican debugging with IDE]

== Automation Details ==

* [https://wiki.openstack.org/wiki/BarbicanAutomation Barbican Automation (Chef)]

== Technical Details ==

* [https://github.com/cloudkeep/barbican/wiki/Application-Programming-Interface API Documentation]
* [https://github.com/cloudkeep/barbican/wiki/Architecture Architecture]

== Presentations ==

* [http://www.openstack.org/summit/openstack-summit-hong-kong-2013/session-videos/presentation/barbican-1-0-open-source-key-management-for-openstack Havanna Summit in Hong Kong], [https://speakerdeck.com/jraim/barbican-1-dot-0 Presentation]
* [http://www.openstack.org/summit/portland-2013/session-videos/presentation/cloud-keep-openstack-key-management-as-a-service Grizzly Summit in Portand], [https://speakerdeck.com/jraim/secret-as-a-service-barbican Presentation]
* [https://speakerdeck.com/jraim/cloudifying-key-management Cloudifying Key Management] from BSides Austin 2013

[[category: KeyManagementService]]

Barbican

2015-02-06T18:31:32Z

Woodster: /* Discussions / Etherpads */

Barbican is a ReST API designed for the secure storage, provisioning and management of secrets. It is aimed at being useful for all environments, including large ephemeral Clouds. All documentation and work can be found on either Launchpad or Github at the following locations:

* [https://github.com/openstack/barbican Source Code]
* [https://launchpad.net/barbican Launchpad]
* [https://github.com/cloudkeep/barbican/wiki Wiki]
* [https://github.com/cloudkeep/barbican/wiki/Gerrit-Review-Process Contributing]

== Core Team ==

* [PTL] Douglas Mendizábal (redrobot)
* Ade Lee (alee)
* Chad Lung (chadlung)
* Donald Stufft (dstufft)
* Jarret Raim (jraim)
* John Vrbanac (jvrbanac)
* John Wood (woodster)
* Juan Antonio Osorio Robles (jaosorior)
* Nathan Reller (rellerreller)
* Paul Kehrer (reaperhulk)
* Steve Heyman (hockeynut)
* Steven Gonzales (codekobe)

== Resources ==

{| border="1" cellpadding="2" cellspacing="0"
| Meetings
| [[Meetings/Barbican]]
|-
| IRC
| #openstack-barbican on Freenode
|-
| Milestones
| https://launchpad.net/barbican/+milestones
|-
| Integration
| [[Barbican/Integration]]
|-
| Planning
| http://git.openstack.org/cgit/openstack/barbican-specs
|-
| Mailing List
| [http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev OpenStack dev] list, prefix with [barbican]
|-
| Bug tracker
| https://bugs.launchpad.net/barbican
|-
| Blueprints
| [[Barbican/Blueprints]]
|}

== Discussions / Etherpads ==
* [https://etherpad.openstack.org/p/barbican-L-design-sessions Barbican "L" Design Sessions Etherpad]
* [https://wiki.openstack.org/wiki/Sprints/BarbicanKiloSprint Barbican Kilo Midcycle Meetup]
* [[Barbican/Kilo]]
* [https://etherpad.openstack.org/p/barbican-juno-meetup Barbican Juno Midcycle Meetup]
* [https://etherpad.openstack.org/p/barbican-juno-roadmap Barbican Juno Roadmap]
* [https://etherpad.openstack.org/p/barbican-events Barbican Events (Atlanta Summit 2014)]
* [https://etherpad.openstack.org/p/barbican-plugins Barbican Plugins (Atlanta Summit 2014)]
* [https://etherpad.openstack.org/p/barbican-asym Barbican Asymmetric Support (Atlanta Summit 2014)]
* [https://etherpad.openstack.org/p/barbican-kite Kite (Atlanta Summit 2014)]
* [https://etherpad.openstack.org/p/secret-store Secret Store (Atlanta Summit 2014)]
* [[Barbican/Certmonger]]
* [[Barbican/Discussion-Plugin-Design]]

== Getting Started ==

* [https://github.com/cloudkeep/barbican/wiki/Barbican-Getting-Started-Guide Getting Started]
* [https://wiki.openstack.org/wiki/BarbicanDevStack Running Barbican via DevStack]
* [https://github.com/cloudkeep-ops/barbican-vagrant-zero/blob/master/README.md Setting up a local Barbican Cluster using Vagrant]
* [https://wiki.openstack.org/wiki/Barbican/Barbican-IDE-Debugging Barbican debugging with IDE]

== Automation Details ==

* [https://wiki.openstack.org/wiki/BarbicanAutomation Barbican Automation (Chef)]

== Technical Details ==

* [https://github.com/cloudkeep/barbican/wiki/Application-Programming-Interface API Documentation]
* [https://github.com/cloudkeep/barbican/wiki/Architecture Architecture]

== Presentations ==

* [http://www.openstack.org/summit/openstack-summit-hong-kong-2013/session-videos/presentation/barbican-1-0-open-source-key-management-for-openstack Havanna Summit in Hong Kong], [https://speakerdeck.com/jraim/barbican-1-dot-0 Presentation]
* [http://www.openstack.org/summit/portland-2013/session-videos/presentation/cloud-keep-openstack-key-management-as-a-service Grizzly Summit in Portand], [https://speakerdeck.com/jraim/secret-as-a-service-barbican Presentation]
* [https://speakerdeck.com/jraim/cloudifying-key-management Cloudifying Key Management] from BSides Austin 2013

[[category: KeyManagementService]]

Barbican

2015-02-06T16:37:08Z

Woodster: /* Discussions / Etherpads */

Barbican is a ReST API designed for the secure storage, provisioning and management of secrets. It is aimed at being useful for all environments, including large ephemeral Clouds. All documentation and work can be found on either Launchpad or Github at the following locations:

* [https://github.com/openstack/barbican Source Code]
* [https://launchpad.net/barbican Launchpad]
* [https://github.com/cloudkeep/barbican/wiki Wiki]
* [https://github.com/cloudkeep/barbican/wiki/Gerrit-Review-Process Contributing]

== Core Team ==

* [PTL] Douglas Mendizábal (redrobot)
* Ade Lee (alee)
* Chad Lung (chadlung)
* Donald Stufft (dstufft)
* Jarret Raim (jraim)
* John Vrbanac (jvrbanac)
* John Wood (woodster)
* Juan Antonio Osorio Robles (jaosorior)
* Nathan Reller (rellerreller)
* Paul Kehrer (reaperhulk)
* Steve Heyman (hockeynut)
* Steven Gonzales (codekobe)

== Resources ==

{| border="1" cellpadding="2" cellspacing="0"
| Meetings
| [[Meetings/Barbican]]
|-
| IRC
| #openstack-barbican on Freenode
|-
| Milestones
| https://launchpad.net/barbican/+milestones
|-
| Integration
| [[Barbican/Integration]]
|-
| Planning
| http://git.openstack.org/cgit/openstack/barbican-specs
|-
| Mailing List
| [http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev OpenStack dev] list, prefix with [barbican]
|-
| Bug tracker
| https://bugs.launchpad.net/barbican
|-
| Blueprints
| [[Barbican/Blueprints]]
|}

== Discussions / Etherpads ==
* [https://wiki.openstack.org/wiki/Sprints/BarbicanKiloSprint Barbican Kilo Midcycle Meetup]
* [[Barbican/Kilo]]
* [https://etherpad.openstack.org/p/barbican-juno-meetup Barbican Juno Midcycle Meetup]
* [https://etherpad.openstack.org/p/barbican-juno-roadmap Barbican Juno Roadmap]
* [https://etherpad.openstack.org/p/barbican-events Barbican Events (Atlanta Summit 2014)]
* [https://etherpad.openstack.org/p/barbican-plugins Barbican Plugins (Atlanta Summit 2014)]
* [https://etherpad.openstack.org/p/barbican-asym Barbican Asymmetric Support (Atlanta Summit 2014)]
* [https://etherpad.openstack.org/p/barbican-kite Kite (Atlanta Summit 2014)]
* [https://etherpad.openstack.org/p/secret-store Secret Store (Atlanta Summit 2014)]
* [[Barbican/Certmonger]]
* [[Barbican/Discussion-Plugin-Design]]

== Getting Started ==

* [https://github.com/cloudkeep/barbican/wiki/Barbican-Getting-Started-Guide Getting Started]
* [https://wiki.openstack.org/wiki/BarbicanDevStack Running Barbican via DevStack]
* [https://github.com/cloudkeep-ops/barbican-vagrant-zero/blob/master/README.md Setting up a local Barbican Cluster using Vagrant]
* [https://wiki.openstack.org/wiki/Barbican/Barbican-IDE-Debugging Barbican debugging with IDE]

== Automation Details ==

* [https://wiki.openstack.org/wiki/BarbicanAutomation Barbican Automation (Chef)]

== Technical Details ==

* [https://github.com/cloudkeep/barbican/wiki/Application-Programming-Interface API Documentation]
* [https://github.com/cloudkeep/barbican/wiki/Architecture Architecture]

== Presentations ==

* [http://www.openstack.org/summit/openstack-summit-hong-kong-2013/session-videos/presentation/barbican-1-0-open-source-key-management-for-openstack Havanna Summit in Hong Kong], [https://speakerdeck.com/jraim/barbican-1-dot-0 Presentation]
* [http://www.openstack.org/summit/portland-2013/session-videos/presentation/cloud-keep-openstack-key-management-as-a-service Grizzly Summit in Portand], [https://speakerdeck.com/jraim/secret-as-a-service-barbican Presentation]
* [https://speakerdeck.com/jraim/cloudifying-key-management Cloudifying Key Management] from BSides Austin 2013

[[category: KeyManagementService]]

BarbicanDevStack

2014-06-05T19:48:45Z

Woodster: Added note about running DevStack from a dedicated VM.

== Running Barbican via DevStack ==

Steps to run Barbican via DevStack (assuming Ubuntu 12.04 LTS or Ubuntu 14.04 LTS):

Note: Due to the amount of dependencies and configuration performed by the DevStack process, we suggest running DevStack from a dedicated/temporary virtual machine rather than your local workstation.

<pre><nowiki>
$ sudo apt-get update
$ sudo apt-get install libffi-dev libssl-dev git -y
$ git clone https://github.com/openstack-dev/devstack.git
$ git clone https://github.com/openstack/barbican.git
$ mv barbican/contrib/devstack/lib/barbican devstack/lib/
$ mv barbican/contrib/devstack/local.conf devstack/
$ mv barbican/contrib/devstack/extras.d/70-barbican.sh devstack/extras.d/
$ sudo ./devstack/tools/create-stack-user.sh
$ sudo mv devstack/ /opt/stack/
$ rm -rf barbican/
$ chown -R stack:stack /opt/stack/devstack/
$ su - stack
$ cd /opt/stack/devstack/
</nowiki></pre>

This next step will take a while to run but the end result is you should have Barbican running under DevStack:

<pre><nowiki>
$ ./stack.sh
</nowiki></pre>

To shutdown Barbican (and it's dependent services) simply run this:

<pre><nowiki>
$ ./unstack.sh
</nowiki></pre>

'''Note:''' If you have issues (compute might be throwing some errors - assuming you've enabled compute because we don't by default in the local.conf) you can ease back the firewall. Keep in mind this opens your computer up completely. '''Only do this if you know what you are doing and if you are having issues'''. Adjust as needed.

<pre><nowiki>
# Flush all firewall rules for DevStack
# (Only do this if you really know what you're doing and have problems running DevStack)
$ iptables -F
$ iptables -P INPUT ACCEPT
$ iptables -P FORWARD ACCEPT
$ iptables -P OUTPUT ACCEPT
</nowiki></pre>

Barbican/Blueprints

2014-05-30T14:58:45Z

Woodster: Added link to the barbican gerrit setup wiki page.

== Blueprints ==
At the Juno design summit in Atlanta, Barbican decided to adopt a new process for creating and approving blueprints. This process is based on similar processes that many other teams have adopted for the Juno cycle. You can review the high level blueprinting process [https://wiki.openstack.org/wiki/Blueprints here]. We are reproducing the Nova process below so as to prevent any inadvertent changes to the Barbican process if Nova decides to update theirs, but the rest of the content on the page should be useful.

=== Creating a Blueprint ===
Barbican has a team of people, led by the PTL, that are responsible for reviewing blueprints. That is the [https://launchpad.net/~barbican-core/+members#active barbican-core] team.

To aid the reviewing of a blueprint by a team of people, the the design specifications related to each blueprint are reviewed in Gerrit, and stored in git, just like all the code:
http://git.openstack.org/cgit/stackforge/barbican-specs (TODO [JDR]:This still needs to be created).

Starting with the Juno release, the process for getting your blueprint in Barbican is:
* Register your blueprint in launchpad (as normal)
* Upload a design specification in the "specs/<release>" folder in nova-specs
** e.g. http://git.openstack.org/cgit/stackforge/barbican-specs/tree/specs/juno/name-of-your-blueprint-in-launchpad
** it should be based on the [http://git.openstack.org/cgit/stackforge/barbican-specs/tree/specs/template.rst template], see the instructions in the template for more details
** get it reviewed by submitting your patch using Gerrit, in the usual way: [[Gerrit_Workflow]]
*** For new contributors, [https://github.com/cloudkeep/barbican/wiki/Gerrit-Review-Process this page] may also be helpful for setting up Gerrit locally.
** at the end of each release, non-completed specs will be removed
** you need to re-submit for the following release, should the blueprint slip
* Once your design specification has been committed to barbican-specs:
** Update your blueprint's specification URL to point to the design specification in barbican-specs
** Propose your blueprint, as above, by selecting the milestone in which you plan to complete the blueprint
* barbican-core will approve blueprint once:
** proposer has picked a target milestone, URL points to barbican-specs correctly, and code has been started
** barbican-specs check there is code in gerrit, such that they think the proposed target is achievable
** should blueprint slip to the next milestone, blueprint will be re-evaluated
* Now everything continues as it would with any other OpenStack project

Current Launchpad blueprints for Barbican can be found [https://blueprints.launchpad.net/barbican here].

Barbican/Discussion-Plugin-Design

2014-05-05T21:14:52Z

Woodster: Added emphasis to last line.

== Contents ==

'''This wiki page is a work in progress, intended to get contributors thinking about how to manage Barbican plugins and workflows for the Juno effort and beyond.'''

This page explores design concepts for Barbican plugin interfaces. Barbican currently uses plugins to interface with cryptographic resources such as hardware security modules (HSMs). This page also discusses how the plugin approach could accommodate the planned addition of SSL certificate generation and management to the ''orders'' resource.

== Overview ==

The following figure depicts a generic plugin dataflow within Barbican. Note the separation of 'core' Barbican functionality (available in the main Barbican repository and representing work done on behalf of plugins) from 'plugin' functionality to perform some type of work, which might include interaction with external services. Plugins can be invoked via synchronous or asynchronous processes, such as for encryption/decryption/validation or for order processing, respectively. The source code for these 'plugins' may or may not be available in the Barbican code base.

[[File:general-plugin-flow.jpg|framed|center|Generic Plugin Data Flow]]

Focusing on Barbican Core, a plugin must be selectable from more than one potential implementing plugin based on some criteria, such as first one to support a feature. Plugin selection is discussed in a later section.

Barbican must then provide inputs to the plugin to do its work. If the plugin is stateful across multiple calls to the plugin, then Barbican should store this state on the plugin's behalf, keying this data to an flow instance such as a specific order process. Note that Barbican may also pass an 'inversion of control' (IoC) component into the plugin, which would allow the plugin to interact with Barbican services (such as event generation) without knowledge of how Barbican implements these services.

When the plugin is invoked, the plugin performs its work, which may include interacting with an external service. For synchronous work flows (such as Barbican API processing), these service calls should be made as fast as possible since the response back to the client will be blocked until they complete.

Once a plugin returns, Barbican Core can persist the results. State can also be persisted into the Barbican Core data store if required for follow on plugin calls (such as extended workflow processing of a given SSL certificate). Barbican Core could also support if a plugin needs to be called again on a scheduled basis.

== Asynchronous Order Processing Plugins ==

The Overview section detailed Barbican plugin flows. This section adds more detail for asynchronous order process flows, especially for [[Barbican/Blueprints/ssl-certificates|SSL certification generation involving interacting with a certification authority (CA)]]. The following figure depicts asynchronous processing by the Barbican Core worker process, invoked via RPC calls from the oslo.messaging queue service.

[[File:async-ssl-cert-plugin.jpg|framed|center|Asynchronous Order Processing Plugin Data Flow]]

For SSL certification generation, more than one vendor plugin may be available such as for Dogtag or Symantec, hence the order's details should include which vendor to use for the selection criteria, or else Barbican should support specifying a default vendor/plugin to use. The same vendor plugin could be use to validate inputs (esp. for the many fields needed for a CSR) as well as for asynchronous worker-side order processing.

Barbican then retrieves any state associated with a given order instance, probably via order meta data information stored along with the order record. The plugin could define this status as key/value pairs for example. Since the same plugin may be called multiple times for the same order instance, this persisted state might include a state machine state name that directs which business logic to use within the vendor plugin. If the order instance needs to 'link' to an external system's order reference (such as for Symantec) this could be stored in the meta data as well (as determined by the plugin).

Next Barbican Core must provide IoC components to allow plugins to perform system interaction (such as database updates and event notification) without them directly accessing these critical core components. As depicted in the figure, one IoC handler could present specific methods such as 'notify_ssl_cert_is_ready()' which are handled by Barbican Core as simple log messages for out-of-the-box deployments, or else as CADF messages sent via oslo incubator or Ceilometer for external systems to consume in deployment/company-specific ways. Another IoC handler could 'wrap' data model operations such as 'generate_private_key()' which Barbican Core would implement as a generate/encrypt/store operation in the crypto package.

The order processing plugin can be invoked, perhaps routing flow based on the previous state information, such as for state machine processing for SSL certification processing. The plugin might respond with a status that the Barbican Core logic could use to determine what to do with the plugin next...for example, 'Done' might indicate order processing is completed, 'Continue' might mean persist plugin state with the order for a future plugin call (say via invoked scheduled batch update from the CA), and 'Retry' might mean call the plugin again at a future time to retry an operation.

== Scheduled and Batch Processing ==

The discussion so far has focused on synchronous and asynchronous invocation of plugins for a given workflow or order instance, but some processes might require batch processing across multiple instances. For example, SSL certification processing may involve requesting status from a CA on a scheduled basis. This status might be a batch of multiple SSL certificate order statuses at once, so Barbican would need to iterate through these order statuses and individually invoke plugin tasks for them. The plugin might provide a batch method that Barbican Core could invoke on a scheduled basis, with a callback function passed in that the plugin calls for each order instance seen in the batch. Barbican Core would implement the callback by enqueueing a plugin RPC task for a worker nodes to process.

To implement the scheduled processes, Barbican could use the Nova approach, that uses oslo-incubator's ''periodic_tasks'' annotation on Service methods that should be scheduled. It uses Eventlet greenpools under the hood. Currently the worker servers extend olso's Service, so they are the logical place for the scheduled processes to reside as well. A concern here though is that for reliability, each of the multiple workers should be able to schedule tasks such as the SSL batch status task above. ''If these separate scheduled tasks are in turn enqueuing single-order update tasks, it would be possible for more than one worker to be processing the same order instance.''

== Plugin Source Code Organization ==

The source code for 'Barbican Core' is found in the [https://github.com/stackforge/barbican stackforge/barbican repository] and includes logic supported the left hand side of the figures above, and an abstract base class defining the interactions to the plugins in the middle of the diagrams. Core should also always include simple example and standalone plugin implementations that are enabled out-of-the-box on local installations. They shouldn't require network access to function and demonstrate, should be well unit-tested and should provide a good example to developers of new plugins.

Beyond these simple default plugins however, it is not as obvious how to manage specific plugin implementations' source code. On one hand it is convenient to bundle with Core source code for specific plugin implementations that are likely to be used for production Barbican installations. For example, Barbican Core does currently include PKCS11 and Dogtag based crypto plugins. On the other hand, these plugins usually have dependencies on libraries that are not part of the OpenStack global requirements, and therefore have to accommodate out-of-the-box deployments that don't have those dependencies installed. Hence thorough unit testing is more difficult (via patching) and code logic is a bit more complicated to deal with missing imports.

Another option is to create separate git repositories for the plugin implementation source files, with a dependency on the Barbican Core source base such as to extend abstract plugin contracts. This approach would simplify the Barbican Core code base, but would require integrating multiple repositories for testing purposes. It would also require mechanisms to extend Barbican to include these external dependencies at package time. This is explored in the next section.

== Discovering 3rd Party Plugins ==

With the bundled crypto plugin implementations that Barbican Core includes now (such as PKCS11 and Dogtag), activating them for usage just requires including their dependencies in the deployed Python package or deployment, and then enabling them via configurations in the /etc/barbican/barbican-api.conf file. Stevedore provides the ability to load these plugins and then to select/use them at run time.

For plugins developed outside of Barbican Core, Stevedore could still be used and in addition to installing non-OpenStack dependencies and adding configuration items to barbican-api.conf, would also require adding a setup.cfg file that defines the new plugin namespaces, aliases and classpaths. A new custom-deployment package could then be created and installed.

Another option is to use a plugin module discovery process similar to the Heat project's Resource discovery. Heat defines a folder location that is searched for new Resources, in the form of Python source files that extend a base Resource. A similar approach could be used to discover plugin implementations.

Barbican/Discussion-Plugin-Design

2014-05-05T21:12:56Z

Woodster: Added last sentence about how the Heat approach could be used in barbican.

== Contents ==

'''This wiki page is a work in progress, intended to get contributors thinking about how to manage Barbican plugins and workflows for the Juno effort and beyond.'''

This page explores design concepts for Barbican plugin interfaces. Barbican currently uses plugins to interface with cryptographic resources such as hardware security modules (HSMs). This page also discusses how the plugin approach could accommodate the planned addition of SSL certificate generation and management to the ''orders'' resource.

== Overview ==

The following figure depicts a generic plugin dataflow within Barbican. Note the separation of 'core' Barbican functionality (available in the main Barbican repository and representing work done on behalf of plugins) from 'plugin' functionality to perform some type of work, which might include interaction with external services. Plugins can be invoked via synchronous or asynchronous processes, such as for encryption/decryption/validation or for order processing, respectively. The source code for these 'plugins' may or may not be available in the Barbican code base.

[[File:general-plugin-flow.jpg|framed|center|Generic Plugin Data Flow]]

Focusing on Barbican Core, a plugin must be selectable from more than one potential implementing plugin based on some criteria, such as first one to support a feature. Plugin selection is discussed in a later section.

Barbican must then provide inputs to the plugin to do its work. If the plugin is stateful across multiple calls to the plugin, then Barbican should store this state on the plugin's behalf, keying this data to an flow instance such as a specific order process. Note that Barbican may also pass an 'inversion of control' (IoC) component into the plugin, which would allow the plugin to interact with Barbican services (such as event generation) without knowledge of how Barbican implements these services.

When the plugin is invoked, the plugin performs its work, which may include interacting with an external service. For synchronous work flows (such as Barbican API processing), these service calls should be made as fast as possible since the response back to the client will be blocked until they complete.

Once a plugin returns, Barbican Core can persist the results. State can also be persisted into the Barbican Core data store if required for follow on plugin calls (such as extended workflow processing of a given SSL certificate). Barbican Core could also support if a plugin needs to be called again on a scheduled basis.

== Asynchronous Order Processing Plugins ==

The Overview section detailed Barbican plugin flows. This section adds more detail for asynchronous order process flows, especially for [[Barbican/Blueprints/ssl-certificates|SSL certification generation involving interacting with a certification authority (CA)]]. The following figure depicts asynchronous processing by the Barbican Core worker process, invoked via RPC calls from the oslo.messaging queue service.

[[File:async-ssl-cert-plugin.jpg|framed|center|Asynchronous Order Processing Plugin Data Flow]]

For SSL certification generation, more than one vendor plugin may be available such as for Dogtag or Symantec, hence the order's details should include which vendor to use for the selection criteria, or else Barbican should support specifying a default vendor/plugin to use. The same vendor plugin could be use to validate inputs (esp. for the many fields needed for a CSR) as well as for asynchronous worker-side order processing.

Barbican then retrieves any state associated with a given order instance, probably via order meta data information stored along with the order record. The plugin could define this status as key/value pairs for example. Since the same plugin may be called multiple times for the same order instance, this persisted state might include a state machine state name that directs which business logic to use within the vendor plugin. If the order instance needs to 'link' to an external system's order reference (such as for Symantec) this could be stored in the meta data as well (as determined by the plugin).

Next Barbican Core must provide IoC components to allow plugins to perform system interaction (such as database updates and event notification) without them directly accessing these critical core components. As depicted in the figure, one IoC handler could present specific methods such as 'notify_ssl_cert_is_ready()' which are handled by Barbican Core as simple log messages for out-of-the-box deployments, or else as CADF messages sent via oslo incubator or Ceilometer for external systems to consume in deployment/company-specific ways. Another IoC handler could 'wrap' data model operations such as 'generate_private_key()' which Barbican Core would implement as a generate/encrypt/store operation in the crypto package.

The order processing plugin can be invoked, perhaps routing flow based on the previous state information, such as for state machine processing for SSL certification processing. The plugin might respond with a status that the Barbican Core logic could use to determine what to do with the plugin next...for example, 'Done' might indicate order processing is completed, 'Continue' might mean persist plugin state with the order for a future plugin call (say via invoked scheduled batch update from the CA), and 'Retry' might mean call the plugin again at a future time to retry an operation.

== Scheduled and Batch Processing ==

The discussion so far has focused on synchronous and asynchronous invocation of plugins for a given workflow or order instance, but some processes might require batch processing across multiple instances. For example, SSL certification processing may involve requesting status from a CA on a scheduled basis. This status might be a batch of multiple SSL certificate order statuses at once, so Barbican would need to iterate through these order statuses and individually invoke plugin tasks for them. The plugin might provide a batch method that Barbican Core could invoke on a scheduled basis, with a callback function passed in that the plugin calls for each order instance seen in the batch. Barbican Core would implement the callback by enqueueing a plugin RPC task for a worker nodes to process.

To implement the scheduled processes, Barbican could use the Nova approach, that uses oslo-incubator's ''periodic_tasks'' annotation on Service methods that should be scheduled. It uses Eventlet greenpools under the hood. Currently the worker servers extend olso's Service, so they are the logical place for the scheduled processes to reside as well. A concern here though is that for reliability, each of the multiple workers should be able to schedule tasks such as the SSL batch status task above. If these separate scheduled tasks are in turn enqueuing single-order update tasks, it would be possible for more than one worker to be processing the same order instance.

== Plugin Source Code Organization ==

The source code for 'Barbican Core' is found in the [https://github.com/stackforge/barbican stackforge/barbican repository] and includes logic supported the left hand side of the figures above, and an abstract base class defining the interactions to the plugins in the middle of the diagrams. Core should also always include simple example and standalone plugin implementations that are enabled out-of-the-box on local installations. They shouldn't require network access to function and demonstrate, should be well unit-tested and should provide a good example to developers of new plugins.

Beyond these simple default plugins however, it is not as obvious how to manage specific plugin implementations' source code. On one hand it is convenient to bundle with Core source code for specific plugin implementations that are likely to be used for production Barbican installations. For example, Barbican Core does currently include PKCS11 and Dogtag based crypto plugins. On the other hand, these plugins usually have dependencies on libraries that are not part of the OpenStack global requirements, and therefore have to accommodate out-of-the-box deployments that don't have those dependencies installed. Hence thorough unit testing is more difficult (via patching) and code logic is a bit more complicated to deal with missing imports.

Another option is to create separate git repositories for the plugin implementation source files, with a dependency on the Barbican Core source base such as to extend abstract plugin contracts. This approach would simplify the Barbican Core code base, but would require integrating multiple repositories for testing purposes. It would also require mechanisms to extend Barbican to include these external dependencies at package time. This is explored in the next section.

== Discovering 3rd Party Plugins ==

With the bundled crypto plugin implementations that Barbican Core includes now (such as PKCS11 and Dogtag), activating them for usage just requires including their dependencies in the deployed Python package or deployment, and then enabling them via configurations in the /etc/barbican/barbican-api.conf file. Stevedore provides the ability to load these plugins and then to select/use them at run time.

For plugins developed outside of Barbican Core, Stevedore could still be used and in addition to installing non-OpenStack dependencies and adding configuration items to barbican-api.conf, would also require adding a setup.cfg file that defines the new plugin namespaces, aliases and classpaths. A new custom-deployment package could then be created and installed.

Another option is to use a plugin module discovery process similar to the Heat project's Resource discovery. Heat defines a folder location that is searched for new Resources, in the form of Python source files that extend a base Resource. A similar approach could be used to discover plugin implementations.

Barbican/Discussion-Plugin-Design

2014-05-05T21:11:54Z

Woodster: Add a blank line after figure

== Contents ==

'''This wiki page is a work in progress, intended to get contributors thinking about how to manage Barbican plugins and workflows for the Juno effort and beyond.'''

This page explores design concepts for Barbican plugin interfaces. Barbican currently uses plugins to interface with cryptographic resources such as hardware security modules (HSMs). This page also discusses how the plugin approach could accommodate the planned addition of SSL certificate generation and management to the ''orders'' resource.

== Overview ==

The following figure depicts a generic plugin dataflow within Barbican. Note the separation of 'core' Barbican functionality (available in the main Barbican repository and representing work done on behalf of plugins) from 'plugin' functionality to perform some type of work, which might include interaction with external services. Plugins can be invoked via synchronous or asynchronous processes, such as for encryption/decryption/validation or for order processing, respectively. The source code for these 'plugins' may or may not be available in the Barbican code base.

[[File:general-plugin-flow.jpg|framed|center|Generic Plugin Data Flow]]

Focusing on Barbican Core, a plugin must be selectable from more than one potential implementing plugin based on some criteria, such as first one to support a feature. Plugin selection is discussed in a later section.

Barbican must then provide inputs to the plugin to do its work. If the plugin is stateful across multiple calls to the plugin, then Barbican should store this state on the plugin's behalf, keying this data to an flow instance such as a specific order process. Note that Barbican may also pass an 'inversion of control' (IoC) component into the plugin, which would allow the plugin to interact with Barbican services (such as event generation) without knowledge of how Barbican implements these services.

When the plugin is invoked, the plugin performs its work, which may include interacting with an external service. For synchronous work flows (such as Barbican API processing), these service calls should be made as fast as possible since the response back to the client will be blocked until they complete.

Once a plugin returns, Barbican Core can persist the results. State can also be persisted into the Barbican Core data store if required for follow on plugin calls (such as extended workflow processing of a given SSL certificate). Barbican Core could also support if a plugin needs to be called again on a scheduled basis.

== Asynchronous Order Processing Plugins ==

The Overview section detailed Barbican plugin flows. This section adds more detail for asynchronous order process flows, especially for [[Barbican/Blueprints/ssl-certificates|SSL certification generation involving interacting with a certification authority (CA)]]. The following figure depicts asynchronous processing by the Barbican Core worker process, invoked via RPC calls from the oslo.messaging queue service.

[[File:async-ssl-cert-plugin.jpg|framed|center|Asynchronous Order Processing Plugin Data Flow]]

For SSL certification generation, more than one vendor plugin may be available such as for Dogtag or Symantec, hence the order's details should include which vendor to use for the selection criteria, or else Barbican should support specifying a default vendor/plugin to use. The same vendor plugin could be use to validate inputs (esp. for the many fields needed for a CSR) as well as for asynchronous worker-side order processing.

Barbican then retrieves any state associated with a given order instance, probably via order meta data information stored along with the order record. The plugin could define this status as key/value pairs for example. Since the same plugin may be called multiple times for the same order instance, this persisted state might include a state machine state name that directs which business logic to use within the vendor plugin. If the order instance needs to 'link' to an external system's order reference (such as for Symantec) this could be stored in the meta data as well (as determined by the plugin).

Next Barbican Core must provide IoC components to allow plugins to perform system interaction (such as database updates and event notification) without them directly accessing these critical core components. As depicted in the figure, one IoC handler could present specific methods such as 'notify_ssl_cert_is_ready()' which are handled by Barbican Core as simple log messages for out-of-the-box deployments, or else as CADF messages sent via oslo incubator or Ceilometer for external systems to consume in deployment/company-specific ways. Another IoC handler could 'wrap' data model operations such as 'generate_private_key()' which Barbican Core would implement as a generate/encrypt/store operation in the crypto package.

The order processing plugin can be invoked, perhaps routing flow based on the previous state information, such as for state machine processing for SSL certification processing. The plugin might respond with a status that the Barbican Core logic could use to determine what to do with the plugin next...for example, 'Done' might indicate order processing is completed, 'Continue' might mean persist plugin state with the order for a future plugin call (say via invoked scheduled batch update from the CA), and 'Retry' might mean call the plugin again at a future time to retry an operation.

== Scheduled and Batch Processing ==

The discussion so far has focused on synchronous and asynchronous invocation of plugins for a given workflow or order instance, but some processes might require batch processing across multiple instances. For example, SSL certification processing may involve requesting status from a CA on a scheduled basis. This status might be a batch of multiple SSL certificate order statuses at once, so Barbican would need to iterate through these order statuses and individually invoke plugin tasks for them. The plugin might provide a batch method that Barbican Core could invoke on a scheduled basis, with a callback function passed in that the plugin calls for each order instance seen in the batch. Barbican Core would implement the callback by enqueueing a plugin RPC task for a worker nodes to process.

To implement the scheduled processes, Barbican could use the Nova approach, that uses oslo-incubator's ''periodic_tasks'' annotation on Service methods that should be scheduled. It uses Eventlet greenpools under the hood. Currently the worker servers extend olso's Service, so they are the logical place for the scheduled processes to reside as well. A concern here though is that for reliability, each of the multiple workers should be able to schedule tasks such as the SSL batch status task above. If these separate scheduled tasks are in turn enqueuing single-order update tasks, it would be possible for more than one worker to be processing the same order instance.

== Plugin Source Code Organization ==

The source code for 'Barbican Core' is found in the [https://github.com/stackforge/barbican stackforge/barbican repository] and includes logic supported the left hand side of the figures above, and an abstract base class defining the interactions to the plugins in the middle of the diagrams. Core should also always include simple example and standalone plugin implementations that are enabled out-of-the-box on local installations. They shouldn't require network access to function and demonstrate, should be well unit-tested and should provide a good example to developers of new plugins.

Beyond these simple default plugins however, it is not as obvious how to manage specific plugin implementations' source code. On one hand it is convenient to bundle with Core source code for specific plugin implementations that are likely to be used for production Barbican installations. For example, Barbican Core does currently include PKCS11 and Dogtag based crypto plugins. On the other hand, these plugins usually have dependencies on libraries that are not part of the OpenStack global requirements, and therefore have to accommodate out-of-the-box deployments that don't have those dependencies installed. Hence thorough unit testing is more difficult (via patching) and code logic is a bit more complicated to deal with missing imports.

Another option is to create separate git repositories for the plugin implementation source files, with a dependency on the Barbican Core source base such as to extend abstract plugin contracts. This approach would simplify the Barbican Core code base, but would require integrating multiple repositories for testing purposes. It would also require mechanisms to extend Barbican to include these external dependencies at package time. This is explored in the next section.

== Discovering 3rd Party Plugins ==

With the bundled crypto plugin implementations that Barbican Core includes now (such as PKCS11 and Dogtag), activating them for usage just requires including their dependencies in the deployed Python package or deployment, and then enabling them via configurations in the /etc/barbican/barbican-api.conf file. Stevedore provides the ability to load these plugins and then to select/use them at run time.

For plugins developed outside of Barbican Core, Stevedore could still be used and in addition to installing non-OpenStack dependencies and adding configuration items to barbican-api.conf, would also require adding a setup.cfg file that defines the new plugin namespaces, aliases and classpaths. A new custom-deployment package could then be created and installed.

Another option is to use a plugin module discovery process similar to the Heat project's Resource discovery. Heat defines a folder location that is searched for new Resources, in the form of Python source files that extend a base Resource.

Barbican/Discussion-Plugin-Design

2014-05-05T21:11:36Z

Woodster: Add a blank line after figure

== Contents ==

'''This wiki page is a work in progress, intended to get contributors thinking about how to manage Barbican plugins and workflows for the Juno effort and beyond.'''

This page explores design concepts for Barbican plugin interfaces. Barbican currently uses plugins to interface with cryptographic resources such as hardware security modules (HSMs). This page also discusses how the plugin approach could accommodate the planned addition of SSL certificate generation and management to the ''orders'' resource.

== Overview ==

The following figure depicts a generic plugin dataflow within Barbican. Note the separation of 'core' Barbican functionality (available in the main Barbican repository and representing work done on behalf of plugins) from 'plugin' functionality to perform some type of work, which might include interaction with external services. Plugins can be invoked via synchronous or asynchronous processes, such as for encryption/decryption/validation or for order processing, respectively. The source code for these 'plugins' may or may not be available in the Barbican code base.

[[File:general-plugin-flow.jpg|framed|center|Generic Plugin Data Flow]]

Focusing on Barbican Core, a plugin must be selectable from more than one potential implementing plugin based on some criteria, such as first one to support a feature. Plugin selection is discussed in a later section.

Barbican must then provide inputs to the plugin to do its work. If the plugin is stateful across multiple calls to the plugin, then Barbican should store this state on the plugin's behalf, keying this data to an flow instance such as a specific order process. Note that Barbican may also pass an 'inversion of control' (IoC) component into the plugin, which would allow the plugin to interact with Barbican services (such as event generation) without knowledge of how Barbican implements these services.

When the plugin is invoked, the plugin performs its work, which may include interacting with an external service. For synchronous work flows (such as Barbican API processing), these service calls should be made as fast as possible since the response back to the client will be blocked until they complete.

Once a plugin returns, Barbican Core can persist the results. State can also be persisted into the Barbican Core data store if required for follow on plugin calls (such as extended workflow processing of a given SSL certificate). Barbican Core could also support if a plugin needs to be called again on a scheduled basis.

== Asynchronous Order Processing Plugins ==

The Overview section detailed Barbican plugin flows. This section adds more detail for asynchronous order process flows, especially for [[Barbican/Blueprints/ssl-certificates|SSL certification generation involving interacting with a certification authority (CA)]]. The following figure depicts asynchronous processing by the Barbican Core worker process, invoked via RPC calls from the oslo.messaging queue service.

[[File:async-ssl-cert-plugin.jpg|framed|center|Asynchronous Order Processing Plugin Data Flow]]

For SSL certification generation, more than one vendor plugin may be available such as for Dogtag or Symantec, hence the order's details should include which vendor to use for the selection criteria, or else Barbican should support specifying a default vendor/plugin to use. The same vendor plugin could be use to validate inputs (esp. for the many fields needed for a CSR) as well as for asynchronous worker-side order processing.

Barbican then retrieves any state associated with a given order instance, probably via order meta data information stored along with the order record. The plugin could define this status as key/value pairs for example. Since the same plugin may be called multiple times for the same order instance, this persisted state might include a state machine state name that directs which business logic to use within the vendor plugin. If the order instance needs to 'link' to an external system's order reference (such as for Symantec) this could be stored in the meta data as well (as determined by the plugin).

Next Barbican Core must provide IoC components to allow plugins to perform system interaction (such as database updates and event notification) without them directly accessing these critical core components. As depicted in the figure, one IoC handler could present specific methods such as 'notify_ssl_cert_is_ready()' which are handled by Barbican Core as simple log messages for out-of-the-box deployments, or else as CADF messages sent via oslo incubator or Ceilometer for external systems to consume in deployment/company-specific ways. Another IoC handler could 'wrap' data model operations such as 'generate_private_key()' which Barbican Core would implement as a generate/encrypt/store operation in the crypto package.

The order processing plugin can be invoked, perhaps routing flow based on the previous state information, such as for state machine processing for SSL certification processing. The plugin might respond with a status that the Barbican Core logic could use to determine what to do with the plugin next...for example, 'Done' might indicate order processing is completed, 'Continue' might mean persist plugin state with the order for a future plugin call (say via invoked scheduled batch update from the CA), and 'Retry' might mean call the plugin again at a future time to retry an operation.

== Scheduled and Batch Processing ==

The discussion so far has focused on synchronous and asynchronous invocation of plugins for a given workflow or order instance, but some processes might require batch processing across multiple instances. For example, SSL certification processing may involve requesting status from a CA on a scheduled basis. This status might be a batch of multiple SSL certificate order statuses at once, so Barbican would need to iterate through these order statuses and individually invoke plugin tasks for them. The plugin might provide a batch method that Barbican Core could invoke on a scheduled basis, with a callback function passed in that the plugin calls for each order instance seen in the batch. Barbican Core would implement the callback by enqueueing a plugin RPC task for a worker nodes to process.

To implement the scheduled processes, Barbican could use the Nova approach, that uses oslo-incubator's ''periodic_tasks'' annotation on Service methods that should be scheduled. It uses Eventlet greenpools under the hood. Currently the worker servers extend olso's Service, so they are the logical place for the scheduled processes to reside as well. A concern here though is that for reliability, each of the multiple workers should be able to schedule tasks such as the SSL batch status task above. If these separate scheduled tasks are in turn enqueuing single-order update tasks, it would be possible for more than one worker to be processing the same order instance.

== Plugin Source Code Organization ==

The source code for 'Barbican Core' is found in the [https://github.com/stackforge/barbican stackforge/barbican repository] and includes logic supported the left hand side of the figures above, and an abstract base class defining the interactions to the plugins in the middle of the diagrams. Core should also always include simple example and standalone plugin implementations that are enabled out-of-the-box on local installations. They shouldn't require network access to function and demonstrate, should be well unit-tested and should provide a good example to developers of new plugins.

Beyond these simple default plugins however, it is not as obvious how to manage specific plugin implementations' source code. On one hand it is convenient to bundle with Core source code for specific plugin implementations that are likely to be used for production Barbican installations. For example, Barbican Core does currently include PKCS11 and Dogtag based crypto plugins. On the other hand, these plugins usually have dependencies on libraries that are not part of the OpenStack global requirements, and therefore have to accommodate out-of-the-box deployments that don't have those dependencies installed. Hence thorough unit testing is more difficult (via patching) and code logic is a bit more complicated to deal with missing imports.

Another option is to create separate git repositories for the plugin implementation source files, with a dependency on the Barbican Core source base such as to extend abstract plugin contracts. This approach would simplify the Barbican Core code base, but would require integrating multiple repositories for testing purposes. It would also require mechanisms to extend Barbican to include these external dependencies at package time. This is explored in the next section.

== Discovering 3rd Party Plugins ==

With the bundled crypto plugin implementations that Barbican Core includes now (such as PKCS11 and Dogtag), activating them for usage just requires including their dependencies in the deployed Python package or deployment, and then enabling them via configurations in the /etc/barbican/barbican-api.conf file. Stevedore provides the ability to load these plugins and then to select/use them at run time.

For plugins developed outside of Barbican Core, Stevedore could still be used and in addition to installing non-OpenStack dependencies and adding configuration items to barbican-api.conf, would also require adding a setup.cfg file that defines the new plugin namespaces, aliases and classpaths. A new custom-deployment package could then be created and installed.

Another option is to use a plugin module discovery process similar to the Heat project's Resource discovery. Heat defines a folder location that is searched for new Resources, in the form of Python source files that extend a base Resource.

Barbican/Discussion-Plugin-Design

2014-05-05T20:56:28Z

Woodster: Minor wording changes.

== Contents ==

'''This wiki page is a work in progress, intended to get contributors thinking about how to manage Barbican plugins and workflows for the Juno effort and beyond.'''

This page explores design concepts for Barbican plugin interfaces. Barbican currently uses plugins to interface with cryptographic resources such as hardware security modules (HSMs). This page also discusses how the plugin approach could accommodate the planned addition of SSL certificate generation and management to the ''orders'' resource.

== Overview ==

The following figure depicts a generic plugin dataflow within Barbican. Note the separation of 'core' Barbican functionality (available in the main Barbican repository and representing work done on behalf of plugins) from 'plugin' functionality to perform some type of work, which might include interaction with external services. Plugins can be invoked via synchronous or asynchronous processes, such as for encryption/decryption/validation or for order processing, respectively. The source code for these 'plugins' may or may not be available in the Barbican code base.

[[File:general-plugin-flow.jpg|framed|center|Generic Plugin Data Flow]]

Focusing on Barbican Core, a plugin must be selectable from more than one potential implementing plugin based on some criteria, such as first one to support a feature. Plugin selection is discussed in a later section.

Barbican must then provide inputs to the plugin to do its work. If the plugin is stateful across multiple calls to the plugin, then Barbican should store this state on the plugin's behalf, keying this data to an flow instance such as a specific order process. Note that Barbican may also pass an 'inversion of control' (IoC) component into the plugin, which would allow the plugin to interact with Barbican services (such as event generation) without knowledge of how Barbican implements these services.

When the plugin is invoked, the plugin performs its work, which may include interacting with an external service. For synchronous work flows (such as Barbican API processing), these service calls should be made as fast as possible since the response back to the client will be blocked until they complete.

Once a plugin returns, Barbican Core can persist the results. State can also be persisted into the Barbican Core data store if required for follow on plugin calls (such as extended workflow processing of a given SSL certificate). Barbican Core could also support if a plugin needs to be called again on a scheduled basis.

== Asynchronous Order Processing Plugins ==

The Overview section detailed Barbican plugin flows. This section adds more detail for asynchronous order process flows, especially for [[Barbican/Blueprints/ssl-certificates|SSL certification generation involving interacting with a certification authority (CA)]]. The following figure depicts asynchronous processing by the Barbican Core worker process, invoked via RPC calls from the oslo.messaging queue service.

[[File:async-ssl-cert-plugin.jpg|framed|center|Asynchronous Order Processing Plugin Data Flow]]

For SSL certification generation, more than one vendor plugin may be available such as for Dogtag or Symantec, hence the order's details should include which vendor to use for the selection criteria, or else Barbican should support specifying a default vendor/plugin to use. The same vendor plugin could be use to validate inputs (esp. for the many fields needed for a CSR) as well as for asynchronous worker-side order processing.

Barbican then retrieves any state associated with a given order instance, probably via order meta data information stored along with the order record. The plugin could define this status as key/value pairs for example. Since the same plugin may be called multiple times for the same order instance, this persisted state might include a state machine state name that directs which business logic to use within the vendor plugin. If the order instance needs to 'link' to an external system's order reference (such as for Symantec) this could be stored in the meta data as well (as determined by the plugin).

Next Barbican Core must provide IoC components to allow plugins to perform system interaction (such as database updates and event notification) without them directly accessing these critical core components. As depicted in the figure, one IoC handler could present specific methods such as 'notify_ssl_cert_is_ready()' which are handled by Barbican Core as simple log messages for out-of-the-box deployments, or else as CADF messages sent via oslo incubator or Ceilometer for external systems to consume in deployment/company-specific ways. Another IoC handler could 'wrap' data model operations such as 'generate_private_key()' which Barbican Core would implement as a generate/encrypt/store operation in the crypto package.

The order processing plugin can be invoked, perhaps routing flow based on the previous state information, such as for state machine processing for SSL certification processing. The plugin might respond with a status that the Barbican Core logic could use to determine what to do with the plugin next...for example, 'Done' might indicate order processing is completed, 'Continue' might mean persist plugin state with the order for a future plugin call (say via invoked scheduled batch update from the CA), and 'Retry' might mean call the plugin again at a future time to retry an operation.

== Scheduled and Batch Processing ==

The discussion so far has focused on synchronous and asynchronous invocation of plugins for a given workflow or order instance, but some processes might require batch processing across multiple instances. For example, SSL certification processing may involve requesting status from a CA on a scheduled basis. This status might be a batch of multiple SSL certificate order statuses at once, so Barbican would need to iterate through these order statuses and individually invoke plugin tasks for them. The plugin might provide a batch method that Barbican Core could invoke on a scheduled basis, with a callback function passed in that the plugin calls for each order instance seen in the batch. Barbican Core would implement the callback by enqueueing a plugin RPC task for a worker nodes to process.

To implement the scheduled processes, Barbican could use the Nova approach, that uses oslo-incubator's ''periodic_tasks'' annotation on Service methods that should be scheduled. It uses Eventlet greenpools under the hood. Currently the worker servers extend olso's Service, so they are the logical place for the scheduled processes to reside as well. A concern here though is that for reliability, each of the multiple workers should be able to schedule tasks such as the SSL batch status task above. If these separate scheduled tasks are in turn enqueuing single-order update tasks, it would be possible for more than one worker to be processing the same order instance.

== Plugin Source Code Organization ==

The source code for 'Barbican Core' is found in the [https://github.com/stackforge/barbican stackforge/barbican repository] and includes logic supported the left hand side of the figures above, and an abstract base class defining the interactions to the plugins in the middle of the diagrams. Core should also always include simple example and standalone plugin implementations that are enabled out-of-the-box on local installations. They shouldn't require network access to function and demonstrate, should be well unit-tested and should provide a good example to developers of new plugins.

Beyond these simple default plugins however, it is not as obvious how to manage specific plugin implementations' source code. On one hand it is convenient to bundle with Core source code for specific plugin implementations that are likely to be used for production Barbican installations. For example, Barbican Core does currently include PKCS11 and Dogtag based crypto plugins. On the other hand, these plugins usually have dependencies on libraries that are not part of the OpenStack global requirements, and therefore have to accommodate out-of-the-box deployments that don't have those dependencies installed. Hence thorough unit testing is more difficult (via patching) and code logic is a bit more complicated to deal with missing imports.

Another option is to create separate git repositories for the plugin implementation source files, with a dependency on the Barbican Core source base such as to extend abstract plugin contracts. This approach would simplify the Barbican Core code base, but would require integrating multiple repositories for testing purposes. It would also require mechanisms to extend Barbican to include these external dependencies at package time. This is explored in the next section.

== Discovering 3rd Party Plugins ==

With the bundled crypto plugin implementations that Barbican Core includes now (such as PKCS11 and Dogtag), activating them for usage just requires including their dependencies in the deployed Python package or deployment, and then enabling them via configurations in the /etc/barbican/barbican-api.conf file. Stevedore provides the ability to load these plugins and then to select/use them at run time.

For plugins developed outside of Barbican Core, Stevedore could still be used and in addition to installing non-OpenStack dependencies and adding configuration items to barbican-api.conf, would also require adding a setup.cfg file that defines the new plugin namespaces, aliases and classpaths. A new custom-deployment package could then be created and installed.

Another option is to use a plugin module discovery process similar to the Heat project's Resource discovery. Heat defines a folder location that is searched for new Resources, in the form of Python source files that extend a base Resource.

Barbican/Discussion-Plugin-Design

2014-05-05T20:47:36Z

Woodster: Added implementation details per Nova.

== Contents ==

'''This wiki page is a work in progress, intended to get contributors thinking about how to manage Barbican plugins and workflows for the Juno effort and beyond.'''

This page explores design concepts for Barbican plugin interfaces. Barbican currently uses plugins to interface with cryptographic resources such as hardware security modules (HSMs). This page also discusses how the plugin approach could accommodate the planned addition of SSL certificate generation and management to the ''orders'' resource.

== Overview ==

The following figure depicts a generic plugin dataflow within Barbican. Note the separation of 'core' Barbican functionality (available in the main Barbican repository and representing work done on behalf of plugins) from 'plugin' functionality to perform some type of work, which might include interaction with external services. Plugins can be invoked via synchronous or asynchronous processes, such as for encryption/decryption/validation or for order processing, respectively. The source code for these 'plugins' may or may not be available in the Barbican code base.

[[File:general-plugin-flow.jpg|framed|center|Generic Plugin Data Flow]]

Focusing on Barbican Core, a plugin must be selectable from more than one potential implementing plugin based on some criteria, such as first one to support a feature. Plugin selection is discussed in a later section.

Barbican must then provide inputs to the plugin to do its work. If the plugin is stateful across multiple calls to the plugin, then Barbican should store this state on the plugin's behalf, keying this data to an flow instance such as a specific order process. Note that Barbican may also pass an 'inversion of control' (IoC) component into the plugin, which would allow the plugin to interact with Barbican services (such as event generation) without knowledge of how Barbican implements these services.

When the plugin is invoked, the plugin performs its work, which may include interacting with an external service. For synchronous work flows (such as Barbican API processing), these service calls should be made as fast as possible since the response back to the client will be blocked until they complete.

Once a plugin returns, Barbican Core can persist the results. State can also be persisted into the Barbican Core data store if required for follow on plugin calls (such as extended workflow processing of a given SSL certificate). Barbican Core could also support if a plugin needs to be called again on a scheduled basis.

== Asynchronous Order Processing Plugins ==

The Overview section detailed Barbican plugin flows. This section adds more detail for asynchronous order process flows, especially for [[Barbican/Blueprints/ssl-certificates|SSL certification generation involving interacting with a certification authority (CA)]]. The following figure depicts asynchronous processing by the Barbican Core worker process, invoked via RPC calls from the oslo.messaging queue service.

[[File:async-ssl-cert-plugin.jpg|framed|center|Asynchronous Order Processing Plugin Data Flow]]

For SSL certification generation, more than one vendor plugin may be available such as for Dogtag or Symantec, hence the order's details should include which vendor to use for the selection criteria, or else Barbican should support specifying a default vendor/plugin to use. The same vendor plugin could be use to validate inputs (esp. for the many fields needed for a CSR) as well as for asynchronous worker-side order processing.

Barbican then retrieves any state associated with a given order instance, probably via order meta data information stored along with the order record. The plugin could define this status as key/value pairs for example. Since the same plugin may be called multiple times for the same order instance, this persisted state might include a state machine state name that directs which business logic to use within the vendor plugin. If the order instance needs to 'link' to an external system's order reference (such as for Symantec) this could be stored in the meta data as well (as determined by the plugin).

Next Barbican Core must provide IoC components to allow plugins to perform system interaction (such as database updates and event notification) without them directly accessing these critical core components. As depicted in the figure, one IoC handler could present specific methods such as 'notify_ssl_cert_is_ready()' which are handled by Barbican Core as simple log messages for out-of-the-box deployments, or else as CADF messages sent via oslo incubator or Ceilometer for external systems to consume in deployment/company-specific ways. Another IoC handler could 'wrap' data model operations such as 'generate_private_key()' which Barbican Core would implement as a generate/encrypt/store operation in the crypto package.

The order processing plugin can be invoked, perhaps routing flow based on the previous state information, such as for state machine processing for SSL certification processing. The plugin might respond with a status that the Barbican Core logic could use to determine what to do with the plugin next...for example, 'Done' might indicate order processing is completed, 'Continue' might mean persist plugin state with the order for a future plugin call (say via invoked scheduled batch update from the CA), and 'Retry' might mean call the plugin again at a future time to retry an operation.

== Scheduled and Batch Processing ==

The discussion so far has focused on synchronous and asynchronous invocation of plugins for a given workflow or order instance, but some processes might require batch processing across multiple instances. For example, SSL certification processing may involve requesting status from a CA on a scheduled basis. This status might be a batch of multiple SSL certificate order statuses at once, so Barbican would need to iterate through these order statuses and individually invoke plugin tasks for them. The plugin might provide a batch method that Barbican Core could invoke on a scheduled basis, with a callback function passed in that the plugin calls for each order instance seen in the batch. Barbican Core would implement the callback by enqueueing a plugin RPC task for a worker nodes to process.

To implement the scheduled processes, Barbican could use the Nova approach, that uses oslo-incubator's ''periodic_tasks'' annotation on Service methods that should be scheduled. It uses Eventlet greenpools under the hood. Currently the worker servers extend olso's Service, so they are the logical place for the scheduled processes to reside as well. A concern here though is that for reliability, each of the multiple workers should be able to schedule tasks such as the SSL batch status task above. If these separate scheduled tasks are in turn enqueuing single-order update tasks, it would be possible for more than one worker to be processing the same order instance.

== Plugin Source Code Organization ==

The term 'Barbican Core' refers to code found in the [https://github.com/stackforge/barbican stackforge/barbican repository] and includes logic supported the left hand side of the figures above, and an abstract base class defining the interactions to the plugins in the middle of the diagrams. Core should also always include simple example and standalone plugin implementations that are enabled out-of-the-box on local installations. They shouldn't require network access to function and demonstrate, should be well unit-tested and should provide a good example to developers of new plugins.

Beyond these simple default plugins however, it is not as obvious how to manage specific plugin implementations' source code. On one hand it is convenient to bundle with Core source code for specific plugin implementations that are likely to be used for production Barbican installations. For example, Barbican Core does currently include PKCS11 and Dogtag based crypto plugins. On the other hand, these plugins usually have dependencies on libraries that are not part of the OpenStack global requirements, and therefore have to accommodate out-of-the-box deployments that don't have those dependencies installed. Hence thorough unit testing is more difficult (via patching) and code logic is a bit more complicated to deal with missing imports.

Another option is to create separate git repositories for the plugin implementation source files, with a dependency on the Barbican Core source base such as to extend abstract plugin contracts. This approach would simplify the Barbican Core code base, but would require integrating multiple repositories for testing purposes. It would also require mechanisms to extend Barbican to include these external dependencies at package time. This is explored in the next section.

== Discovering 3rd Party Plugins ==

With the bundled crypto plugin implementations that Barbican Core includes now (such as PKCS11 and Dogtag), activating them for usage just requires including their dependencies in the deployed Python package or deployment, and then enabling them via configurations in the /etc/barbican/barbican-api.conf file. Stevedore provides the ability to load these plugins and then to select/use them at run time.

For plugins developed outside of Barbican Core, Stevedore could still be used and in addition to installing non-OpenStack dependencies and adding configuration items to barbican-api.conf, would also require adding a setup.cfg file that defines the new plugin namespaces, aliases and classpaths. A new custom-deployment package could then be created and installed.

Another option is to use a plugin module discovery process similar to the Heat project's Resource discovery. Heat defines a folder location that is searched for new Resources, in the form of Python source files that extend a base Resource.

Barbican/Discussion-Plugin-Design

2014-05-05T00:52:49Z

Woodster: Added note about needed to validate many CSR inputs.

== Contents ==

'''This wiki page is a work in progress, intended to get contributors thinking about how to manage Barbican plugins and workflows for the Juno effort and beyond.'''

This page explores design concepts for Barbican plugin interfaces. Barbican currently uses plugins to interface with cryptographic resources such as hardware security modules (HSMs). This page also discusses how the plugin approach could accommodate the planned addition of SSL certificate generation and management to the ''orders'' resource.

== Overview ==

The following figure depicts a generic plugin dataflow within Barbican. Note the separation of 'core' Barbican functionality (available in the main Barbican repository and representing work done on behalf of plugins) from 'plugin' functionality to perform some type of work, which might include interaction with external services. Plugins can be invoked via synchronous or asynchronous processes, such as for encryption/decryption/validation or for order processing, respectively. The source code for these 'plugins' may or may not be available in the Barbican code base.

[[File:general-plugin-flow.jpg|framed|center|Generic Plugin Data Flow]]

Focusing on Barbican Core, a plugin must be selectable from more than one potential implementing plugin based on some criteria, such as first one to support a feature. Plugin selection is discussed in a later section.

Barbican must then provide inputs to the plugin to do its work. If the plugin is stateful across multiple calls to the plugin, then Barbican should store this state on the plugin's behalf, keying this data to an flow instance such as a specific order process. Note that Barbican may also pass an 'inversion of control' (IoC) component into the plugin, which would allow the plugin to interact with Barbican services (such as event generation) without knowledge of how Barbican implements these services.

When the plugin is invoked, the plugin performs its work, which may include interacting with an external service. For synchronous work flows (such as Barbican API processing), these service calls should be made as fast as possible since the response back to the client will be blocked until they complete.

Once a plugin returns, Barbican Core can persist the results. State can also be persisted into the Barbican Core data store if required for follow on plugin calls (such as extended workflow processing of a given SSL certificate). Barbican Core could also support if a plugin needs to be called again on a scheduled basis.

== Asynchronous Order Processing Plugins ==

The Overview section detailed Barbican plugin flows. This section adds more detail for asynchronous order process flows, especially for [[Barbican/Blueprints/ssl-certificates|SSL certification generation involving interacting with a certification authority (CA)]]. The following figure depicts asynchronous processing by the Barbican Core worker process, invoked via RPC calls from the oslo.messaging queue service.

[[File:async-ssl-cert-plugin.jpg|framed|center|Asynchronous Order Processing Plugin Data Flow]]

For SSL certification generation, more than one vendor plugin may be available such as for Dogtag or Symantec, hence the order's details should include which vendor to use for the selection criteria, or else Barbican should support specifying a default vendor/plugin to use. The same vendor plugin could be use to validate inputs (esp. for the many fields needed for a CSR) as well as for asynchronous worker-side order processing.

Barbican then retrieves any state associated with a given order instance, probably via order meta data information stored along with the order record. The plugin could define this status as key/value pairs for example. Since the same plugin may be called multiple times for the same order instance, this persisted state might include a state machine state name that directs which business logic to use within the vendor plugin. If the order instance needs to 'link' to an external system's order reference (such as for Symantec) this could be stored in the meta data as well (as determined by the plugin).

Next Barbican Core must provide IoC components to allow plugins to perform system interaction (such as database updates and event notification) without them directly accessing these critical core components. As depicted in the figure, one IoC handler could present specific methods such as 'notify_ssl_cert_is_ready()' which are handled by Barbican Core as simple log messages for out-of-the-box deployments, or else as CADF messages sent via oslo incubator or Ceilometer for external systems to consume in deployment/company-specific ways. Another IoC handler could 'wrap' data model operations such as 'generate_private_key()' which Barbican Core would implement as a generate/encrypt/store operation in the crypto package.

The order processing plugin can be invoked, perhaps routing flow based on the previous state information, such as for state machine processing for SSL certification processing. The plugin might respond with a status that the Barbican Core logic could use to determine what to do with the plugin next...for example, 'Done' might indicate order processing is completed, 'Continue' might mean persist plugin state with the order for a future plugin call (say via invoked scheduled batch update from the CA), and 'Retry' might mean call the plugin again at a future time to retry an operation.

== Scheduled and Batch Processing ==

The discussion so far has focused on synchronous and asynchronous invocation of plugins for a given workflow or order instance, but some processes might require batch processing across multiple instances. For example, SSL certification processing may involve requesting status from a CA on a scheduled basis. This status might be a batch of multiple SSL certificate orders at once, so Barbican would need to iterate through these order statuses and individually invoke plugin tasks for them. The plugin might provide a batch method that Barbican Core could invoke on a scheduled basis, with a callback function passed in that the plugin calls for each order instance seen in the batch. Barbican Core would implement the callback by enqueueing a plugin RPC task for a worker nodes to process.

== Plugin Source Code Organization ==

The term 'Barbican Core' refers to code found in the [https://github.com/stackforge/barbican stackforge/barbican repository] and includes logic supported the left hand side of the figures above, and an abstract base class defining the interactions to the plugins in the middle of the diagrams. Core should also always include simple example and standalone plugin implementations that are enabled out-of-the-box on local installations. They shouldn't require network access to function and demonstrate, should be well unit-tested and should provide a good example to developers of new plugins.

Beyond these simple default plugins however, it is not as obvious how to manage specific plugin implementations' source code. On one hand it is convenient to bundle with Core source code for specific plugin implementations that are likely to be used for production Barbican installations. For example, Barbican Core does currently include PKCS11 and Dogtag based crypto plugins. On the other hand, these plugins usually have dependencies on libraries that are not part of the OpenStack global requirements, and therefore have to accommodate out-of-the-box deployments that don't have those dependencies installed. Hence thorough unit testing is more difficult (via patching) and code logic is a bit more complicated to deal with missing imports.

Another option is to create separate git repositories for the plugin implementation source files, with a dependency on the Barbican Core source base such as to extend abstract plugin contracts. This approach would simplify the Barbican Core code base, but would require integrating multiple repositories for testing purposes. It would also require mechanisms to extend Barbican to include these external dependencies at package time. This is explored in the next section.

== Discovering 3rd Party Plugins ==

With the bundled crypto plugin implementations that Barbican Core includes now (such as PKCS11 and Dogtag), activating them for usage just requires including their dependencies in the deployed Python package or deployment, and then enabling them via configurations in the /etc/barbican/barbican-api.conf file. Stevedore provides the ability to load these plugins and then to select/use them at run time.

For plugins developed outside of Barbican Core, Stevedore could still be used and in addition to installing non-OpenStack dependencies and adding configuration items to barbican-api.conf, would also require adding a setup.cfg file that defines the new plugin namespaces, aliases and classpaths. A new custom-deployment package could then be created and installed.

Another option is to use a plugin module discovery process similar to the Heat project's Resource discovery. Heat defines a folder location that is searched for new Resources, in the form of Python source files that extend a base Resource.

Barbican/Discussion-Plugin-Design

2014-05-05T00:41:51Z

Woodster: Added sych/async sentence in first paragraph.

== Contents ==

'''This wiki page is a work in progress, intended to get contributors thinking about how to manage Barbican plugins and workflows for the Juno effort and beyond.'''

This page explores design concepts for Barbican plugin interfaces. Barbican currently uses plugins to interface with cryptographic resources such as hardware security modules (HSMs). This page also discusses how the plugin approach could accommodate the planned addition of SSL certificate generation and management to the ''orders'' resource.

== Overview ==

The following figure depicts a generic plugin dataflow within Barbican. Note the separation of 'core' Barbican functionality (available in the main Barbican repository and representing work done on behalf of plugins) from 'plugin' functionality to perform some type of work, which might include interaction with external services. Plugins can be invoked via synchronous or asynchronous processes, such as for encryption/decryption/validation or for order processing, respectively. The source code for these 'plugins' may or may not be available in the Barbican code base.

[[File:general-plugin-flow.jpg|framed|center|Generic Plugin Data Flow]]

Focusing on Barbican Core, a plugin must be selectable from more than one potential implementing plugin based on some criteria, such as first one to support a feature. Plugin selection is discussed in a later section.

Barbican must then provide inputs to the plugin to do its work. If the plugin is stateful across multiple calls to the plugin, then Barbican should store this state on the plugin's behalf, keying this data to an flow instance such as a specific order process. Note that Barbican may also pass an 'inversion of control' (IoC) component into the plugin, which would allow the plugin to interact with Barbican services (such as event generation) without knowledge of how Barbican implements these services.

When the plugin is invoked, the plugin performs its work, which may include interacting with an external service. For synchronous work flows (such as Barbican API processing), these service calls should be made as fast as possible since the response back to the client will be blocked until they complete.

Once a plugin returns, Barbican Core can persist the results. State can also be persisted into the Barbican Core data store if required for follow on plugin calls (such as extended workflow processing of a given SSL certificate). Barbican Core could also support if a plugin needs to be called again on a scheduled basis.

== Asynchronous Order Processing Plugins ==

The Overview section detailed Barbican plugin flows. This section adds more detail for asynchronous order process flows, especially for [[Barbican/Blueprints/ssl-certificates|SSL certification generation involving interacting with a certification authority (CA)]]. The following figure depicts asynchronous processing by the Barbican Core worker process, invoked via RPC calls from the oslo.messaging queue service.

[[File:async-ssl-cert-plugin.jpg|framed|center|Asynchronous Order Processing Plugin Data Flow]]

For SSL certification generation, more than one vendor plugin may be available such as for Dogtag or Symantec, hence the order's details should include which vendor to use for the selection criteria, or else Barbican should support specifying a default vendor/plugin to use.

Barbican then retrieves any state associated with a given order instance, probably via order meta data information stored along with the order record. The plugin could define this status as key/value pairs for example. Since the same plugin may be called multiple times for the same order instance, this persisted state might include a state machine state name that directs which business logic to use within the vendor plugin. If the order instance needs to 'link' to an external system's order reference (such as for Symantec) this could be stored in the meta data as well (as determined by the plugin).

Next Barbican Core must provide IoC components to allow plugins to perform system interaction (such as database updates and event notification) without them directly accessing these critical core components. As depicted in the figure, one IoC handler could present specific methods such as 'notify_ssl_cert_is_ready()' which are handled by Barbican Core as simple log messages for out-of-the-box deployments, or else as CADF messages sent via oslo incubator or Ceilometer for external systems to consume in deployment/company-specific ways. Another IoC handler could 'wrap' data model operations such as 'generate_private_key()' which Barbican Core would implement as a generate/encrypt/store operation in the crypto package.

The order processing plugin can be invoked, perhaps routing flow based on the previous state information, such as for state machine processing for SSL certification processing. The plugin might respond with a status that the Barbican Core logic could use to determine what to do with the plugin next...for example, 'Done' might indicate order processing is completed, 'Continue' might mean persist plugin state with the order for a future plugin call (say via invoked scheduled batch update from the CA), and 'Retry' might mean call the plugin again at a future time to retry an operation.

== Scheduled and Batch Processing ==

The discussion so far has focused on synchronous and asynchronous invocation of plugins for a given workflow or order instance, but some processes might require batch processing across multiple instances. For example, SSL certification processing may involve requesting status from a CA on a scheduled basis. This status might be a batch of multiple SSL certificate orders at once, so Barbican would need to iterate through these order statuses and individually invoke plugin tasks for them. The plugin might provide a batch method that Barbican Core could invoke on a scheduled basis, with a callback function passed in that the plugin calls for each order instance seen in the batch. Barbican Core would implement the callback by enqueueing a plugin RPC task for a worker nodes to process.

== Plugin Source Code Organization ==

The term 'Barbican Core' refers to code found in the [https://github.com/stackforge/barbican stackforge/barbican repository] and includes logic supported the left hand side of the figures above, and an abstract base class defining the interactions to the plugins in the middle of the diagrams. Core should also always include simple example and standalone plugin implementations that are enabled out-of-the-box on local installations. They shouldn't require network access to function and demonstrate, should be well unit-tested and should provide a good example to developers of new plugins.

Beyond these simple default plugins however, it is not as obvious how to manage specific plugin implementations' source code. On one hand it is convenient to bundle with Core source code for specific plugin implementations that are likely to be used for production Barbican installations. For example, Barbican Core does currently include PKCS11 and Dogtag based crypto plugins. On the other hand, these plugins usually have dependencies on libraries that are not part of the OpenStack global requirements, and therefore have to accommodate out-of-the-box deployments that don't have those dependencies installed. Hence thorough unit testing is more difficult (via patching) and code logic is a bit more complicated to deal with missing imports.

Another option is to create separate git repositories for the plugin implementation source files, with a dependency on the Barbican Core source base such as to extend abstract plugin contracts. This approach would simplify the Barbican Core code base, but would require integrating multiple repositories for testing purposes. It would also require mechanisms to extend Barbican to include these external dependencies at package time. This is explored in the next section.

== Discovering 3rd Party Plugins ==

With the bundled crypto plugin implementations that Barbican Core includes now (such as PKCS11 and Dogtag), activating them for usage just requires including their dependencies in the deployed Python package or deployment, and then enabling them via configurations in the /etc/barbican/barbican-api.conf file. Stevedore provides the ability to load these plugins and then to select/use them at run time.

For plugins developed outside of Barbican Core, Stevedore could still be used and in addition to installing non-OpenStack dependencies and adding configuration items to barbican-api.conf, would also require adding a setup.cfg file that defines the new plugin namespaces, aliases and classpaths. A new custom-deployment package could then be created and installed.

Another option is to use a plugin module discovery process similar to the Heat project's Resource discovery. Heat defines a folder location that is searched for new Resources, in the form of Python source files that extend a base Resource.

Barbican/Discussion-Plugin-Design

2014-05-05T00:07:55Z

Woodster: Added discovering 3rd party plugins section.

== Contents ==

'''This wiki page is a work in progress, intended to get contributors thinking about how to manage Barbican plugins and workflows for the Juno effort and beyond.'''

This page explores design concepts for Barbican plugin interfaces. Barbican currently uses plugins to interface with cryptographic resources such as hardware security modules (HSMs). This page also discusses how the plugin approach could accommodate the planned addition of SSL certificate generation and management to the ''orders'' resource.

== Overview ==

The following figure depicts a generic plugin dataflow within Barbican. Note the separation of 'core' Barbican functionality (available in the main Barbican repository and representing work done on behalf of plugins) from 'plugin' functionality to perform some type of work, which might include interaction with external services. The source code for these 'plugins' may or may not be available in the Barbican code base.

[[File:general-plugin-flow.jpg|framed|center|Generic Plugin Data Flow]]

Focusing on Barbican Core, a plugin must be selectable from more than one potential implementing plugin based on some criteria, such as first one to support a feature. Plugin selection is discussed in a later section.

Barbican must then provide inputs to the plugin to do its work. If the plugin is stateful across multiple calls to the plugin, then Barbican should store this state on the plugin's behalf, keying this data to an flow instance such as a specific order process. Note that Barbican may also pass an 'inversion of control' (IoC) component into the plugin, which would allow the plugin to interact with Barbican services (such as event generation) without knowledge of how Barbican implements these services.

When the plugin is invoked, the plugin performs its work, which may include interacting with an external service. For synchronous work flows (such as Barbican API processing), these service calls should be made as fast as possible since the response back to the client will be blocked until they complete.

Once a plugin returns, Barbican Core can persist the results. State can also be persisted into the Barbican Core data store if required for follow on plugin calls (such as extended workflow processing of a given SSL certificate). Barbican Core could also support if a plugin needs to be called again on a scheduled basis.

== Asynchronous Order Processing Plugins ==

The Overview section detailed Barbican plugin flows. This section adds more detail for asynchronous order process flows, especially for [[Barbican/Blueprints/ssl-certificates|SSL certification generation involving interacting with a certification authority (CA)]]. The following figure depicts asynchronous processing by the Barbican Core worker process, invoked via RPC calls from the oslo.messaging queue service.

[[File:async-ssl-cert-plugin.jpg|framed|center|Asynchronous Order Processing Plugin Data Flow]]

For SSL certification generation, more than one vendor plugin may be available such as for Dogtag or Symantec, hence the order's details should include which vendor to use for the selection criteria, or else Barbican should support specifying a default vendor/plugin to use.

Barbican then retrieves any state associated with a given order instance, probably via order meta data information stored along with the order record. The plugin could define this status as key/value pairs for example. Since the same plugin may be called multiple times for the same order instance, this persisted state might include a state machine state name that directs which business logic to use within the vendor plugin. If the order instance needs to 'link' to an external system's order reference (such as for Symantec) this could be stored in the meta data as well (as determined by the plugin).

Next Barbican Core must provide IoC components to allow plugins to perform system interaction (such as database updates and event notification) without them directly accessing these critical core components. As depicted in the figure, one IoC handler could present specific methods such as 'notify_ssl_cert_is_ready()' which are handled by Barbican Core as simple log messages for out-of-the-box deployments, or else as CADF messages sent via oslo incubator or Ceilometer for external systems to consume in deployment/company-specific ways. Another IoC handler could 'wrap' data model operations such as 'generate_private_key()' which Barbican Core would implement as a generate/encrypt/store operation in the crypto package.

The order processing plugin can be invoked, perhaps routing flow based on the previous state information, such as for state machine processing for SSL certification processing. The plugin might respond with a status that the Barbican Core logic could use to determine what to do with the plugin next...for example, 'Done' might indicate order processing is completed, 'Continue' might mean persist plugin state with the order for a future plugin call (say via invoked scheduled batch update from the CA), and 'Retry' might mean call the plugin again at a future time to retry an operation.

== Scheduled and Batch Processing ==

The discussion so far has focused on synchronous and asynchronous invocation of plugins for a given workflow or order instance, but some processes might require batch processing across multiple instances. For example, SSL certification processing may involve requesting status from a CA on a scheduled basis. This status might be a batch of multiple SSL certificate orders at once, so Barbican would need to iterate through these order statuses and individually invoke plugin tasks for them. The plugin might provide a batch method that Barbican Core could invoke on a scheduled basis, with a callback function passed in that the plugin calls for each order instance seen in the batch. Barbican Core would implement the callback by enqueueing a plugin RPC task for a worker nodes to process.

== Plugin Source Code Organization ==

The term 'Barbican Core' refers to code found in the [https://github.com/stackforge/barbican stackforge/barbican repository] and includes logic supported the left hand side of the figures above, and an abstract base class defining the interactions to the plugins in the middle of the diagrams. Core should also always include simple example and standalone plugin implementations that are enabled out-of-the-box on local installations. They shouldn't require network access to function and demonstrate, should be well unit-tested and should provide a good example to developers of new plugins.

Beyond these simple default plugins however, it is not as obvious how to manage specific plugin implementations' source code. On one hand it is convenient to bundle with Core source code for specific plugin implementations that are likely to be used for production Barbican installations. For example, Barbican Core does currently include PKCS11 and Dogtag based crypto plugins. On the other hand, these plugins usually have dependencies on libraries that are not part of the OpenStack global requirements, and therefore have to accommodate out-of-the-box deployments that don't have those dependencies installed. Hence thorough unit testing is more difficult (via patching) and code logic is a bit more complicated to deal with missing imports.

Another option is to create separate git repositories for the plugin implementation source files, with a dependency on the Barbican Core source base such as to extend abstract plugin contracts. This approach would simplify the Barbican Core code base, but would require integrating multiple repositories for testing purposes. It would also require mechanisms to extend Barbican to include these external dependencies at package time. This is explored in the next section.

== Discovering 3rd Party Plugins ==

With the bundled crypto plugin implementations that Barbican Core includes now (such as PKCS11 and Dogtag), activating them for usage just requires including their dependencies in the deployed Python package or deployment, and then enabling them via configurations in the /etc/barbican/barbican-api.conf file. Stevedore provides the ability to load these plugins and then to select/use them at run time.

For plugins developed outside of Barbican Core, Stevedore could still be used and in addition to installing non-OpenStack dependencies and adding configuration items to barbican-api.conf, would also require adding a setup.cfg file that defines the new plugin namespaces, aliases and classpaths. A new custom-deployment package could then be created and installed.

Another option is to use a plugin module discovery process similar to the Heat project's Resource discovery. Heat defines a folder location that is searched for new Resources, in the form of Python source files that extend a base Resource.