Jump to: navigation, search

Difference between revisions of "Cinder/how-to-contribute-a-driver"

(update gerrit workflow link)
m (Before you write any code)
 
(60 intermediate revisions by 9 users not shown)
Line 1: Line 1:
 
= How To Contribute a driver to Cinder =
 
= How To Contribute a driver to Cinder =
  
== Deadline for Kilo ==
+
=== Deadline for the Current Development Cycle ===
Review the OpenStack dev mailing list [http://lists.openstack.org/pipermail/openstack-dev/2014-October/049512.html post] about his.
+
The deadline for new backend drivers is Milestone 2 of the current development cycle.  By this deadline, your driver should have working third party CI and no code review issues.  You can find the date of Milestone 2 on the [https://releases.openstack.org/index.html current release] schedule. (Take the link for the "Series" name, and a link to the release schedule will be at the top of the page.)  If you are working toward the ''next'' cycle, and the schedule has not yet been posted, you can get a rough idea by looking at previous release schedules.
  
== Third Party CI Requirement Policy ==
+
For clarification, a patch is also considered a new driver when it's introducing a new protocol into the driver. For example, if your driver supports iSCSI, but your patch proposes support for FibreChannel, that's a new driver patch.
One thing that has been lacking for plugins over the past is any sort of formal CI testing.  The OpenStack Project as a whole has had an extremely comprehensive CI implementation for some time now that runs against reference implementations and some of the more common optional configs that are out there. As a Community Cinder (and other projects as well) have agreed that if a vendor wishes to submit a driver for their particular storage device that said vendor should also be required to set up a third party CI system in their lab which runs tempest-dsvm-full against their device for every Cinder commit, and provides feedback in to Gerrit.
 
  
This is very much an evolving process and is subject to change, but currently here's where we stand as of Nov, 2014:
+
=== All About Cinder Drivers ===
* All vendors with a driver in the Cinder code-base merged before Kilo opened are required to have 3'rd party CI testing prior to K-2 (Feb 5th 2015)
 
* Drivers merged in Kilo should target CI by the end of Kilo or risk removal in early L.
 
* Every commit made to Cinder should be ran against the vendors 3'rd party CI environment
 
* Currently the tests that should be run are 'tempest-dsvm-full'
 
* Your version should follow a naming template like: "tempest-dsvm-full-<DriverName>"
 
* Results/logs should be reported just as they are with the OpenStack Infra CI systems
 
* If a vendor has more than one driver, they need more than one CI system.  In other words if you have 3 drivers, you'll be expected to test all 3 of those drivers.
 
  
There are a number of resources out there to help deploy your own CI environment.  One of the best sources currently is a series of blog posting from Jay Pipes that can be found here: http://www.joinfu.com/2014/02/setting-up-an-openstack-external-testing-system-part-2/
+
See [https://docs.openstack.org/cinder/latest/drivers-all-about.html All About Cinder Drivers] for general information about drivers, including a discussion of "supported" vs. "unsupported" drivers and driver compliance.
  
As I mentioned this is an evolving process, there's a good deal more information that's needed here to help folks and we'll get that flushed out as we go along and start having some successes on this.
+
=== Third Party CI Requirement Policy ===
 +
See [https://wiki.openstack.org/wiki/Cinder/tested-3rdParty-drivers third party CI] wiki.
  
== Before you write any code ==
+
For people working on getting their CI to handle Python 3, see [https://wiki.openstack.org/wiki/Cinder/3rdParty-drivers-py3-update Cinder Third Party CI update to Python 3.7]
 +
 
 +
=== Before you write any code ===
 
* Read the [https://wiki.openstack.org/wiki/How_To_Contribute#If_you.27re_a_developer How To Contribute Page].
 
* Read the [https://wiki.openstack.org/wiki/How_To_Contribute#If_you.27re_a_developer How To Contribute Page].
  
Line 29: Line 24:
 
* Cinder offers a reference implementation that should be used as a model.  The reference implementation driver file is [https://github.com/openstack/cinder/blob/master/cinder/volume/drivers/lvm.py cinder/volume/drivers/lvm.py], not to be mistaken for cinder/volume/driver.py which is the base class that all of the drivers inherit from. Note that there are a lot of options that show up there regarding iSCSI targets etc, but this gives you an idea of the expectations in terms of features that are implemented and some of the behaviors.  I strongly recommend loading up [http://devstack.org devstack] (you're going to need it to test your driver anyway) and play around with the default LVM.  It's really important that you get a feel for how Cinder works and interacts with the other OpenStack projects before you get too far along.
 
* Cinder offers a reference implementation that should be used as a model.  The reference implementation driver file is [https://github.com/openstack/cinder/blob/master/cinder/volume/drivers/lvm.py cinder/volume/drivers/lvm.py], not to be mistaken for cinder/volume/driver.py which is the base class that all of the drivers inherit from. Note that there are a lot of options that show up there regarding iSCSI targets etc, but this gives you an idea of the expectations in terms of features that are implemented and some of the behaviors.  I strongly recommend loading up [http://devstack.org devstack] (you're going to need it to test your driver anyway) and play around with the default LVM.  It's really important that you get a feel for how Cinder works and interacts with the other OpenStack projects before you get too far along.
  
* You don't need a [https://github.com/openstack/cinder-specs cinder spec] for most drivers. You always need to submit a [[Blueprints|blueprint]] in [https://launchpad.net/cinder Launchpad] introducing your driver, so that it can be targeted for a release.
+
* You must submit a [[Blueprints|blueprint]] in [https://launchpad.net/cinder Launchpad] introducing your driver, so that it can be targeted for a release, and prioritized to receive code reviews.  This will also get your effort to contribute a driver some visibility within the Cinder community.  Do this right away so that you don't get any surprises later.  You don't need a [https://github.com/openstack/cinder-specs cinder spec] for most drivers. (If a spec is required for some reason, we will let you know.)
  
* We have a development channel on freenode: #openstack-cinder. There are developers here round the clock, it's a great resource for you get started.  Log in, ask questions, don't stare at code in isolation for a week... if you're stuck on something just ask.  There's also no need to start off with "Can I ask a question"... you likely won't get a response.  Just type in your question, that way anybody monitoring the channel that might know the answer can step in and answer.
+
* We have a development channel on OFTC: #openstack-cinder. There are developers here round the clock, it's a great resource for you get started.  Log in, ask questions, don't stare at code in isolation for a week... if you're stuck on something just ask.  There's also no need to start off with "Can I ask a question"... you likely won't get a response.  Just type in your question, that way anybody monitoring the channel that might know the answer can step in and answer.
  
== Writing Code ==
+
=== Writing Code ===
* You must implement all of the methods that exist as [http://docs.openstack.org/developer/cinder/devref/drivers.html core features].
+
* You must implement all of the methods that exist as [https://docs.openstack.org/cinder/latest/contributor/drivers.html core features].
  
* Your driver should not make any state changes. (e.g. make Cinder database calls). The [https://github.com/openstack/cinder/blob/master/cinder/volume/manager.py volume manager] is responsible for making state changes after the driver is done talking to the storage backend.
+
* Your driver should not make any state changes. (e.g. make Cinder database calls). The [https://github.com/openstack/cinder/blob/master/cinder/volume/manager.py volume manager] is responsible for making state changes after the driver is done talking to the storage backend. Your driver should try not to read from database if possible.
  
* Unit tests for new code are required.  We're in the process of converting everything to use [http://www.voidspace.org.uk/python/mock/ mock] (rather than mox) for our unit tests.  Be sure when writing unit tests and setting up fakes to use mock, examples of it's usage can be found in the existing tests like cinder/tests/test_volume.py.
+
* Unit tests for new code are required.
  
* Make sure you're not duplicating a configuration option that already exists. To verify this, you'll need to need look at the cinder/etc/cinder.conf.sample file. To generate this file:
+
* Make sure you're not duplicating a configuration option that already exists. To verify this, you'll need to need look at the [https://docs.openstack.org/cinder/latest/configuration/block-storage/samples/cinder.conf.html cinder/etc/cinder.conf.sample file].
** Install [https://pypi.python.org/pypi/tox tox]
+
** Every effort should be made to reuse common config options before introducing driver-specific ones.
** Run tox -egenconfig
+
** If options are added, you will need to run "tox -e genopts" to update the opts.py file that is used to generate our configuration references.
  
 
* Make sure to follow the [http://docs.openstack.org/developer/hacking/ OpenStack Style Guide]. Very likely you'll get nit pick reviews otherwise, which is not productive either way.
 
* Make sure to follow the [http://docs.openstack.org/developer/hacking/ OpenStack Style Guide]. Very likely you'll get nit pick reviews otherwise, which is not productive either way.
  
* Use [http://docs.openstack.org/developer/oslo.i18n/usage.html log markers] by importing [https://github.com/openstack/cinder/blob/master/cinder/i18n.py cinder.i18n]
+
* Cinder's [https://github.com/openstack/cinder/blob/master/cinder/volume/manager.py manager layer] will log useful information like failures from the driver that are raised. However, you're more than welcome to add additional logging, but please follow our [http://specs.openstack.org/openstack/openstack-specs/specs/log-guidelines.html logging guideline].  (NOTE: In order to assist operators using public internet search engines to troubleshoot problems, log messages are no longer translated.)
 +
 
 +
* Cryptography
 +
** Drivers must not use md5 for any security-related purpose.  (In fact, drivers should avoid using it at all, because some security audits only allow a "yes"/"no" checkbox for md5 use ... but that's up to the vendor.)
 +
*** If md5 *is* being used for a non security-related purpose, the code must use oslo.utils and not call hashlib directly to access md5.  Here's an example of how to do this: https://review.opendev.org/c/openstack/os-brick/+/756151
 +
** Any cryptography done by a driver should be implemented by using a well-respected cryptographic library.  ''Under no circumstances should a driver implement its own cryptographic functions.''
 +
 
 +
=== Submitting Driver For Review ===
 +
 
 +
When submitting your driver, please include a release note along with your patch. See the [http://docs.openstack.org/developer/reno/usage.html#creating-new-release-notes Reno Documentation] for details on how to generate new release notes.
 +
 
 +
The release note should be something along the lines of:
 +
 
 +
---
 +
features:
 +
  - Added backend driver for ''vendor'' storage.
  
== Submit Driver For Review ==
+
* All new code must be Python 3 only. Unit tests will be run with all of the current cycle's supported runtimes, but additional runtime testing should be performed. Check the [https://governance.openstack.org/tc/reference/runtimes/ supported runtimes] for the current cycle.
* Do '''NOT''' bother the Cinder core team for reviews. We are aware of your patch being posted.
+
** for the Yoga cycle: python 3.8 and python 3.9
 +
 
 +
* Do '''NOT''' bother the Cinder team for reviews. We are aware of your patch being posted.
  
 
* Make sure your commit message follows the [[GitCommitMessages|OpenStack project guidelines]].
 
* Make sure your commit message follows the [[GitCommitMessages|OpenStack project guidelines]].
  
* Make sure your driver has appropriate [[Cinder/tested-3rdParty-drivers|third party testing]] done.
+
* Make sure your driver has appropriate [[Cinder/tested-3rdParty-drivers|third party testing]] done. It is required that your CI posts the [https://wiki.openstack.org/wiki/Cinder/tested-3rdParty-drivers#What_tests_do_I_use.3F necessary tests pass]. Since your driver is not yet merged, follow [https://wiki.openstack.org/wiki/Cinder/tested-3rdParty-drivers#How_do_I_run_my_CI_to_test_all_cinder_patches_with_my_driver_not_yet_merged.3F instructions] to have you unmerged driver properly tested.
 +
 
 +
* Documentation is now kept in the openstack/cinder repo. At a minimum, please add a basic description of your driver and its configuration options under the [https://github.com/openstack/cinder/tree/master/doc/source/configuration/block-storage/drivers volume driver section of the Configuration reference].
  
 
*[http://docs.openstack.org/infra/manual/developers.html#development-workflow Post your review].
 
*[http://docs.openstack.org/infra/manual/developers.html#development-workflow Post your review].
 +
 +
=== After Your Driver Is Added ===
 +
Congratulations! You're not done yet though. After your driver has been merged there are still some things that need to be done.
 +
* Make sure there is documentation for your driver so that operators know how to configure it
 +
** documentation is stored in the same repository where you committed your driver code
 +
** files go in '''cinder/doc/source/configuration/block-storage/drivers'''
 +
** look at the other files in that directory to get an idea of what's expected
 +
*** note: for a backup driver, your doc goes in '''cinder/doc/source/configuration/block-storage/backup'''
 +
* Update '''cinder/doc/source/reference/support-matrix.ini''' with the appropriate information for your driver
 +
* Continue to be available on IRC and attend the weekly meetings in case questions come up.
 +
* Subscribe to receive bugs for your driver! The Cinder team will be triaging bugs and will tag bugs with the name of your company that are related to your driver. To subscribe:
 +
** Go to https://bugs.launchpad.net/openstack/cinder/+bug
 +
** Click 'Subscribe to bug mail'
 +
** Click the radio button "are added and changed in any way"
 +
** Click checkbox 'Bugs must match this filter (...)"
 +
** Click tags
 +
** With "match all tags" selected type in the field the name of your company.
 +
 +
[[Category: Cinder]]

Latest revision as of 13:22, 19 July 2022

How To Contribute a driver to Cinder

Deadline for the Current Development Cycle

The deadline for new backend drivers is Milestone 2 of the current development cycle. By this deadline, your driver should have working third party CI and no code review issues. You can find the date of Milestone 2 on the current release schedule. (Take the link for the "Series" name, and a link to the release schedule will be at the top of the page.) If you are working toward the next cycle, and the schedule has not yet been posted, you can get a rough idea by looking at previous release schedules.

For clarification, a patch is also considered a new driver when it's introducing a new protocol into the driver. For example, if your driver supports iSCSI, but your patch proposes support for FibreChannel, that's a new driver patch.

All About Cinder Drivers

See All About Cinder Drivers for general information about drivers, including a discussion of "supported" vs. "unsupported" drivers and driver compliance.

Third Party CI Requirement Policy

See third party CI wiki.

For people working on getting their CI to handle Python 3, see Cinder Third Party CI update to Python 3.7

Before you write any code

  • Understand how Cinder works, what it's used for, why the other projects in OpenStack may or may not use it. Fully understand the difference between ephemeral storage on the Nova side versus the persistent storage offered by Cinder
  • Cinder offers a reference implementation that should be used as a model. The reference implementation driver file is cinder/volume/drivers/lvm.py, not to be mistaken for cinder/volume/driver.py which is the base class that all of the drivers inherit from. Note that there are a lot of options that show up there regarding iSCSI targets etc, but this gives you an idea of the expectations in terms of features that are implemented and some of the behaviors. I strongly recommend loading up devstack (you're going to need it to test your driver anyway) and play around with the default LVM. It's really important that you get a feel for how Cinder works and interacts with the other OpenStack projects before you get too far along.
  • You must submit a blueprint in Launchpad introducing your driver, so that it can be targeted for a release, and prioritized to receive code reviews. This will also get your effort to contribute a driver some visibility within the Cinder community. Do this right away so that you don't get any surprises later. You don't need a cinder spec for most drivers. (If a spec is required for some reason, we will let you know.)
  • We have a development channel on OFTC: #openstack-cinder. There are developers here round the clock, it's a great resource for you get started. Log in, ask questions, don't stare at code in isolation for a week... if you're stuck on something just ask. There's also no need to start off with "Can I ask a question"... you likely won't get a response. Just type in your question, that way anybody monitoring the channel that might know the answer can step in and answer.

Writing Code

  • You must implement all of the methods that exist as core features.
  • Your driver should not make any state changes. (e.g. make Cinder database calls). The volume manager is responsible for making state changes after the driver is done talking to the storage backend. Your driver should try not to read from database if possible.
  • Unit tests for new code are required.
  • Make sure you're not duplicating a configuration option that already exists. To verify this, you'll need to need look at the cinder/etc/cinder.conf.sample file.
    • Every effort should be made to reuse common config options before introducing driver-specific ones.
    • If options are added, you will need to run "tox -e genopts" to update the opts.py file that is used to generate our configuration references.
  • Make sure to follow the OpenStack Style Guide. Very likely you'll get nit pick reviews otherwise, which is not productive either way.
  • Cinder's manager layer will log useful information like failures from the driver that are raised. However, you're more than welcome to add additional logging, but please follow our logging guideline. (NOTE: In order to assist operators using public internet search engines to troubleshoot problems, log messages are no longer translated.)
  • Cryptography
    • Drivers must not use md5 for any security-related purpose. (In fact, drivers should avoid using it at all, because some security audits only allow a "yes"/"no" checkbox for md5 use ... but that's up to the vendor.)
    • Any cryptography done by a driver should be implemented by using a well-respected cryptographic library. Under no circumstances should a driver implement its own cryptographic functions.

Submitting Driver For Review

When submitting your driver, please include a release note along with your patch. See the Reno Documentation for details on how to generate new release notes.

The release note should be something along the lines of:

---
features:
 - Added backend driver for vendor storage.
  • All new code must be Python 3 only. Unit tests will be run with all of the current cycle's supported runtimes, but additional runtime testing should be performed. Check the supported runtimes for the current cycle.
    • for the Yoga cycle: python 3.8 and python 3.9
  • Do NOT bother the Cinder team for reviews. We are aware of your patch being posted.

After Your Driver Is Added

Congratulations! You're not done yet though. After your driver has been merged there are still some things that need to be done.

  • Make sure there is documentation for your driver so that operators know how to configure it
    • documentation is stored in the same repository where you committed your driver code
    • files go in cinder/doc/source/configuration/block-storage/drivers
    • look at the other files in that directory to get an idea of what's expected
      • note: for a backup driver, your doc goes in cinder/doc/source/configuration/block-storage/backup
  • Update cinder/doc/source/reference/support-matrix.ini with the appropriate information for your driver
  • Continue to be available on IRC and attend the weekly meetings in case questions come up.
  • Subscribe to receive bugs for your driver! The Cinder team will be triaging bugs and will tag bugs with the name of your company that are related to your driver. To subscribe:
    • Go to https://bugs.launchpad.net/openstack/cinder/+bug
    • Click 'Subscribe to bug mail'
    • Click the radio button "are added and changed in any way"
    • Click checkbox 'Bugs must match this filter (...)"
    • Click tags
    • With "match all tags" selected type in the field the name of your company.