Jump to: navigation, search

Difference between revisions of "Cinder/tested-3rdParty-drivers"

m
(add info about third-party CI testing of os-brick changes)
 
(33 intermediate revisions by 7 users not shown)
Line 3: Line 3:
 
=== Description ===
 
=== Description ===
  
The Cinder community (and other OpenStack projects) have agreed that if a vendor wishes to submit a driver for their particular storage device that said vendor should also be required to set up a third party CI system in their lab which runs [https://wiki.openstack.org/wiki/Cinder/tested-3rdParty-drivers#What_tests_do_I_use.3F Tempest volume tests] against their storage device for every Cinder commit, and provides feedback in to Gerrit.
+
The Cinder community (and other OpenStack projects) have agreed that if a vendor wishes to submit a driver for their particular storage device that said vendor should also be required to set up a third party CI system in their lab which runs [https://wiki.openstack.org/wiki/Cinder/tested-3rdParty-drivers#What_tests_do_I_use.3F Tempest] against their storage device for every relevant Cinder project commit (usually cinder '''and''' os-brick), and provides feedback in to Gerrit.
  
 
=== Third Party CI Requirements ===
 
=== Third Party CI Requirements ===
 
* See the official [http://ci.openstack.org/third_party.html Third Party Testing] wiki.
 
* See the official [http://ci.openstack.org/third_party.html Third Party Testing] wiki.
* Test all volume drivers your company has integrated in Cinder.
+
* Test all volume/backup/connector drivers your company has integrated in Cinder with each of the supported backends.
 +
* Run on each change to cinder '''and''' os-brick.  (Cinder tests use the latest ''released'' version of os-brick, so bad changes to os-brick master can slip in if you aren't checking.)
 
* Post all test results. Even if you have failure results, those need to be posted.
 
* Post all test results. Even if you have failure results, those need to be posted.
 +
** The posted results must be browseable directly from the web so they can be easily viewed by reviewers.  Do not post a link to an archive file that must be downloaded.
 
* Test all fabrics your solution uses.
 
* Test all fabrics your solution uses.
 +
* Setup your CI to rerun if it is [[Cinder/tested-3rdParty-drivers#How_do_I_trigger_my_CI_to_rerun_on_gerrit_comments.3F|triggered]] to do so.
 +
* You must execute your tests using at least one of the Python runtimes for the current development cycle.
 +
** The current Python runtimes are determined by the OpenStack Technical Committee. See [https://governance.openstack.org/tc/reference/project-testing-interface.html#tested-runtimes Tested Runtimes] in the OpenStack governance documents.
  
  
For example, if your company has two volume drivers in Cinder and they both use ISCSI and FibreChannel, you would need to have a CI that tests against four backends and reports the results for each backend, for every Cinder upstream patch
+
For example, if your company has two volume drivers in Cinder and they both use ISCSI and FibreChannel, you would need to have a CI that tests against four backends and reports the results for each backend, for every Cinder upstream patch.  Likewise, if your company has one volume driver that supports two different backends, you would need to have a CI that tests that driver against both backends and reports results for each backend, for every Cinder upstream patch.
  
 
=== Existing CI Solutions ===
 
=== Existing CI Solutions ===
* Puppet modules for deploying OpenStack CI
+
The CI solution you use is completely up to you.
** [https://github.com/rasselin/os-ext-testing Git repo]
 
** [https://github.com/rasselin/os-ext-testing-data/blob/master/etc/jenkins_jobs/config/dsvm-cinder-driver.yaml.sample Sample Jenkins Job Builder Job Template for Cinder drivers]
 
** Fork of Jay Pipe's external test repo and more up-to-date. Uses nodepool.
 
  
* Jay Pipe's external testing series
+
That being said, we (the Cinder project team) are interested in fostering a community of Third Party CI maintainers so that people can more easily get help in setting up and maintaining a CI system.  (That's because what seems to happen is that a CI gets stood up, works great, the person who set it up moves on to other duties (at the same company or elsewhere), and suddenly one day it's no longer reporting, and a new person has to scramble to figure out how to fix it while things are on fire.)
 +
 
 +
* Our current (as of January 2020) recommendation is: [https://www.softwarefactory-project.io/ Software Factory]
 +
** It's being used by RDO for their CI system
 +
** [https://www.softwarefactory-project.io/docs/3.4/operator/index.html Software Factory Operator Documentation] (lots of other docs & guides, but this is good for a quick look)
 +
** if you try it out, please give us feedback in IRC (#openstack-cinder) or on the mailing list or on this wiki page
 +
 
 +
* You can also try puppet modules for deploying OpenStack CI
 +
** [https://github.com/openstack-infra/puppet-openstackci/tree/master/contrib Git repo]
 +
** [http://lists.openstack.org/pipermail/openstack-dev/2015-November/080058.html Mailing list information about CI]
 +
** Questions can be answered on the #openstack-infra IRC channel
 +
 
 +
* For historical background you can review Jay Pipe's external testing series
 
** '''Note:''' Jay's repo is outdated, but the articles are useful to read. A more updated fork exists.
 
** '''Note:''' Jay's repo is outdated, but the articles are useful to read. A more updated fork exists.
 
** [https://github.com/jaypipes/os-ext-testing Git Repo]
 
** [https://github.com/jaypipes/os-ext-testing Git Repo]
 
** [http://www.joinfu.com/2014/01/understanding-the-openstack-ci-system/ Understanding the OpenStack CI]
 
** [http://www.joinfu.com/2014/01/understanding-the-openstack-ci-system/ Understanding the OpenStack CI]
** [http://www.joinfu.com/2014/02/setting-up-an-external-openstack-testing-system/ Setting up CI Part 1]
+
** [http://www.joinfu.com/2014/02/setting-up-an-external-openstack-testing-system/ Setting up CI Part 1] '''Outdated'''
** [http://www.joinfu.com/2014/02/setting-up-an-openstack-external-testing-system-part-2 Setting up a CI Part 2]
+
** [http://www.joinfu.com/2014/02/setting-up-an-openstack-external-testing-system-part-2 Setting up a CI Part 2] '''Outdated'''
 +
** [http://docs.openstack.org/infra/openstackci/third_party_ci.html Official Third-Party CI Setup Example]
  
 
=== Current Reporting Cinder CI's ===
 
=== Current Reporting Cinder CI's ===
See the [https://wiki.openstack.org/wiki/Cinder/third-party-ci-status list].
+
See the [http://ci-watch.tintri.com/project?project=cinder&time=7+days list].
 +
 
 +
=== Non-Compliance Policy ===
 +
The current policy for CI compliance is:
 +
 
 +
* CI's must report on every patch, whether the code change is in their own driver code or not
 +
* The CI comments must be properly formatted to show up in the CI summary in gerrit
 +
 
 +
 
 +
Non-compliant drivers will be tagged as unsupported if:
 +
* No CI success reporting occurs within a two week span
 +
* The CI is found to not be testing the expected driver (CI runs using the default LVM driver, etc.)
 +
* Other issues are found but failed to be addressed in a timely manner
 +
 
 +
Note: Comment in the #openstack-cinder IRC to let the team know if the CI is down for any period of time. This does not mean it will be exempt from the non-compliance policy, but it will let us know that you are aware that your CI is down and you are working on it.
 +
 
 +
If a driver CI is found non compliant:
 +
* A driver patch will be submitted flagging it as unsupported
 +
** This will cause warning messages to be logged in the c-vol log stating it is unsupported and deprecated
 +
** Operators will need to set enable_unsupported_drivers=True in cinder.conf or the service will fail to load
 +
* If the issue is not corrected before the next release, the driver will subject to removal.  Please see the Cinder [https://docs.openstack.org/cinder/latest/drivers-all-about.html#driver-removal Driver Removal Policy] for details
 +
 
 +
CI results will be reviewed on a regular basis and if found non-compliant, drivers will be flagged. We will try to do a final review around the third milestone of the release. If marked as unsupported, vendors will have until the RC tag to become compliant. The flag can be reverted if addressed in time.
 +
 
 +
CI results are currently posted here: http://cinderstats.ivehearditbothways.com/cireport.txt
  
 
=== Questions ===
 
=== Questions ===
Line 37: Line 76:
  
 
==== What tests do I use? ====
 
==== What tests do I use? ====
Use the OpenStack integration test suite [http://git.openstack.org/cgit/openstack/tempest Tempest]. The volume related tests can be started with the following command from a Tempest repo:
+
You must use ''both'':
 +
* the OpenStack integration test suite [http://git.openstack.org/cgit/openstack/tempest Tempest]
 +
* the [https://opendev.org/openstack/cinder-tempest-plugin cinder-tempest-plugin] which will run additional tempest-based tests specifically for Cinder
 +
 
 +
 
 +
Tempest will automatically discover any installed plugins when it is run. So by just installing the python packages containing the cinder-tempest-plugin, you’ll be using it with Tempest, nothing else is really required.
 +
 
 +
===== Volume & Connector Drivers =====
 +
The volume related tests can be started with the following command from a Tempest repo:
  
 
<pre>
 
<pre>
Line 46: Line 93:
 
<pre>
 
<pre>
 
export DEVSTACK_GATE_TEMPEST_REGEX="volume"
 
export DEVSTACK_GATE_TEMPEST_REGEX="volume"
 +
</pre>
 +
 +
===== Backup Drivers =====
 +
 +
<pre>
 +
tox -e all -- volume_backup | tee -a console.log.out
 +
</pre>
 +
 +
For those using devstack-gate export this variable before running the job:
 +
<pre>
 +
export DEVSTACK_GATE_TEMPEST_REGEX="volume_backup"
 
</pre>
 
</pre>
  
Line 65: Line 123:
  
 
# These options allow you to specify a branch other than "master" be used
 
# These options allow you to specify a branch other than "master" be used
CINDER_REPO=https://review.openstack.org/openstack/cinder
+
CINDER_REPO=https://opendev.org/openstack/cinder
 
CINDER_BRANCH=refs/changes/83/72183/4
 
CINDER_BRANCH=refs/changes/83/72183/4
  
Line 76: Line 134:
 
volume_driver=cinder.volume.drivers.foo.FooDriver
 
volume_driver=cinder.volume.drivers.foo.FooDriver
 
</pre>
 
</pre>
 +
 +
==== What changes should I test on? ====
 +
 +
You should test on all changes to '''cinder''' and '''os-brick'''.
 +
* You want to protect yourself from cinder changes that somehow impact your driver.  Note that because third-party CI systems are non-voting, a failure will '''not''' block a patch from merging.  Reviewers will (usually) check your third-party CI results on a change specifically to your driver's code, but they may not notice on patches to main cinder code that doesn't directly touch your driver.  So it's important for you to keep an eye on your CI's failures, and add a manual -1 if it's a real failure.
 +
 +
* The os-brick used in '''cinder''' CI testing is the latest appropriate '''released''' version of os-brick.  Thus, it is possible for changes to be happening in os-brick that might impact the functionality of your driver.  If you aren't testing os-brick changes, you won't find out about these until after the next os-brick release, which will really suck for your customers.
 +
 +
* NOTE: The expectation that third-party CIs would test os-brick changes had not been communicated clearly, and hence some drivers test and some don't.  Thus, this is not a hard requirement until the OpenStack 2023.1 release development cycle (that is, the cycle after the Zed release).
  
 
==== How do I run my CI to test all cinder patches with my driver not yet merged? ====
 
==== How do I run my CI to test all cinder patches with my driver not yet merged? ====
Line 86: Line 153:
  
 
     # fetch latest patchset from gerrit
 
     # fetch latest patchset from gerrit
     UPSTREAM_REMOTE=https://review.openstack.org/openstack/cinder
+
     UPSTREAM_REMOTE=https://opendev.org/openstack/cinder
  
 
     # set this to your gerrit change number
 
     # set this to your gerrit change number
Line 124: Line 191:
 
==== When thirdparty CI voting will be required? ====
 
==== When thirdparty CI voting will be required? ====
 
Once third party CI's become more common and stable, we'll revisit the subject. This was agreed in a  [http://eavesdrop.openstack.org/meetings/cinder/2014/cinder.2014-10-15-16.00.log.html discussion] on October 15th 2014. In a [http://eavesdrop.openstack.org/meetings/cinder/2015/cinder.2015-04-08-16.00.log.html#l-34 discussion] on April 8th 2015, it was agreed that open source solutions that have Infra hosted CI's could have the chance to vote.
 
Once third party CI's become more common and stable, we'll revisit the subject. This was agreed in a  [http://eavesdrop.openstack.org/meetings/cinder/2014/cinder.2014-10-15-16.00.log.html discussion] on October 15th 2014. In a [http://eavesdrop.openstack.org/meetings/cinder/2015/cinder.2015-04-08-16.00.log.html#l-34 discussion] on April 8th 2015, it was agreed that open source solutions that have Infra hosted CI's could have the chance to vote.
 +
 +
As of July 2022, the third party CI systems are unfortunately so unstable that voting is out of the question.
  
 
==== How do I test FC drivers? ====
 
==== How do I test FC drivers? ====
 
Use PCI Passthrough to give the VM access to the FC PCI Device
 
Use PCI Passthrough to give the VM access to the FC PCI Device
Scripts available here: https://github.com/openstack/third-party-ci-tools/tree/master/provisioning_scripts/fibre_channel
+
Scripts available here: https://opendev.org/x/third-party-ci-tools/src/branch/master/provisioning_scripts/fibre_channel
 +
 
 +
==== How do I trigger my CI to rerun on gerrit comments? ====
 +
It is required for your CI to rerun jobs if the following comment is posted in a gerrit review:
 +
 
 +
<code>
 +
"run-<CI NAME>"
 +
</code>
 +
 
 +
So if your CI account name is "Super Cool Storage CI", the CI should be able to be retriggered by leaving the comment:
 +
 
 +
<code>
 +
run-Super Cool Storage CI
 +
</code>
 +
 
 +
CI's do not need to kick off tests until Zuul's has given a +1, though it is up to the maintainer if you want to trigger immediately on all patchsets.
 +
 
 +
For the openstack-ci configuration, change the following to switch to only trigger on a Zuul's +1:
 +
 
 +
In layout.yaml:
 +
Delete the line:
 +
<pre>
 +
      - event: patchset-created
 +
</pre>
 +
And then add:
 +
<pre>
 +
      - event: comment-added
 +
        require-approval:
 +
            - verified: 1
 +
              username: zuul
 +
</pre>
 +
 
 +
==== What about stable branch testing? ====
 +
 
 +
The current policy is described in this posting to the openstack-discuss mailing list:
 +
http://lists.openstack.org/pipermail/openstack-discuss/2022-February/027021.html
  
 +
Here's an example of what we're looking for: [https://review.opendev.org/c/openstack/cinder/+/821893/3#message-ade9aa6ad8bd99fefab908c777fe106e907c7636 https://review.opendev.org/c/openstack/cinder/+/821893/]
  
 
[[Category: Cinder]]
 
[[Category: Cinder]]

Latest revision as of 14:08, 19 July 2022

Driver Testing

Description

The Cinder community (and other OpenStack projects) have agreed that if a vendor wishes to submit a driver for their particular storage device that said vendor should also be required to set up a third party CI system in their lab which runs Tempest against their storage device for every relevant Cinder project commit (usually cinder and os-brick), and provides feedback in to Gerrit.

Third Party CI Requirements

  • See the official Third Party Testing wiki.
  • Test all volume/backup/connector drivers your company has integrated in Cinder with each of the supported backends.
  • Run on each change to cinder and os-brick. (Cinder tests use the latest released version of os-brick, so bad changes to os-brick master can slip in if you aren't checking.)
  • Post all test results. Even if you have failure results, those need to be posted.
    • The posted results must be browseable directly from the web so they can be easily viewed by reviewers. Do not post a link to an archive file that must be downloaded.
  • Test all fabrics your solution uses.
  • Setup your CI to rerun if it is triggered to do so.
  • You must execute your tests using at least one of the Python runtimes for the current development cycle.
    • The current Python runtimes are determined by the OpenStack Technical Committee. See Tested Runtimes in the OpenStack governance documents.


For example, if your company has two volume drivers in Cinder and they both use ISCSI and FibreChannel, you would need to have a CI that tests against four backends and reports the results for each backend, for every Cinder upstream patch. Likewise, if your company has one volume driver that supports two different backends, you would need to have a CI that tests that driver against both backends and reports results for each backend, for every Cinder upstream patch.

Existing CI Solutions

The CI solution you use is completely up to you.

That being said, we (the Cinder project team) are interested in fostering a community of Third Party CI maintainers so that people can more easily get help in setting up and maintaining a CI system. (That's because what seems to happen is that a CI gets stood up, works great, the person who set it up moves on to other duties (at the same company or elsewhere), and suddenly one day it's no longer reporting, and a new person has to scramble to figure out how to fix it while things are on fire.)

  • Our current (as of January 2020) recommendation is: Software Factory
    • It's being used by RDO for their CI system
    • Software Factory Operator Documentation (lots of other docs & guides, but this is good for a quick look)
    • if you try it out, please give us feedback in IRC (#openstack-cinder) or on the mailing list or on this wiki page

Current Reporting Cinder CI's

See the list.

Non-Compliance Policy

The current policy for CI compliance is:

  • CI's must report on every patch, whether the code change is in their own driver code or not
  • The CI comments must be properly formatted to show up in the CI summary in gerrit


Non-compliant drivers will be tagged as unsupported if:

  • No CI success reporting occurs within a two week span
  • The CI is found to not be testing the expected driver (CI runs using the default LVM driver, etc.)
  • Other issues are found but failed to be addressed in a timely manner

Note: Comment in the #openstack-cinder IRC to let the team know if the CI is down for any period of time. This does not mean it will be exempt from the non-compliance policy, but it will let us know that you are aware that your CI is down and you are working on it.

If a driver CI is found non compliant:

  • A driver patch will be submitted flagging it as unsupported
    • This will cause warning messages to be logged in the c-vol log stating it is unsupported and deprecated
    • Operators will need to set enable_unsupported_drivers=True in cinder.conf or the service will fail to load
  • If the issue is not corrected before the next release, the driver will subject to removal. Please see the Cinder Driver Removal Policy for details

CI results will be reviewed on a regular basis and if found non-compliant, drivers will be flagged. We will try to do a final review around the third milestone of the release. If marked as unsupported, vendors will have until the RC tag to become compliant. The flag can be reverted if addressed in time.

CI results are currently posted here: http://cinderstats.ivehearditbothways.com/cireport.txt

Questions

  • Join Third Party Meeting
  • Reach out to IRC nicks DuncanT or asselin on Freenode #openstack-cinder.

FAQ

What tests do I use?

You must use both:

  • the OpenStack integration test suite Tempest
  • the cinder-tempest-plugin which will run additional tempest-based tests specifically for Cinder


Tempest will automatically discover any installed plugins when it is run. So by just installing the python packages containing the cinder-tempest-plugin, you’ll be using it with Tempest, nothing else is really required.

Volume & Connector Drivers

The volume related tests can be started with the following command from a Tempest repo:

tox -e all -- volume | tee -a console.log.out

For those using devstack-gate export this variable before running the job:

export DEVSTACK_GATE_TEMPEST_REGEX="volume"
Backup Drivers
tox -e all -- volume_backup | tee -a console.log.out

For those using devstack-gate export this variable before running the job:

export DEVSTACK_GATE_TEMPEST_REGEX="volume_backup"

How do I configure DevStack so my Driver Passes Tempest?

Sample local.conf for devstack setup.

[[local|localrc]]
ADMIN_PASSWORD=password
MYSQL_PASSWORD=password
RABBIT_PASSWORD=password
SERVICE_PASSWORD=password
SERVICE_TOKEN=password

# These options define  expected driver capabilities
TEMPEST_VOLUME_DRIVER=foo
TEMPEST_VOLUME_VENDOR="Foo Inc"
TEMPEST_STORAGE_PROTOCOL=iSCSI

# These options allow you to specify a branch other than "master" be used
CINDER_REPO=https://opendev.org/openstack/cinder
CINDER_BRANCH=refs/changes/83/72183/4

# Disable security groups entirely
Q_USE_SECGROUP=False
LIBVIRT_FIREWALL_DRIVER=nova.virt.firewall.NoopFirewallDriver
CINDER_SECURE_DELETE=False

[[post-config|$CINDER_CONF]]
volume_driver=cinder.volume.drivers.foo.FooDriver

What changes should I test on?

You should test on all changes to cinder and os-brick.

  • You want to protect yourself from cinder changes that somehow impact your driver. Note that because third-party CI systems are non-voting, a failure will not block a patch from merging. Reviewers will (usually) check your third-party CI results on a change specifically to your driver's code, but they may not notice on patches to main cinder code that doesn't directly touch your driver. So it's important for you to keep an eye on your CI's failures, and add a manual -1 if it's a real failure.
  • The os-brick used in cinder CI testing is the latest appropriate released version of os-brick. Thus, it is possible for changes to be happening in os-brick that might impact the functionality of your driver. If you aren't testing os-brick changes, you won't find out about these until after the next os-brick release, which will really suck for your customers.
  • NOTE: The expectation that third-party CIs would test os-brick changes had not been communicated clearly, and hence some drivers test and some don't. Thus, this is not a hard requirement until the OpenStack 2023.1 release development cycle (that is, the cycle after the Zed release).

How do I run my CI to test all cinder patches with my driver not yet merged?

If using devstack-gate use the pre-test-hook to cherry-pick your driver on top of the cinder patch under review. If you've submitted your change to gerrit, you can use something like the following:

function pre_test_hook {
    echo "Cherry-picking latest driver from gerrit"
    cd $BASE/new/cinder

    # fetch latest patchset from gerrit
    UPSTREAM_REMOTE=https://opendev.org/openstack/cinder

    # set this to your gerrit change number
    CHANGE_NUM=999999

    PATCHSET_BASE=refs/changes/${CHANGE_NUM:(-2)}/$CHANGE_NUM
    LATEST_PATCHSET=$(git ls-remote $UPSTREAM_REMOTE $PATCHSET_BASE/\* |
                      sort -t/ -k 5 -n | tail -n1 | cut -d$'\t' -f2)

    if [ -z "$LATEST_PATCHSET" ]; then
        echo "Failed to determine latest patchset of $PATCHSET_BASE from $UPSTREAM_REMOTE"
        exit 1
    fi

    echo "Latest patchset ref is $LATEST_PATCHSET"

    git fetch $UPSTREAM_REMOTE $LATEST_PATCHSET && git cherry-pick FETCH_HEAD
}

If you haven't submitted your change to gerrit yet, you'll have to fetch the change from your own repository (github or otherwise). For example:

function pre_test_hook {
    echo "Cherry-picking latest driver from private repository"
    cd $BASE/new/cinder

    VENDOR_REMOTE=https://www.github.com/vendor/cinder-driver
    REFSPEC=bp/new-cinder-driver

    git fetch $VENDOR_REMOTE $REFSPEC && git cherry-pick FETCH_HEAD
}

Otherwise you can make the changes prior to calling stack.sh or via a custom devstack plugin.

When thirdparty CI voting will be required?

Once third party CI's become more common and stable, we'll revisit the subject. This was agreed in a discussion on October 15th 2014. In a discussion on April 8th 2015, it was agreed that open source solutions that have Infra hosted CI's could have the chance to vote.

As of July 2022, the third party CI systems are unfortunately so unstable that voting is out of the question.

How do I test FC drivers?

Use PCI Passthrough to give the VM access to the FC PCI Device Scripts available here: https://opendev.org/x/third-party-ci-tools/src/branch/master/provisioning_scripts/fibre_channel

How do I trigger my CI to rerun on gerrit comments?

It is required for your CI to rerun jobs if the following comment is posted in a gerrit review:

"run-<CI NAME>"

So if your CI account name is "Super Cool Storage CI", the CI should be able to be retriggered by leaving the comment:

run-Super Cool Storage CI

CI's do not need to kick off tests until Zuul's has given a +1, though it is up to the maintainer if you want to trigger immediately on all patchsets.

For the openstack-ci configuration, change the following to switch to only trigger on a Zuul's +1:

In layout.yaml: Delete the line:

      - event: patchset-created

And then add:

      - event: comment-added
        require-approval:
            - verified: 1
              username: zuul

What about stable branch testing?

The current policy is described in this posting to the openstack-discuss mailing list: http://lists.openstack.org/pipermail/openstack-discuss/2022-February/027021.html

Here's an example of what we're looking for: https://review.opendev.org/c/openstack/cinder/+/821893/