Cinder/tested-3rdParty-drivers

= Driver Testing =

Description
The Cinder community (and other OpenStack projects) have agreed that if a vendor wishes to submit a driver for their particular storage device that said vendor should also be required to set up a third party CI system in their lab which runs Tempest against their storage device for every Cinder commit, and provides feedback in to Gerrit.

Third Party CI Requirements

 * See the official Third Party Testing wiki.
 * Test all volume/backup/connector drivers your company has integrated in Cinder.
 * Post all test results. Even if you have failure results, those need to be posted.
 * Test all fabrics your solution uses.
 * Setup your CI to rerun if it is triggered to do so.

For example, if your company has two volume drivers in Cinder and they both use ISCSI and FibreChannel, you would need to have a CI that tests against four backends and reports the results for each backend, for every Cinder upstream patch

Existing CI Solutions

 * Puppet modules for deploying OpenStack CI
 * Git repo
 * Mailing list information about CI
 * Questions can be answered on the #openstack-infra IRC channel


 * For historical background you can review Jay Pipe's external testing series
 * Note: Jay's repo is outdated, but the articles are useful to read. A more updated fork exists.
 * Git Repo
 * Understanding the OpenStack CI
 * Setting up CI Part 1 Outdated
 * Setting up a CI Part 2 Outdated
 * Official Third-Party CI Setup Example

Current Reporting Cinder CI's
See the list.

Non-Compliance Policy
The current policy for CI compliance is:


 * CI's must report on every patch, whether the code change is in their own driver code or not
 * The CI comments must be properly formatted to show up in the CI summary in gerrit

Non-compliant drivers will be tagged as unsupported if:
 * No CI success reporting occurs within a two week span
 * The CI is found to not be testing the expected driver (CI runs using the default LVM driver, etc.)
 * Other issues are found but failed to be addressed in a timely manner

Note: Comment in the #openstack-cinder IRC to let the team know if the CI is down for any period of time. This does not mean it will be exempt from the non-compliance policy, but it will help to make sure the group is aware that.

If a driver CI is found non compliant:
 * A driver patch will be submitted flagging it as unsupported
 * This will cause warning messages to be logged in the c-vol log stating it is unsupported and deprecated
 * Operators will need to set enable_unsupported_drivers=True in cinder.conf or the service will fail to load
 * If the issue is not corrected before the next release, the driver will be removed from the tree per the normal deprecation policy

CI results will be reviewed on a regular basis and if found non-compliant, drivers will be flagged. We will try to do a final review around the third milestone of the release. If marked as unsupported, vendors will have until the RC tag to become compliant. The flag can be reverted if addressed in time.

CI results are currently posted here: http://cinderstats-dellstorage.rhcloud.com/cireport.txt

Questions

 * Join Third Party Meeting
 * Reach out to IRC nicks DuncanT or asselin on Freenode #openstack-cinder.

What tests do I use?
Use the OpenStack integration test suite Tempest.

Volume & Connector Drivers
The volume related tests can be started with the following command from a Tempest repo:

tox -e all -- volume | tee -a console.log.out

For those using devstack-gate export this variable before running the job: export DEVSTACK_GATE_TEMPEST_REGEX="volume"

Backup Drivers
tox -e all -- volume_backup | tee -a console.log.out

For those using devstack-gate export this variable before running the job: export DEVSTACK_GATE_TEMPEST_REGEX="volume_backup"

How do I configure DevStack so my Driver Passes Tempest?
Sample local.conf for devstack setup.

localrc ADMIN_PASSWORD=password MYSQL_PASSWORD=password RABBIT_PASSWORD=password SERVICE_PASSWORD=password SERVICE_TOKEN=password

TEMPEST_VOLUME_DRIVER=foo TEMPEST_VOLUME_VENDOR="Foo Inc" TEMPEST_STORAGE_PROTOCOL=iSCSI
 * 1) These options define  expected driver capabilities

CINDER_REPO=https://review.openstack.org/openstack/cinder CINDER_BRANCH=refs/changes/83/72183/4
 * 1) These options allow you to specify a branch other than "master" be used

Q_USE_SECGROUP=False LIBVIRT_FIREWALL_DRIVER=nova.virt.firewall.NoopFirewallDriver CINDER_SECURE_DELETE=False
 * 1) Disable security groups entirely

$CINDER_CONF volume_driver=cinder.volume.drivers.foo.FooDriver

How do I run my CI to test all cinder patches with my driver not yet merged?
If using devstack-gate use the pre-test-hook to cherry-pick your driver on top of the cinder patch under review. If you've submitted your change to gerrit, you can use something like the following:

function pre_test_hook { echo "Cherry-picking latest driver from gerrit" cd $BASE/new/cinder

# fetch latest patchset from gerrit UPSTREAM_REMOTE=https://review.openstack.org/openstack/cinder

# set this to your gerrit change number CHANGE_NUM=999999

PATCHSET_BASE=refs/changes/${CHANGE_NUM:(-2)}/$CHANGE_NUM LATEST_PATCHSET=$(git ls-remote $UPSTREAM_REMOTE $PATCHSET_BASE/\* |                     sort -t/ -k 5 -n | tail -n1 | cut -d$'\t' -f2)

if [ -z "$LATEST_PATCHSET" ]; then echo "Failed to determine latest patchset of $PATCHSET_BASE from $UPSTREAM_REMOTE" exit 1 fi

echo "Latest patchset ref is $LATEST_PATCHSET"

git fetch $UPSTREAM_REMOTE $LATEST_PATCHSET && git cherry-pick FETCH_HEAD }

If you haven't submitted your change to gerrit yet, you'll have to fetch the change from your own repository (github or otherwise). For example:

function pre_test_hook { echo "Cherry-picking latest driver from private repository" cd $BASE/new/cinder

VENDOR_REMOTE=https://www.github.com/vendor/cinder-driver REFSPEC=bp/new-cinder-driver

git fetch $VENDOR_REMOTE $REFSPEC && git cherry-pick FETCH_HEAD }

Otherwise you can make the changes prior to calling stack.sh or via a custom devstack plugin.

When thirdparty CI voting will be required?
Once third party CI's become more common and stable, we'll revisit the subject. This was agreed in a discussion on October 15th 2014. In a discussion on April 8th 2015, it was agreed that open source solutions that have Infra hosted CI's could have the chance to vote.

How do I test FC drivers?
Use PCI Passthrough to give the VM access to the FC PCI Device Scripts available here: http://git.openstack.org/cgit/openstack/third-party-ci-tools/tree/provisioning_scripts/fibre_channel

How do I trigger my CI to rerun on gerrit comments?
It is required for your CI to rerun jobs if the following comment is posted in a gerrit review:

So if your CI account name is "Super Cool Storage CI", the CI should be able to be retriggered by leaving the comment:

Also, your CI should also support the traditional re-trigger keyword, which is required by Third Party Testing wiki. For example:

CI's do not need to kick off tests until Zuul's has given a +1, though it is up to the maintainer if you want to trigger immediately on all patchsets.

For the openstack-ci configuration, change the following to switch to only trigger on a Zuul's +1:

In layout.yaml: Delete the line: - event: patchset-created And then add: - event: comment-added require-approval: - verified: 1 username: zuul