Ironic/Testing

Testing your changes locally
If you just want to test your changes locally (which you should), the Developer's Guide provides a good starting point.

Upstream CI Testing
We have several classes of tests run within OpenStack CI:
 * pep8 and python unit tests, run from within Ironic.
 * Tempest API tests
 * http://git.openstack.org/cgit/openstack/tempest/tree/tempest/api/baremetal/base.py
 * Devstack starts the Ironic service and create nodes with the "fake" driver.
 * Tempest exercises Ironic's API.
 * Other services are not involved in this test, but may be started by devstack none the less.
 * functional testing, using Nova with Ironic to provision emulated hardware, also run by Tempest.
 * http://git.openstack.org/cgit/openstack/tempest/tree/tempest/scenario/test_baremetal_basic_ops.py
 * Devstack configures Nova to use the nova.virt.ironic driver, and starts all Ironic services.
 * Devstack creates virtual machine(s) and enrolls them in Ironic
 * Tempest issues commands to Nova, and validates the resulting operations in Ironic.

Testing Hardware Drivers
Drivers are a central component of Ironic's architecture, but many of them can not be tested without access to specialized hardware. For instance, even the IPMIToolPowerDriver module can not be tested without having access to hardware with a BMC that supports at least IPMI v1.5, which is not present in most laptops or workstations, and is not emulatable. As such, vendor-authors of hardware drivers are expected to contribute CI resources for the drivers they have authored. A list of the status of CI testing on each supported driver is maintained at Ironic/Drivers and/or ThirdPartySystems.

What follows is a discussion of the CI testing efforts for Ironic in general, and for the third-party drivers specifically.

Third-party Driver Requirements
Third-party (aka "vendor") drivers are drivers which can not be tested upstream because they depend on specific capabilities of physical hardware. Such drivers may be allowed in Ironic if they meet the following requirements:


 * 1) each driver must adhere to the existing driver interfaces.
 * 2) drivers must function in a common environment with other drivers.
 * 3) each driver must have comprehensive unit test coverage and sufficient in-line documentation. The unit tests should mock any third-party libraries, so that they can be run without those libraries.
 * 4) vendors are responsible for fixing bugs in their driver in a timely fashion.
 * 5) vendors provide third-party, non-voting tests on supported hardware platform.
 * 6) vendors contribute (at least) one developer to upstream participation.

These guidelines are evolving and were originally proposed here: http://lists.openstack.org/pipermail/openstack-dev/2014-January/024823.html

Description
The Ironic community have agreed that if a vendor wishes to submit a driver that said vendor should also be required to set up a third party CI system in their lab which runs appropriate tests against their actual hardware for Ironic patches, and provides feedback into Gerrit.

Third Party CI Requirements

 * See the official Third Party Testing wiki.
 * Test all drivers your company has integrated in Ironic.
 * Post all test results. Even if you have failure results, those need to be posted.

For example, if your company has two drivers in Ironic, you would need to have a CI that tests against the two drivers and reports the results for each driver, for Ironic upstream patches.

Existing CI Solutions

 * Puppet OpenStack CI
 * Repoː https://git.openstack.org/cgit/openstack-infra/puppet-openstackci/tree/
 * Instructionsː https://git.openstack.org/cgit/openstack-infra/puppet-openstackci/tree/contrib/README.md

Deadlines

 * Mitaka-2ː Driver teams will have registered their intent to run CI by creating system accounts and identifying a point of contact for their CI team
 * Mitaka Feature Freezeː All driver systems show the ability to receive events and post comments in the sandbox.
 * N release feature freezeː Per patch testing and posting comments.

Questions

 * Join Third Party Meeting
 * Join #openstack-third-party-ci and ask your question

What tests do I use?
You should have a Jenkins job like {pipeline}-tempest-dsvm-ironic-pxe_ipa{job-suffix}

How do I trigger my CI to rerun on gerrit comments?
It is required for your CI to rerun jobs if the following comment is posted in a gerrit review:

So if your CI account name is "vendor", the CI should be able to be re-triggered by leaving the comment:

CIs today with the scheme we just described would look like: cisco-ironic-recheck, ufcg-oneview-recheck, ibm-powerkvm-recheck, hp-proliant-recheck, dell-ironic-recheck, fujitsu-irmc-recheck etc..

Also, your CI should also support the traditional re-trigger keyword, which is required by Third Party Testing wiki. For example:

In the layout.yaml file, the regex to parse this for example should look like this trigger: gerrit: - event: patchset-created - event: change-restored - event: comment-added comment: (?i)^(Patch Set [0-9]+:)?( [\w\\+-]*)*(\n\n)?\s*(recheck|oneview-recheck)\s*$

CI's do not need to kick off tests until Jenkin's has given a +1, though it is up to the maintainer if you want to trigger immediately on all patch-sets.

For the openstack-ci configuration, change the following to switch to only trigger on a Jenkin's +1.

In layout.yaml, delete the line - event: patchset-created And then add - event: comment-added require-approval: - verified: 1 username: Jenkins

How do I run my CI to test all Ironic patches with my driver not yet merged?
If using devstack-gate use the pre-test-hook to cherry-pick your driver on top of the Ironic patch under review. If you've submitted your change to gerrit, you can use something like the following:

function pre_test_hook { echo "Cherry-picking latest driver from gerrit" cd $BASE/new/ironic

# fetch latest patchset from gerrit UPSTREAM_REMOTE=https://review.openstack.org/openstack/ironic

# set this to your gerrit change number CHANGE_NUM=999999

PATCHSET_BASE=refs/changes/${CHANGE_NUM:(-2)}/$CHANGE_NUM LATEST_PATCHSET=$(git ls-remote $UPSTREAM_REMOTE $PATCHSET_BASE/\* |                     sort -t/ -k 5 -n | tail -n1 | cut -d$'\t' -f2)

if [ -z "$LATEST_PATCHSET" ]; then echo "Failed to determine latest patchset of $PATCHSET_BASE from $UPSTREAM_REMOTE" exit 1 fi

echo "Latest patchset ref is $LATEST_PATCHSET"

git fetch $UPSTREAM_REMOTE $LATEST_PATCHSET && git cherry-pick FETCH_HEAD }

If you haven't submitted your change to gerrit yet, you'll have to fetch the change from your own repository (github or otherwise). For example:

function pre_test_hook { echo "Cherry-picking latest driver from private repository" cd $BASE/new/ironic

VENDOR_REMOTE=https://www.github.com/vendor/ironic-driver REFSPEC=bp/new-ironic-driver

git fetch $VENDOR_REMOTE $REFSPEC && git cherry-pick FETCH_HEAD }

Otherwise you can make the changes prior to calling stack.sh or use devstack-gate or via a custom devstack plugin.