Jump to: navigation, search

Difference between revisions of "NeutronThirdPartyTesting"

(What Is Expected of Third Party CI System for Neutron)
(Replaced content with "This document has moved: http://git.openstack.org/cgit/openstack/neutron/tree/doc/source/policies/thirdparty-ci.rst")
(One intermediate revision by one other user not shown)
Line 1: Line 1:
This page documents the requirements, expectations, and ideally the results of all third party testing done by open source and vendor plugins for the Neutron project.
This document has moved: http://git.openstack.org/cgit/openstack/neutron/tree/doc/source/policies/thirdparty-ci.rst
== What Is Expected of Third Party CI System for Neutron ==
Neutron expects Third Party CI systems to follow the [http://ci.openstack.org/third_party.html requirements set by the Infrastructure team] as well as the Neutron Third Party CI guidelines below. Please ping the PTL in #openstack-neutron or send an email to the openstack-dev ML (with subject [neutron]) with any questions. Be aware that the Infrastructure documentation as well as this wikipage are living documents and undergo changes. Track changes to the infrastructure documentation using [https://review.openstack.org/#/q/status:open+project:openstack-infra/config+branch:master+topic:third-party,n,z this url] (and please review the patches) and check this page on a regular basis for updates.
=== What Changes to Run Against ===
Ideally, your 3rd Party CI system will run against all changes pushed against Neutron. You may even want to run against certain Nova changes which affect the interfaces between Nova and Neutron. Realistically, this may be too many changes for your CI system to handle. A recommendation in this case is as follows:
* Run against all changes made to your own driver/plugin.
* Run against basic changes to the Neutron DB code.
* Run against all changes to the sub-system your driver belongs to:
** For ML2, all ML2 changes.
** For LBaaS, all LBaaS changes.
** For VPNaaS, all VPNaaS changes.
** For FWaaS, all FWaaS changes.
==== Juno Release Minimal Requirements ====
Neutron team has decided to enforce a minimal set of requirements for all plugins and drivers in order to be part of the Neutron tree.
This is just for Juno release. During the Kilo summit, a specific session will be presented to go over the next cycle requirements.
These are just minimal requirements, if your CI system already covers more tests, do not change it at all.
* Tempest tests that should be included: (Only Neutron Core APIs)
    * tempest/api/network/test_networks.py
    * tempest/api/network/test_networks_negative.py
    * tempest/api/network/test_ports.py
* Run against the following changes:
    * All changes made to your own driver/plugin
    * neutron/agents/.* (at least the agents you use)
    * neutron/openstack/common/.*
    * neutron/notifiers/.* (if your drivers report vif plugging events to nova)
    * neutron/db/.*
    * neutron/services/ (at least the services you use)
* All logs should be available
==== When Your 3rd Party CI is Blocked on an Un-Merged Bug Fix ====
Once in a while, a bug may get introduced (regression) which causes consistent failures on your CI system.
Let's say you have a fix identified, but the change set for this has not been merged upstream, and all other
change sets that do not contain the fix will fail in your CI. Similarly, let's say you are introducing a new plugin
into Neutron that contains code that enables hardware in your CI test bed, but the initial commit of plugin code
has not been merged, so CI tests on all other change sets (that don't include the hardware-enabling
code) will fail on your test bed. In either of these cases, you'd like to continue testing against the subset of
changes that contain the fix or the plugin, and "abstain" from voting on all other changes. In this case, the appropriate
way to vote into Gerrit is with a "SKIPPED" status on the changes that do not contain the fix/plugin. This will
be an indication that the change has triggered your 3rd party testing, but you are temporarily unable to test against this
change set.
=== What Tests To Run ===
The short answer here is: More is always better. The more tests you run, the better covered you are. A possibly more realistic answer is as follows:
* Network API tests ([http://git.openstack.org/cgit/openstack/tempest/tree/tempest/api/network git link]).
* Network scenario tests (The test_network_* tests [http://git.openstack.org/cgit/openstack/tempest/tree/tempest/scenario here]).
* Any tests written specifically for your setup.
* Run with the test filter: 'network'. This will include all neutron specific tests as well as any other tests that are tagged as requiring networking.  An example tempest setup for devstack-gate:
    export DEVSTACK_GATE_TEMPEST_REGEX='(?!.*\[.*\bslow\b.*\])((network)|(neutron))'
... an example setup for LBaaS:
    export DEVSTACK_GATE_TEMPEST_REGEX='(?!.*\[.*\bslow\b.*\])(alancer|SimpleReadOnlyNeutron|tempest.api.network)'
=== Third Party CI Voting ===
By default, the infra team will not allow voting by new 3rd party CI systems. The way to get your 3rd party CI system to vote is to talk with the Neutron PTL, who will let infra know the system is ready to vote. The requirements for a new system to be given voting rights are as follows:
* A new system must be up and running for a month, with a track record of voting on the sandbox system.
* A new system must correctly run and pass tests on patches for the third party driver/plugin for a month.
* A new system must have a logfile setup and retention setup similar to the below.
Once the system has been running for a month, the owner of the third party CI system can contact the Neutron PTL to have a conversation about getting voting rights upstream.
The general process to get these voting rights is outlined [http://ci.openstack.org/third_party.html#permissions-on-your-third-party-system here]. Please follow that, taking note of the guidelines Neutron also places on voting for it's CI systems.
A third party system can have it's voting rights removed as well. If the system becomes unstable (stops running, voting, or start providing inaccurate results), the Neutron PTL will make an attempt to contact the owner and copy the openstack-dev mailing list. If no response is received within 2 days, the Neutron PTL will remove voting rights for the third party CI system. If a response is received, the owner will work to correct the issue. If the issue cannot be addressed in a reasonable amount of time, the voting rights will be temporarily removed.
Keep in mind that a running third party CI setup is required for upstream plugins and drivers. If the system is not functioning, the code for which it is designed to protect becomes at risk of being removed from upstream.
== Log & Test Results Filesystem Layout ==
Third-Party CI systems '''MUST''' provide [http://ci.openstack.org/third_party.html#requirements logs and configuration data] to help developers troubleshoot test failures. A third-party CI that '''DOES NOT''' post logs should be a candidate for removal, and new CI systems '''MUST''' post logs before they can be awarded voting privileges.
Third party CI systems should follow the filesystem layout convention of the OpenStack CI system. Please store your logs as viewable in a web browser, in a directory structure. Requiring the user to download a giant tarball is not acceptable.
At the root of the results - there should be the following:
* console.html.gz - contains the output of stdout of the test run
* local.conf / localrc - contains the setup used for this run
* logs/
Logs must be a directory, which contains the following:
* Log files for each screen session that DevStack creates and launches an OpenStack component in
* Test result files
** testr_results.html.gz
** tempest.txt.gz
== Helpful Hints ==
* To re-run a failed job without spamming gerrit, use something like this on your zuul node:
  if [ -z "$1" ]; then
  echo "error: `basename $0` <gerrit-num> <gerrit-patchset>"
  exit 1
  zuul enqueue --trigger gerrit --pipeline check --project openstack/neutron --change ${1},${2}

Latest revision as of 18:19, 26 June 2015

This document has moved: http://git.openstack.org/cgit/openstack/neutron/tree/doc/source/policies/thirdparty-ci.rst