Jump to: navigation, search

Obsolete:ProjectTestingInterface

Revision as of 17:57, 25 February 2012 by MontyTaylor (talk)

Consistent Operational Interface

OpenStack has a lot of projects. For each project, the OpenStack Jenkins needs to be able to perform a lot of tasks. If each project has a slightly different way to accomplish those tasks, it makes the management of a consistent testing infrastructure very difficult to deal with. Additionally, because of the high volume of development changes and testing, the testing infrastructure has to be able to pre-cache artifacts that are normally fetched over the internet. To that end, each project should support a consistent interface for driving tests and other necessary tasks.

End results needed

Each python project must be able to do:

  • Unit tests for python2.6
  • Unit tests for python2.7
  • pep8 checks
  • Testing Coverage Report
  • source tarball generation
  • Translations import/export and merge
  • Documentation generation

Specific commands

To drive the above tasks, the following commands should be supported in a clean tree:

  • tox -ejenkins26
  • tox -ejenkins27
  • tox -epep8
  • tox -ejenkinscover
  • tox -ejenkinsvenv python setup.py sdist
  • tox -ejenkinsvenv python setup.py extract_messages
  • tox -ejenkinsvenv python setup.py update_catalog
  • tox -ejenkinsvenv python setup.py build_sphinx

Requirements Listing

Each project should list its operations dependencies in tools/pip-requires and additional dependencies required for testing in tools/test-requires.

Additionally, there is code in openstack-common to support 'freezing' requirements into a requirements.txt file in the root directory based on what versions of dependencies were actually installed at the time of testing. If this is enabled, the requirements.txt file will only be generated if the tarball is created inside of a virtualenv, and if it's not there, it will be ignored.

Virtual Environment Management

To support sensible testing across multiple python versions, we've started adding tox config files to the projects with the hope that eventually we can remove having install_venv.py copied in to each of the projects. There are a couple of quirks that we're working to fix with upstream which should be noted, and a couple of behaviors that the tox configs want to support.

The first is for pip caching purposes, Jenkins pre-builds a bundle of all needed requirements so that PyPI does not need to be queried at test time. Jenkins injects this into the directory as .cache.bundle. At the moment, because of the way tox does dependencies, the jobs listed above call tox environments prefixed with the work jenkins. Each tox config should have non-jenkins-prefixed jobs that are for devs to run locally that directly work off of tools/pip-requires, and then the jenkins-prefixed ones that use .cache.bundle. We're working on a patch to tox to allow it to notice a .cache.bundle file automatically, use it if it's there, and otherwise use the configured depends.

unittest running

Nova uses a specialized test runner, which is a subclass of nose. Most of the functionality of this has been extracted in to openstack.nose_plugin. There are still a few test failures currently when running nova unittests directly under nose, but once those are solved, the projects should really all have a config for openstack.nose_plugin and then support running nose directly with no special setup... this will help in writing code to exploit the features of nose.

Helper Scripts

The projects up until now have all had a run_tests.sh and a with_venv.sh script. run_tests.sh should be able to be easily re-written to pass things along to the above tox commands. with_venv.sh is also easy - the tox venv environment (tox -evenv) is available to run arbitrary commands in the context of a tox virtualenv.

Translations

To support translations processing, projects should have a valid babel config. There should be a locale package inside of the top project module, and in that dir should be the $project.pot file. For instance, for nova, there should be nova/locale/nova.pot. Babel commands should be configured out output their .mo files in to $project/locale as well.

Documentation

To support documentation generation, projects should have sphinx documentation source in doc/source and build_sphinx should output the documentation to doc/build

Current Status

Project tox config pip-requires split .cache.bundle job 2.6/2.7 job
nova x x x
nova stable/diablo
python-novaclient
swift
swift stable/diablo
glance x x x x
glance stable/diablo
keystone
keystone stable/diablo
python-keystoneclient
horizon
quantum
python-quantumclient
melange
python-melangeclient