Revision as of 05:58, 26 February 2014

Testing OpenStack Projects

Projects generally have three categories of tests that commits are must pass before being ready for review:

• Unit Tests

Self contained in each project

• Style Checks

Based on flake8 and hacking

• Integration Tests

Part of tempest

Unit Tests

See also oslo-incubator/TESTING.rst.

There are two ways to run the unit tests for a project: using tox, or using the project's run_tests.sh script. Using tox has the advantage of being similar to what the gate does.

There are two popular contexts in which to run unit tests: in a standalone project directory produced by "git clone", or in a project directory created by DevStack. The latter has the advantage of being conveniently located for integration testing as well as unit testing, but has the disadvantage that DevStack does not promise to not smash your work; fortunately you can work in a standalone project directory and have DevStack use that code instead of cloning from the main repository for that project.

Unit Testing with Tox

It is suggested you install tox [1] with pip [2].

[apt-get | yum] install python-pip

if necessary and then

pip install tox

At the moment there is a bug in the latest tox; see https://bugs.launchpad.net/openstack-ci/+bug/1274135 for a workaround; when installing from scratch, install a suitable downlevel version:

pip install 'tox<1.7'

Sample tox.ini file https://github.com/openstack/nova/blob/master/tox.ini

There will be a tox.ini file in the root directory of every project.

Within the tox.ini file you might see any of the following envlist options:

[tox]	
envlist = py26,py27,py33,pep8,pylint

In order to run tox against one of these environments, you run, for example:

tox -e py27

Most projects configure tox to use testr to run the tests; see Testr. In such cases you have the option to use testr to run the tests without a virtualenv, if your main environment is suitable.

Tox Run Time

Why does tox take so long to run? The reason tox takes a long time is two-fold: On the first run it has to create a virtual environment, which can take anywhere from 5 to 30+ minutes depending on the project and the system. The other reason is that it just takes a long time to run all of the test cases in some of the projects.

run_tests.sh

There is an obsolescent convention, as follows. Most projects have a shell script, named "run_tests.sh", that runs the unit tests of that project. To invoke it, simply "cd" to the project's directory and invoke the script. For example, to run the Nova unit tests in a default DevStack installation,

cd /opt/stack/nova
./run_tests.sh

Style Checks

Style checks can be run via tox with `tox -epep8` or manually using `flake8`.

IntegrationTests

Manual Integration Tests

DevStack produces a running system, with all the processes running under `screen`.

screen -r

will re-attach to that screen session (if it is the only one). For manual system testing: change your code in place and recycle the relevant processes, in the right order. You have to understand the code in order to know which are the relevant processes and what is the right order.

Automated Integration Tests

See tempest.

Running Swift Tests

Swift has functional tests, functional nose tests, and unit tests available in the source in the test directory. At Rackspace we test several environments including a staging area with 36 storage nodes.