Jump to: navigation, search

Difference between revisions of "Testing"

(Failed Dependencies)
(Unit Tests: Noted primacy of project's documentation, preference for tox, and gotcha wrt system packages)
Line 12: Line 12:
  
 
=== Unit Tests ===
 
=== Unit Tests ===
 +
 +
'''''FIRST AND FOREMOST''''', see the documentation for the particular project on which you are working.  Start at its HACKING.rst file, and search on if you do not find sufficient answers there.
 +
 +
Following are some general remarks that may be of help if the project's documentation is not adequate.
  
 
See also [https://github.com/openstack/oslo-incubator/blob/master/TESTING.rst oslo-incubator/TESTING.rst].
 
See also [https://github.com/openstack/oslo-incubator/blob/master/TESTING.rst oslo-incubator/TESTING.rst].
  
There are two ways to run the unit tests for a project: using tox, or using the project's run_tests.sh script.  Using tox has the advantage of being similar to what the gate does.
+
There are two ways to run the unit tests for a project: using tox, or using the project's run_tests.sh script.  Using tox is the newer convention and is generally preferred; it also has the advantage of being similar to what the gate does.
  
 
There are two popular contexts in which to run unit tests: in a standalone project directory produced by "git clone", or in a project directory created by DevStack.  The latter has the advantage of being conveniently located for integration testing as well as unit testing, but has the disadvantage that DevStack does not promise to not smash your work; fortunately you can work in a standalone project directory and have DevStack use that code instead of cloning from the main repository for that project.
 
There are two popular contexts in which to run unit tests: in a standalone project directory produced by "git clone", or in a project directory created by DevStack.  The latter has the advantage of being conveniently located for integration testing as well as unit testing, but has the disadvantage that DevStack does not promise to not smash your work; fortunately you can work in a standalone project directory and have DevStack use that code instead of cloning from the main repository for that project.
 +
 +
Some projects have dependencies on system packages --- that is, packages that '''CAN NOT''' be installed by pip.  In this case the only general way to get those dependencies satisfied is to do an install with DevStack; the project's documentation may note other ways.
  
 
==== Unit Testing with Tox ====
 
==== Unit Testing with Tox ====

Revision as of 05:47, 18 June 2014

Testing OpenStack Projects

Projects generally have three categories of tests that commits are must pass before being ready for review:

  • Unit Tests
Self contained in each project
  • Style Checks
Based on flake8 and hacking
  • Integration Tests
Part of tempest

Unit Tests

FIRST AND FOREMOST, see the documentation for the particular project on which you are working. Start at its HACKING.rst file, and search on if you do not find sufficient answers there.

Following are some general remarks that may be of help if the project's documentation is not adequate.

See also oslo-incubator/TESTING.rst.

There are two ways to run the unit tests for a project: using tox, or using the project's run_tests.sh script. Using tox is the newer convention and is generally preferred; it also has the advantage of being similar to what the gate does.

There are two popular contexts in which to run unit tests: in a standalone project directory produced by "git clone", or in a project directory created by DevStack. The latter has the advantage of being conveniently located for integration testing as well as unit testing, but has the disadvantage that DevStack does not promise to not smash your work; fortunately you can work in a standalone project directory and have DevStack use that code instead of cloning from the main repository for that project.

Some projects have dependencies on system packages --- that is, packages that CAN NOT be installed by pip. In this case the only general way to get those dependencies satisfied is to do an install with DevStack; the project's documentation may note other ways.

Unit Testing with Tox

It is suggested you install tox [1] with pip [2].

[apt-get | yum] install python-pip

if necessary and then

pip install tox

At the moment there is a bug in the latest tox; see https://bugs.launchpad.net/openstack-ci/+bug/1274135 for a workaround; when installing from scratch, install a suitable downlevel version:

pip install 'tox<1.7'

Sample tox.ini file https://github.com/openstack/nova/blob/master/tox.ini

There will be a tox.ini file in the root directory of every project.

Within the tox.ini file you might see any of the following envlist options:

[tox]	
envlist = py26,py27,py33,pep8,pylint

In order to run tox against one of these environments, you run, for example:

tox -e py27

Most projects configure tox to use testr to run the tests; see Testr. In such cases you have the option to use testr to run the tests without a virtualenv, if your main environment is suitable.

Tox Run Time

Why does tox take so long to run? The reason tox takes a long time is two-fold: On the first run it has to create a virtual environment, which can take anywhere from 5 to 30+ minutes depending on the project and the system. The other reason is that it just takes a long time to run all of the test cases in some of the projects.

Failed Dependencies

If you encounter failed dependencies when running tox please refer to the installation instructions of the project you are testing. This can normally be found under doc/source/installation.rst in the project tree or rendered as HTML on the projects page under http://docs.openstack.org/developer/openstack-projects.html

run_tests.sh

There is an older convention, as follows. Most projects have a shell script, named "run_tests.sh", that runs the unit tests of that project. To invoke it, simply "cd" to the project's directory and invoke the script. For example, to run the Nova unit tests in a default DevStack installation,

cd /opt/stack/nova
./run_tests.sh

The "run_tests.sh" script supports several flags. You can view a list of flags by doing:

run_tests.sh -h

The "run_tests.sh" script is typically a wrapper around (a) the testr or nose testrunner and (b) the pep8 or flake8 checker. Using the relevant (a) syntax you can select a subset of the tests.

Style Checks

Style checks can be run via tox with `tox -epep8` or manually using `flake8`.

IntegrationTests

Manual Integration Tests

DevStack produces a running system, with all the processes running under `screen`. Then

screen -r

will re-attach to that screen session (if it is the only one). You will find that there is a window for each OpenStack process that DevStack created. You can recycle one of those processes by killing it (^C) and re-issuing the last command in that window (which you can recall with an up-arrow).

There are two approaches to using DevStack to do integration testing of code changes on which you are working: direct and indirect. In the direct approach you make the changes in a project directory that DevStack created with "git clone". In the indirect approach you make your changes primarily in distinct standalone project directory that you created with "git clone" and configure DevStack to get that's project's code from your directory. The latter is recommended because DevStack will sometimes discard local changes.

Direct Approach

In the direct approach your git working directory is one of those that DevStack produced by "git clone" to create a working system, usually at "/opt/stack". For example, if you are working on changes to Nova, your git working directory will be "/opt/stack/nova". You change your code in place and recycle the relevant processes, in the right order. You have to understand the code in order to know which are the relevant processes and what is the right order.

Indirect Approach

In the indirect approach your primary git working directory is NOT one created by DevStack. For example, suppose you are working on Nova to fix bug 24601 and will use DevStack to install at "/opt/stack". Your primary git working directory will be elsewhere, such as "~/code/nova". You might establish that as follows (supposing you have already done the Gerrit setup that precedes "git review -s"; see GerritWorkflow).

mkdir ~/code
cd ~/code
git clone git://git.openstack.org/openstack/nova.git
cd nova
git review -s
git checkout -b bug/24601

You would then make your changes and commit them to the local repo you just made, updating your local master branch (for example).

After that you use DevStack to create a running system. To make DevStack use your commit instead of the master branch from git.openstack.org, you make your local.conf file (being a modern gal you have stopped using localrc) override the relevant REPO and BRANCH variables. You can see the defaults for these variables in DevStack's stackrc script. For example, if your changes were committed to the master branch in /home/ubuntu/code/nova, you would put the following in your local.conf.

NOVA_REPO=/home/ubuntu/code/nova
NOVA_BRANCH=bug/24601

Once DevStack is finished, you have a running system that is using Nova as it is found in the master branch of your local working directory (code/nova).

When you need to make further changes: make them in your working directory, and commit them to that local repo. Then use git to pull them over to the directory made by DevStack:

cd /opt/stack/nova
git pull

Once that is done, recycle the relevant processes just like in the direct approach.

Automated Integration Tests

See tempest.

Running Swift Tests

Swift has functional tests, functional nose tests, and unit tests available in the source in the test directory. At Rackspace we test several environments including a staging area with 36 storage nodes.