QuantumTesting
Quantum has (or ehem, will have) three types of tests: unit tests, functional tests, and system tests.
As part of defining our testing strategy, we want to propose definitions for each type of testing, then start documenting what testing exists, what we plan to create, and when/how we run each type of test.
Definitions
These definitions are here for discussion, not written in stone. The goal is just that we all mean the same thing when we use a term.
- Unit Tests :
- white-box testing
- aim is to provide code coverage as close as possible to 100%
- tries to test at the smallest possible granularity
- usually does not require running system, instead, loads code modules and tests them directly.
- Functional Tests:
- black-box testing
- tests a single "user operation" at a time.
- driven by use cases.
- considers both valid and invalid inputs, focusing on correctness according to a spec.
- will likely require running the system, or at least components of the system.
- System Tests:
- Tests properties of the working system as a whole, including scale, performance, reliability.
- Certainly requires running the system.
Unit Tests
API unit tests have been added in quantum/tests/unit .
Similar to Nova, you can run unit tests with:
./run_tests.sh
There are also vestigial unit tests in the OVS plugin that test the sqlalchemy data model (originally part of the OVS plugin). We need to figure out how to extract the parts that are generic (I have some thoughts on this). The current status of each of this unit tests is covered on the QuantumUnitTestStatus page.
We also need to figure out how to make the unit testing framework automatically run unit tests that are in a particular plugin's directory.
Functional Tests
These tests will be written as code that uses the API client lib.
First step is to outline "user operations" that we want to test.
Question: should these tests actually verify that the plugin correctly forwards packets?
System Tests
TBD
Continuous Integration
OpenStack Jenkins is configured and running Quantum unit tests:
