Jump to: navigation, search

Difference between revisions of "QuantumTesting"

Line 4: Line 4:
 
As part of defining our testing strategy, we want to propose definitions for each type of testing, then start documenting what testing exists, what we plan to create, and when/how we run each type of test.   
 
As part of defining our testing strategy, we want to propose definitions for each type of testing, then start documenting what testing exists, what we plan to create, and when/how we run each type of test.   
  
== Definitions ==  
+
== Definitions ==
  
 
These definitions are here for discussion, not written in stone.  The goal is just that we all mean the same thing when we use a term.   
 
These definitions are here for discussion, not written in stone.  The goal is just that we all mean the same thing when we use a term.   
Line 42: Line 42:
  
 
  TBD
 
  TBD
 +
 +
== Continuous Integration ==
 +
 +
We're trying to figure out how we can leverage Jenkins for continuous integration. 
 +
 +
We're evaluating whether we can get access to the main [[OpenStack]] Jenkins instance, or whether we should spin up our own for the time period before we're in Incubation. 
 +
 +
Carl P. also mentioned that they may be standing up a Jenkins server that we could use.

Revision as of 20:44, 26 July 2011

Quantum has (or ehem, will have) three types of tests: unit tests, functional tests, and system tests.

As part of defining our testing strategy, we want to propose definitions for each type of testing, then start documenting what testing exists, what we plan to create, and when/how we run each type of test.

Definitions

These definitions are here for discussion, not written in stone. The goal is just that we all mean the same thing when we use a term.

  • Unit Tests :
    • white-box testing
    • aim is to provide code coverage as close as possible to 100%
    • tries to test at the smallest possible granularity
    • usually does not require running system, instead, loads code modules and tests them directly.
  • Functional Tests:
    • black-box testing
    • tests a single "user operation" at a time.
    • driven by use cases.
    • considers both valid and invalid inputs, focusing on correctness according to a spec.
    • will likely require running the system, or at least components of the system.
  • System Tests:
    • Tests properties of the working system as a whole, including scale, performance, reliability.
    • Certainly requires running the system.

Unit Tests

API unit tests have been added in quantum/tests/unit .

There are also unit tests in the OVS plugin that test the sqlalchemy data model (originally part of the OVS plugin). We need to figure out how to extract the parts that are generic (I have some thoughts on this). The current status of each of this unit tests is covered on the QuantumUnitTestStatus page.

We also need to figure out how to make the unit testing framework automatically run unit tests that are in a particular plugin's directory.

Functional Tests

These tests will be written as code that uses the API client lib.

First step is to outline "user operations" that we want to test.

Question: should these tests actually verify that the plugin correctly forwards packets?

System Tests

TBD

Continuous Integration

We're trying to figure out how we can leverage Jenkins for continuous integration.

We're evaluating whether we can get access to the main OpenStack Jenkins instance, or whether we should spin up our own for the time period before we're in Incubation.

Carl P. also mentioned that they may be standing up a Jenkins server that we could use.