Blueprint-speed-up-tempest


 * Launchpad Entry: TempestSpec:speed-up-tempest
 * Created: 18 May 2013
 * Contributors:

Summary
Use a horizontally and vertically scalable and extensible test runner.

Rationale
The gate time has impact to the development speed. Parallel test case execution on multiple machines can dramatically speed up Unit Test execution.

User stories

 * 'Gate is tooo slow'
 * 'I would like use the same test runner with all OpenStack component and handling the test runs in similar way'
 * http://refstack.org/

Assumptions

 * Happier developers.
 * Faster user level API validation

Design

 * Replacing the notesttests test runner to testtools/testresources/testrepository combination.
 * Mitigate the common fixture usage
 * Maintain test capability without admin rights

Implementation
The tempest code uses waits for delete to mitigate the chance of running out from resources, this type of issue will happen in parallel.
 * 1) Increase the single thread risks in order to be less confuse-able with parallel issues.
 * 1) fix numbered, order dependent issues
 * 2) traceable logging ??
 * 3) add an option to old and new code path
 * 4) scale horizontally
 * 5) Add test case scheduler which supports resource reuse

UI Changes
output will change. Hopefully we will not see the logging from the skip exception from at the setup classes.

Code Changes
Remove complex skip decisions, test selection should be done by attributes based on the configuration file and skip exceptions.

Additional Notes
The test cases rarely needs to writes to the volumes. Smaller 10GB volume storage will not be enough, but the Thin provisioning can help.

Probably we will need to use a more than 64 MB memory flavor type, it might be necessary for testing heat properly.

Migration
Change the tox settings. Add post processing

Unresolved issues

 * https://bugs.launchpad.net/nova/+bug/1016633
 * admin permissions are generally strong enough to observe and manipulate the system globally
 * whitebox tests usually needs less noise