Jump to: navigation, search

QA/Tempest-test-removal

< QA
Revision as of 02:41, 4 March 2015 by Treinish (talk | contribs)

Tempest Test Removal Procedure

Historically tempest was the only way of doing functional testing and integration testing in OpenStack. However, moving forward we really only want tempest to be an integration test suite testing the high level interactions between projects through rest api requests. In this better defined scope there are probably existing tests that don't fit into the

Propsing a test removal

The first qa irc meeting in a given month will be primarily dedicated to going over removal requests (if there are any). This etherpad will be used for anyone to propose the removal of test:

https://etherpad.openstack.org/p/tempest-test-removals

3 prong rule for removal

In the proposal etherpad we'll be looking for answers to 3 questions

1.  The tests proposed for removal must have equiv. coverage in a different project's test suite (whether this is another gating test project, or an in tree funcitonal test suite) Preferablly the other project will have a similar source of friction to prevent breaking api changes so that we don't regress and let breaking api changes through.
2. The  test proposed for removal has a failure rate <  0.50% in the gate over the past release (the value and interval will likely be adjusted in the future)
3. There must not be an external user/consumer of tempest that depends on the test proposed for removal

The answers to 1 and 2 are easy to verify. For 1 just provide a link to the new test location. If you are linking to the tempest removal patch please also put a Depends-On in the commit message for the commit which moved the test into another repo.

For prong 2 you can use the infra subunit2sql db to query for the test's aggregate results with something like:

SELECT * from tests where test_id like "%test_id%"; (where $test_id is the full test_id, but truncated to the class because of setupclass or teardownclass failures)

You can access the infra mysql subunit2sql db w/ read-only permissions with:

* hostname: logstash.openstack.org
* username: query
* password: query
* db_name: subunit2sql

For example if you were trying to remove the test with the id: tempest.api.compute.admin.test_flavors_negative.FlavorsAdminNegativeTestJSON.test_get_flavor_details_for_deleted_flavor you would run the following:

1. run: "mysql -u query -p -h logstash.openstack.org subunit2sql" to connect to the subunit2sql db
2. run the query: MySQL [subunit2sql]> select * from tests where test_id like "tempest.api.compute.admin.test_flavors_negative.FlavorsAdminNegativeTestJSON%";
which will return a table of all the tests in the class (but it will also catch failures in setupClass and tearDownClass)
3. paste the output table with numbers and the mysql command you ran to generate it into the etherpad.

Eventually a cli interface will be created to make that a bit more friendly. Also a dashboard is in the works so we don't need to manually run the command.

The intent of the 2nd prong is to verify that moving the test into a project specific testing is preventing bugs (assuming the tempest tests were catching issues) from bubbling up a layer into tempest jobs. If we're seeing failure rates above a certain threshold in the gate checks that means the functional testing isn't really being effective in catching that bug (and therefore blocking it from landing) and having the testing run in tempest still has value.

However for the 3rd prong verification is a bit more subjective. The original intent of this prong was mostly for refstack/defcore and things running on the stable branches. We don't want to remove any tests if that would break our api consistency checking between releases, or something that defcore/refstack is depending on being in tempest. Since the meeting time will be well known ahead of time anyone who depends on the tests will have ample time beforehand to outline any concerns on the etherpad before the meeting. To give ample time for people to respond to removal proposals the list will be locked down on the Monday of a removal meeting.

We will also discuss in the meeting each removal proposal so anyone can raise a concern during the meeting for 3. If no one takes advantage of either recourse to raise a concern and the other 2 criteria are met then there shouldn't be a reason to block the removal.

Exceptions to this procedure

For the most part all tempest test removals have to go through this procedure there are a couple of exceptions though.

1. The class of testing has been decided to be outside the scope of tempest. 2. A revert for a patch which added a broken test, or testing which didn't actually run in the gate (basically any revert for something which shouldn't have been added)

For the first exception type the only types of testing in tree which have been declared out of scope at this point are:

* The CLI tests
* Neutron Adv. Services testing