Revision as of 20:32, 26 February 2015

Tempest Test Removal Procedure

Historically tempest was the only way of doing functional testing and integration testing in OpenStack. However, moving forward we really only want tempest to be an integration test suite testing the high level interactions between projects through rest api requests. In this better defined scope there are probably existing tests that don't fit into the

Propsing a test removal

The first qa irc meeting in a given month will be primarily dedicated to going over removal requests (if there are any). This etherpad will be used for anyone to propose the removal of test:

https://etherpad.openstack.org/p/tempest-test-removals

3 prong rule for removal

In the proposal etherpad we'll be looking for answers to 3 questions

1.  The tests proposed for removal must have equiv. coverage in a different project's test suite (whether this is another gating test project, or an in tree funcitonal test suite) Preferablly the other project will have a similar source of friction to prevent breaking api changes so that we don't regress and let breaking api changes through.
2. The  test proposed for removal has a failure rate <  0.50% in the gate over the past release
3. There must not be an external user/consumer of tempest that depends on the test proposed for removal

The answers to 1 and 2 are easy to verify. For 1 just provide a link to the new test location, and for 2 you can use the infra subunit2sql db to query for the test's aggregate results with something like:

SELECT * from tests where test_id like "%test_id%"; (where $test_id is the full test_id, but truncated to the class because of setupclass or teardownclass failures)

You can access the infra mysql subunit2sql db w/ read-only permissions with:

* hostname: logstash.openstack.org
* username: query
* password: query
* db_name: subunit2sql

For example if you were trying to remove the test with the id: tempest.api.compute.admin.test_flavors_negative.FlavorsAdminNegativeTestJSON.test_get_flavor_details_for_deleted_flavor you would run the following:

1. run: "mysql -u query -p -h logstash.openstack.org subunit2sql" to connect to the subunit2sql db
2. run the query: MySQL [subunit2sql]> select * from tests where test_id like "tempest.api.compute.admin.test_flavors_negative.FlavorsAdminNegativeTestJSON%";
which will return a table of all the tests in the class (but it will also catch failures in setupClass and tearDownClass)
3. paste the output table with numbers and the mysql command you ran to generate it into the etherpad.

Eventually a cli interface will be created to make that a bit more friendly. Also a dashboard is in the works so we don't need to manually run the command.

However for the 3rd prong verification is a bit more subjective. Since the meeting time will be well known ahead of time anyone who depends on the tests will have ample time beforehand to outline any concerns on the etherpad before the meeting. To give ample time for people to respond to removal proposals the list will be locked down on the Monday of a removal meeting.

We will also discuss in the meeting each removal proposal so anyone can raise a concern during the meeting for 3. If no one takes advantage of either recourse to raise a concern and the other 2 criteria are met then there shouldn't be a reason to block the removal.