QA/Tempest-test-removal

Tempest Test Removal Procedure

Historically tempest was the only way of doing functional testing and integration testing in OpenStack. However, moving forward we really only want tempest to be an integration test suite testing the high level interactions between projects through rest api requests. In this better defined scope there are probably existing tests that don't fit into the

Propsing a test removal

The first qa irc meeting in a given month will be primarily dedicated to going over removal requests (if there are any). This etherpad will be used for anyone to propose the removal of test:

https://etherpad.openstack.org/p/tempest-test-removals

3 prong rule for removal

In the proposal etherpad we'll be looking for answers to 3 questions

1. The tests proposed for removal must have equiv. coverage in a different project's test suite (whether this is another gating test project, or an in tree funcitonal test suite) Preferablly the other project will have a similar source of friction to prevent breaking api changes so that we don't regress and let breaking api changes through. 2. The test proposed for removal has a failure rate < 0.50% in the gate over the past release 3. There must not be an external user/consumer of tempest that depends on the test proposed for removal

The answers to 1 and 2 are easy to verify. For 1 just provide a link to the new test location, and for 2 you can use the infra subunit2sql db to query for the test's aggregate results with something like:

SELECT * from tests where test_id like "%test_id%"; (where $test_id is the full test_id, but truncated to the class because of setupclass or teardownclass failures)

You can access the infra mysql subunit2sql db w/ read-only permissions with: hostname: logstash.openstack.org username: query password: query db_name: subunit2sql

Eventually a cli interface will be created to make that a bit more friendly. Also a dashboard is in the works so we don't need to manually run the command.

However for the 3rd prong verification is a bit more subjective. Since the meeting time will be well known ahead of time anyone who depends on the tests will have ample time beforehand to outline any concerns on the etherpad before the meeting. To give ample time for people to respond to removal proposals the list will be locked down on the Monday of a removal meeting.

We will also discuss in the meeting each removal proposal so anyone can raise a concern during the meeting for 3. If no one takes advantage of either recourse to raise a concern and the other 2 criteria are met then there shouldn't be a reason to block the removal.