Jump to: navigation, search

Zaqar/bp/refactor-system-tests

Rationale and Existing Problems

Currently, our tests depend on running in a particular order to guarantee test correctness. By current convention, a test with the name test_000 sets up any resources that are to be used for the rest of the test. Also, a test named test_999_... deletes any resources that are created by test_000. The setup convention breaks if a particular test runner does not lexicographically sort tests or if a test runner is run in parallel. The setup/teardown problem can be solved by using class methods setUpClass and tearDownClass. The ordering problem can robustly be solved by removing dependencies between test cases.

Furthermore, the current approach to issuing HTTP calls to the API duplicates a lot of code. Special methods exist for the construction and removal of specific headers.

A final problem with the current testing approach is one of resource clean up. None of the test cases make a guarantee on the clean up of resources in the presence of a test assertion failure. (context managers or try/finally patterns are not used). Ths introduces a subtle dependency between test suite runs. If test suite B tries to create or modify a resource that test suite A created or modified, then the state of the resource that was not rolled back or cleaned in test suite A may modify the outcome of test B. This is fine if this intentional, and can be captured as a particular test case. However, if it is accidental, it goes against test reproducibility. Dependencies across test suite boundaries are fragile.

Proposal

This blueprint proposes to address the existing problems from multiple angles

  1. Provide a Test HTTPClient
  2. Guarantee resource clean up
  3. Remove ordering dependencies

Provide a Test HTTPClient

The TestHTTPClient is a simple class. It forwards operations to python-requests, with the following added features.

  • It's constructor takes a configuration object that exposes enough details to reach Marconi server and maintain an auth token
  • It maintains an auth token in its default headers
  • It mainatains a user agent in its default headers
  • It is possible to modify headers for a given operation (get, put, ...)
  • It is possible to delete headers for a given operation by specifying its value as None
  • Otherwise, default headers are used

Guarantee Resource Clean Up

This is fairly easy to do, and has been prototyped. For each resource that we operate on the API with (queues, messages, claims), provide a decorator. The decorator is responsible for creating the resource, returning a python-requests response object containing the API response for creating this object, and must finally delete the resource once it goes out of scope.

For each test case that depends on a new resource being acquired, it must be done in a with block, including all assertions on the resource itself.

Remove Ordering Dependencies

Given the previous two enhancements, removing ordering dependencies is simple.

First, rename the tests to make it obvious that there is no order needed to them. This communicates to users that it's okay to add new tests.

Then, during setUp and tearDown, acquire resources that should not be shared between test cases. One possibility is to acquire queues using UUID (uuid.uuid1()), which is performed in a prototype implementation.

Finally, rewrite existing tests to guarantee resource cleanup. This further guarantees that test cases will not share any resources across boundaries.

Status [Under Review]

See Refactor system tests