ListTestedApis

Summary
Now adding missing Tempest tests is going forward aggressively in a Tempest BP(https://blueprints.launchpad.net/tempest/+spec/missing-tempest-tests).

This BP promotes to find missing tests effectively by listing all Nova APIs with tested/untested marks.

Developers can start/stop API recording, list tested APIs, and find missing test by looking for untested APIs.

Usage
The main target is test developers. Developers start/stop recording and create a tested API report to find missing tests.

Components to Consider
At first, the main target is Nova. Extend other components after enough mature codes.

Overview
Implement a new middleware and apis to create Nova API list, analyze incoming APIs, and manage actions of start/report/reset recording.



New API specifications
New APIs are based on "nova coverage" APIs.

Recording tested APIs is started by API action "start" and stopped by action "report".

Action "report" also create a report that shows if each API is tested or not.

The record is reset by action "reset".

Action "report" has a mandatory request field 'file' that is a name of a report file.

POST /v2/{project_id}/os-api-coverage/action {"start": {}} POST /v2/{project_id}/os-api-coverage/action {"report": {"file": "xxx"}} POST /v2/{project_id}/os-api-coverage/action {"reset": {}}
 * Start API recording
 * Stop API recording and Output API record
 * Reset API record

Sample Report of Prototype
Tested APIs are marked "Y" at "Tested" column.

Tested API: xxx/yyy Tested No. Method URL                                             Action 000 GET   / 001 POST  /{project_id}/servers/:server_id/consoles 002 GET   /{project_id}/servers/:server_id/consoles 003 PUT   /{project_id}/servers/:server_id/consoles/:(id) 004 DELETE /{project_id}/servers/:server_id/consoles/:(id) 005 GET   /{project_id}/servers/:server_id/consoles/:(id) Y 006 POST  /{project_id}/servers 007 GET   /{project_id}/servers/detail Y 008 GET   /{project_id}/servers Y 009 PUT   /{project_id}/servers/:(id) 010 POST  /{project_id}/servers/:(id)/action               restore 011 POST  /{project_id}/servers/:(id)/action               suspend 012 POST  /{project_id}/servers/:(id)/action               migrate ...      239 GET /{project_id}/os-snapshots/:(id)

Future Work

 * Add an option to Tempest
 * Analyze request body
 * Measure performance of each API