Difference between revisions of "NovaApiValidationFramework"
Ken1ohmichi (talk | contribs) (→Basic validation feature) |
Ken1ohmichi (talk | contribs) (→Combination of v2.1 and v3 APIs) |
||
(32 intermediate revisions by the same user not shown) | |||
Line 36: | Line 36: | ||
In Icehouse development cycle, the web framework of Nova-v3-API would move to Pecan/WSME.<br /> | In Icehouse development cycle, the web framework of Nova-v3-API would move to Pecan/WSME.<br /> | ||
WSME has original API validation mechanism already, and we'd better to find good API validation feature if we need it.<br /> | WSME has original API validation mechanism already, and we'd better to find good API validation feature if we need it.<br /> | ||
− | To find good feature, we | + | To find good feature, we'd better to discuss it based on current Nova-v3-API implementation.<br /> |
This investigation is based on the following commit of Nova source code: | This investigation is based on the following commit of Nova source code: | ||
Line 64: | Line 64: | ||
** '''Optional:''' flavor.ephemeral, flavor.swap, .. | ** '''Optional:''' flavor.ephemeral, flavor.swap, .. | ||
− | === | + | === Auxiliary validation feature === |
− | + | Some parameters have dependency between other parameter.<br /> | |
− | + | For example, name or/and availability_zone should be specified when updating an aggregate.<br /> | |
− | + | The parameter dependencies are few cases, and the dependency validation feature would not be mandatory.<br /> | |
+ | The cases are the following: | ||
+ | * '''Required if not specifying other:''' (update aggregate: name or availability_zone), (host: status or maintenance_mode), (server: os-block-device-mapping:block_device_mapping or image_ref) | ||
+ | * '''Should not specify both:''' (interface_attachment: net_id and port_id), (server: fixed_ip and port) | ||
− | + | == Implementation == | |
− | |||
− | |||
− | + | In Icehouse summit, we have gotten a consensus API validation of v3 API will be implemented with JSONSchema. | |
+ | The implementation is going on https://review.openstack.org/#/q/project:openstack/nova+branch:master+topic:bp/v3-api-schema,n,z | ||
− | + | == Combination of v2.1 and v3 APIs == | |
− | |||
− | |||
− | |||
− | |||
− | |||
− | + | [[File:Nova-v2-v3.png]] | |
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
== Appendix: Nova-v3-API parameters == | == Appendix: Nova-v3-API parameters == | ||
Line 104: | Line 94: | ||
The source files of Nova-v3-API are put under ./nova/api/openstack/compute/plugins/v3/, and there are 79 API methods.<br /> | The source files of Nova-v3-API are put under ./nova/api/openstack/compute/plugins/v3/, and there are 79 API methods.<br /> | ||
In these APIs, 49 APIs use API parameters of a request body, and they have 148 API parameters.<br /> | In these APIs, 49 APIs use API parameters of a request body, and they have 148 API parameters.<br /> | ||
− | (48 | + | (32% API parameters(48/148) are not validated with any ways.)<br/> |
The following table shows all API parameters and each API validating implementation.<br /> | The following table shows all API parameters and each API validating implementation.<br /> | ||
Line 411: | Line 401: | ||
== Appendix: URL == | == Appendix: URL == | ||
− | * | + | * Discussion |
− | ** | + | ** https://etherpad.openstack.org/NovaApiValidationFramework |
− | + | ** http://lists.openstack.org/pipermail/openstack-dev/2013-October/016649.html | |
− | ** http://lists.openstack.org/pipermail/openstack-dev/2013- | + | * API validation coverage improvement |
− | * | + | ** https://etherpad.openstack.org/p/NovaApiValidationCoverage |
− | |||
− | |||
− | ** | ||
− |
Latest revision as of 08:40, 24 April 2014
Nova API Validation Framework
Contents
Discussion
https://etherpad.openstack.org/NovaApiValidationFramework
Introduction
Nova is a RESTful API based HTTP service.
These APIs allow for cloud managed server operations.
For example, we can create a virtual machine instance by sending the following POST request to Nova's URI: /servers.
{ "server" : { "name" : "new-server-test", "imageRef" : "http://servers.api.openstack.org/1234/images/52415800-8b69-11e0-9b19-734f6f006e54", "flavorRef" : 1 } }
Nova contains many RESTful APIs to facilitate its many features.
Moreover, each RESTful API can be controlled flexibly through multiple parameters.
For example, the above "create a virtual machine instance" API has 19 parameters including "name", "imageRef", etc.
The HTTP service need to validate practically every API parameter in terms of acceptable type and minimum/maximum length and range.
Unnecessary workload can be avoided by validating API parameters before executing the API operation.
Moreover, API validation is an important feature from the viewpoint of HTTP service security.
As Nova is an HTTP service, it should be able to validate API parameters as follows:
- Validate every API parameter.
- Return an error response before API operation, if API parameter is invalid.
- Unify the error message format of the response, if the cause is the same.
(ex) ".. is too short.", ".. is too long.", ".. is not integer."
What is necessary for Nova-v3-API
This framework target is Nova-v3-API, because it is new and the API compatibility issues would not happen.
On Havana release, Nova-v3-API is experimental and the API will become generic in Icehouse.
In Icehouse development cycle, the web framework of Nova-v3-API would move to Pecan/WSME.
WSME has original API validation mechanism already, and we'd better to find good API validation feature if we need it.
To find good feature, we'd better to discuss it based on current Nova-v3-API implementation.
This investigation is based on the following commit of Nova source code:
commit 5f0591252dc5d6e3f49682f85dcdbd99f692c07a Merge: 0211752 a19d72b Author: Jenkins <jenkins@review.openstack.org> Date: Mon Oct 7 03:14:43 2013 +0000 Merge "Fixes rescue doesn't honor enable password conf for v3"
Basic validation feature
The table of Appendix: Nova-v3-API parameters shows all parameters of Nova-v3-API.
It seems that we need some basic validation features such as:
- Type validation
- str(name, ..), int(vcpus, ..), float(rxtx_factor), dict(metadata, ..), list(networks, ..), bool(conbine, ..), None(availability_zone)
- String length validation
- 1 - 255
- Value range validation
- value >= 0(rotation, ..), value > 0(vcpus, ..), value >= 1(os-multiple-create:min_count, os-multiple-create:max_count)
- Data format validation
- Pattern: uuid(volume_id, ..), boolean(on_shared_storage, ..), base64encoded(contents), ipv4(access_ip_v4, fixed_ip), ipv6(access_ip_v6)
- Allowed list: 'active' or 'error'(state), 'parent' or 'child'(cells.type), 'MANUAL' or 'AUTO'(os-disk-config:disk_config), ...
- Allowed string: not contain '!' and '.'(cells.name), contain [a-zA-Z0-9_.- ] only(flavor.name, flavor.id)
- Mandatory validation
- Required: server.name, flavor.name, ..
- Optional: flavor.ephemeral, flavor.swap, ..
Auxiliary validation feature
Some parameters have dependency between other parameter.
For example, name or/and availability_zone should be specified when updating an aggregate.
The parameter dependencies are few cases, and the dependency validation feature would not be mandatory.
The cases are the following:
- Required if not specifying other: (update aggregate: name or availability_zone), (host: status or maintenance_mode), (server: os-block-device-mapping:block_device_mapping or image_ref)
- Should not specify both: (interface_attachment: net_id and port_id), (server: fixed_ip and port)
Implementation
In Icehouse summit, we have gotten a consensus API validation of v3 API will be implemented with JSONSchema. The implementation is going on https://review.openstack.org/#/q/project:openstack/nova+branch:master+topic:bp/v3-api-schema,n,z
Combination of v2.1 and v3 APIs
Appendix: Nova-v3-API parameters
This investigation is based on the following commit of Nova source code:
commit 5f0591252dc5d6e3f49682f85dcdbd99f692c07a Merge: 0211752 a19d72b Author: Jenkins <jenkins@review.openstack.org> Date: Mon Oct 7 03:14:43 2013 +0000 Merge "Fixes rescue doesn't honor enable password conf for v3"
The source files of Nova-v3-API are put under ./nova/api/openstack/compute/plugins/v3/, and there are 79 API methods.
In these APIs, 49 APIs use API parameters of a request body, and they have 148 API parameters.
(32% API parameters(48/148) are not validated with any ways.)
The following table shows all API parameters and each API validating implementation.
- NV: any elements are not validated
Class | API method | API parameter | Required/Optional | Type | Length/Range | Data format |
---|---|---|---|---|---|---|
AdminActionsController | _create_backup() | name | Required | NV | ||
backup_type | Required | NV | ||||
rotation | Required | int | >= 0 | |||
metadata | Optional | dict | "{key: value} 1 < len(key) < 255" | |||
AdminActionsController | _migrate_live() | block_migration | Required | NV | ||
disk_over_commit | Required | NV | ||||
host | Required | NV | ||||
AdminActionsController | _reset_state() | state | Required | str | "active"or "error" | |
AdminPasswordController | change_password() | admin_password | Required | str | ||
AgentController | create() | hypervisor | Required | NV | ||
os | Required | NV | ||||
architecture | Required | NV | ||||
version | Required | NV | ||||
url | Required | NV | ||||
md5hash | Required | NV | ||||
AgentController | update() | url | Required | NV | ||
md5hash | Required | NV | ||||
version | Required | NV | ||||
AggregateController | create() | name | Required | str | 1 - 255 | |
availability_zone | Required | None or str | ||||
AggregateController | update() | name | Required if not specifying availability_zone | str | 1 - 255 | |
availability_zone | Required if not specifying name | None or str | ||||
AggregateController | _add_host() | host | Required | str | ||
AggregateController | _remove_host() | host | Required | str | ||
AggregateController | _set_metadata() | metadata | Required | dict | ||
InterfaceAttachmentController | create() | net_id | Optional, Should not specify both net_id and port_id | NV | ||
port_id | Optional, Should not specify both net_id and port_id | NV | ||||
ip_address | Optional, Should specify both req_ip and net_id | NV | ||||
CellsController | create() | name | Required | str | not "!" and "." | |
type | Optional | str | "parent" or "child" | |||
CellsController | update() | name | Optional | str | not "!" and "." | |
type | Optional | str | "parent" or "child" | |||
CellsController | sync_instances() | project_id | Optional | NV | ||
deleted | Optional | NV | ||||
updated_since | Optional | NV | ||||
ConsoleOutputController | get_console_output() | length | Optional | int | Cast by int(str(length)) | |
CoverageController | _start_coverage() | combine | Optional | bool | boolean | |
CoverageController | _report_coverage() | file | Optional | str | filename (if path != os.path.basename(path): Error) | |
xml | Optional | bool | ||||
html | Optional | bool | ||||
ServerDiskConfigController | create() | os-disk-config:disk_config | Optional | str | "MANUAL" or "AUTO" | |
ServerDiskConfigController | update() | os-disk-config:disk_config | Optional | str | "MANUAL" or "AUTO" | |
ServerDiskConfigController | _action_rebuild() | os-disk-config:disk_config | Required | str | "MANUAL" or "AUTO" | |
ServerDiskConfigController | _action_resize() | os-disk-config:disk_config | Required | str | "MANUAL" or "AUTO" | |
EvacuateController | _evacuate() | host | Required | NV | ||
on_shared_storage | Required | str | boolean | |||
admin_password | Optional | NV | ||||
ExtendedVolumesController | swap() | old_volume_id | Required | str | uuid | |
new_volume_id | Required | str | uuid | |||
ExtendedVolumesController | attach() | volume_id | Required | str | uuid | |
device | Required | NV | ||||
ExtendedVolumesController | detach() | volume_id | Required | str | uuid | |
FlavorActionController | _add_tenant_access() | tenant_id | Required | NV | ||
FlavorActionController | _remove_tenant_access() | tenant_id | Required | NV | ||
FlavorManageController | _create() | name | Required | str | 1 - 255 | names can only contain [a-zA-Z0-9_.- ] |
id | Required | int or str | 1 - 255 | id can only contain [a-zA-Z0-9_.-] | ||
ram | Required | int | > 0 | |||
vcpus | Required | int | > 0 | |||
disk | Required | int | >= 0 | |||
ephemeral | Optional | int | >= 0 | |||
swap | Optional | int | >= 0 | |||
rxtx_factor | Optional | float | > 0 | |||
os-flavor-access:is_public | Optional | str | '1', 't', 'true', 'on', 'y', 'yes', '0', 'f', 'false', 'off', 'n', 'no' | |||
FlavorExtraSpecsController | create() | extra_specs | Optional | dict | ||
FlavorExtraSpecsController | update() | Optional | dict | |||
HostController | update() | status | Required if not specifying maintenance_mode | str | "enable" or "disable" | |
maintenance_mode | Required if not specifying status | str | "enable" or "disable" | |||
KeypairController | create() | name | Required | str? (not type checked) | 1 - 255 | |
public_key | Optional | NV | ||||
MultinicController | _add_fixed_ip() | network_id | Required | NV | ||
MultinicController | _remove_fixed_ip() | address | Required | NV | ||
QuotaClassSetsController | update() | instances | Optional | int | ||
cores | Optional | int | ||||
ram | Optional | int | ||||
security_groups | Optional | int | ||||
floating_ips | Optional | int | ||||
fixed_ips | Optional | int | ||||
metadata_items | Optional | int | ||||
injected_files | Optional | int | ||||
injected_file_content_bytes | Optional | int | ||||
injected_file_path_bytes | Optional | int | ||||
security_group_rules | Optional | int | ||||
key_pairs | Optional | int | ||||
QuotaSetsController | update() | instances | Optional | int | ||
cores | Optional | int | ||||
ram | Optional | int | ||||
security_groups | Optional | int | ||||
floating_ips | Optional | int | ||||
fixed_ips | Optional | int | ||||
metadata_items | Optional | int | ||||
injected_files | Optional | int | ||||
injected_file_content_bytes | Optional | int | ||||
injected_file_path_bytes | Optional | int | ||||
security_group_rules | Optional | int | ||||
key_pairs | Optional | int | ||||
force | Optional | str | boolean | |||
RescueController | _rescue() | admin_pass | Optional | NV | ||
SchedulerHintsController | create() | os-scheduler-hints:scheduler_hints | Optional | NV | ||
ServerMetadataController | create() | metadata | Required | dict | ||
ServerMetadataController | update() | metadata | Required | dict | len(dict) == 1 | |
ServerMetadataController | update_all() | metadata | Required | dict | ||
ServersController | create() | admin_pass | Optional | str | ||
name | Required | str | 1 - 255 | |||
image_ref | Required if not specifying os-block-device-mapping:block_device_mapping | str | uuid or ".*/<uuid>" | |||
os-block-device-mapping:block_device_mapping | Required if not specifying image_ref | dict | ||||
source_type | Required | str | 'volume', 'image', 'snapshot', 'blank' | |||
uuid | Required if source_type is not 'blank' | NV | ||||
os-availability-zone:availability_zone | Optional | NV | ||||
os-config-drive:config_drive | Optional | NV | ||||
auto_disk_config | Optional | NV | ||||
key_name | Optional | NV | ||||
os-multiple-create:min_count | Optional | int | >= 1 | |||
os-multiple-create:max_count | Optional | int | >= 1 | |||
os-multiple-create:return_reservation_id | Optional | bool | ||||
personality | Optional | dict | ||||
path | Required | NV | ||||
contents | Required | str | base64encoded | |||
scheduler_hints | Optional | NV | ||||
os-security-groups:security_groups | Optional | list | ||||
name | Optional | str | ||||
os-user-data:user_data | Optional | NV | ||||
networks | Optional | list of dict | ||||
fixed_ip | Optional, Should not specify both fixed_ip and port | ipv4 | ||||
port | Optional, Should not specify both fixed_ip and port | uuid | ||||
access_ip_v4 | Optional | ipv4 | ||||
access_ip_v6 | Optional | ipv6 | ||||
flavor_ref | Required | NV | ||||
metadata | Optional | {key: value}, 1 < len(key) < 255, len(value) < 255" | ||||
ServersController | update() | name | Optional | str | 1 - 255 | |
access_ip_v4 | Optional | ipv4 | ||||
access_ip_v6 | Optional | ipv6 | ||||
auto_disk_config | Optional | NV | ||||
ServersController | _action_reboot() | type | Required | str | "HARD" or "SOFT", not allowed lower case. | |
ServersController | _action_resize() | flavor_ref | Required | NV | validated with the existing flavors. | |
ServersController | _action_rebuild() | image_ref | Required | str | uuid or ".*/<uuid>" | |
admin_pass | Optional | NV | ||||
name | Optional | str | 1 - 255 | |||
access_ip_v4 | Optional | ipv4 | ||||
access_ip_v6 | Optional | ipv6 | ||||
auto_disk_config | Optional | NV | ||||
personality | Optional | NV | ||||
path | Required | NV | ||||
contents | Required | str | base64encoded | |||
ServersController | _action_create_image() | name | Required | NV | ||
metadata | Optional | dict | ||||
ServiceController | update() | host | Required | NV | ||
binary | Required | NV | ||||
disabled_reason | Required if id is disable-log-reason | str | 1 - 255 |
Appendix: URL
- Discussion
- API validation coverage improvement