Jump to: navigation, search

NovaApiValidationFramework

Revision as of 04:28, 5 July 2013 by Ken1ohmichi (talk | contribs) (Web Framework)

Nova API Validation Framework

Discussion

https://etherpad.openstack.org/NovaApiValidationFramework

Introduction

Nova is a RESTful API based HTTP service.
These APIs allow for cloud managed server operations.
For example, we can create a virtual machine instance by sending the following POST request to Nova's URI: /servers.

{
   "server" : {
       "name" : "new-server-test",
       "imageRef" : "http://servers.api.openstack.org/1234/images/52415800-8b69-11e0-9b19-734f6f006e54",
       "flavorRef" : 1
   }
}

Nova contains many RESTful APIs to facilitate its many features.
Moreover, each RESTful API can be controlled flexibly through multiple parameters.
For example, the above "create a virtual machine instance" API has 19 parameters including "name", "imageRef", etc.

The HTTP service need to validate practically every API parameter in terms of acceptable type and minimum/maximum length and range.
Unnecessary workload can be avoided by validating API parameters before executing the API operation.
Moreover, API validation is an important feature from the viewpoint of HTTP service security.

Background

Figure 1 shows the current API validation overview in Nova.

  1. Select API method corresponding to the received API request.
  2. Execute API method.
  3. Possible to execute certain operations even if a parameter is invalid.
Figure 1

Figure 1

Not all API parameters are completely validated, so many API operations are executed without API parameter validation.
Furthermore, it is difficult to handle error messages on the API caller side because error message formats are not unified.

Objectives

As Nova is an HTTP service, it should be able to validate API parameters as follows:

  • Validate every API parameter.
  • Return an error response before API operation, if API parameter is invalid.
  • Unify the error message format of the response, if the cause is the same.
    (ex) ".. is too short.", ".. is too long.", ".. is not integer."

Proposal

We would like to propose an API validation framework to implement comprehensive validation.

Validation point

In this framework, the API validations are separated from API methods, and the API validations are executed before each API method.
Figure 2 shows the API validation of this framework.

  1. Select both the API schema definition and API method corresponding to the received API request.
  2. Validate API parameters based on the API schema definition.
  3. Execute API method if the API validation is successful.
Figure 2

Figure 2

Merits

The OpenStack community releases a new Nova every six months, and the Nova APIs are increasing with each version. Nova currently has hundreds of APIs.
With multiple parameters for each API, a lot of effort is required to review and implement the API schema definition.
However, we believe this framework is worth implementing because of the following merits.

  • Clarification of the API parameter definitions.
    The type and acceptable range and length of each parameter are clarified in the API schema definitions.
    By defining the schemas of every API, developers will be able to create application more easily using Nova APIs.
  • Cleaned up code
    The amount of Nova code can be reduced by merging validations and error response methods.
    For example, in the “create a virtual machine instance” API, the following 8 validation lines exist for the “min_count” parameter.
       try:
           min_count = int(str(min_count))
       except ValueError:
           msg = _('min_count must be an integer value')
           raise exc.HTTPBadRequest(explanation=msg)
       if min_count < 1:
           msg = _('min_count must be > 0')
           raise exc.HTTPBadRequest(explanation=msg)
By using this proposed framework, we can merge these into the following single line.
       'min_count': {'type': 'integer', 'minimum': 1},

Implementation

Using the JSON Schema library, we have created a prototype and examined this API validation framework.
The prototype validates the "create a virtual machine instance" API, as it is an often-used API and has many API parameters.
For other APIs, the intent is to implement them as needed based on OpenStack community discussions.

Web Framework

Current Nova(Grizzly) has been implemented with WSGI framework, which will be changed to the other framework (Pecan/WSME maybe) in the future(Nova v4 API?).
For working on both web frameworks, API validation is operated when calling API method by using decorator.
By this implementation, we will be able to move Pecan/WSME(or any web framework) + jsonschema smoothly in the future.

def validator(request_body_schema):
   schema_validator = APIValidator(request_body_schema)
   def add_validator(func):
       @functools.wraps(func)
       def wrapper(*args, **kwargs):
           schema_validator.validate(kwargs['body'])
           return func(*args, **kwargs)
       return wrapper
   return add_validator

API parameter type

The default types in the JSON Schema are basic types such as integer, float, string, and boolean.
Nova has many APIs, and we need additional types(uuid, url, etc.) for rigorous validation.
Current expected types are the following:

  • integer
    minimum, maximum
  • float
    minimum, maximum
  • string
    minLength, maxLength
  • uuid
  • url
  • ipv4 address
  • ipv6 address
  • base64 encoded data


API schema

For the creation of the prototype, each API schema was defined using JSON Schema.
Each API parameter was defined as needed using the additional types described in "3.1. API parameter type".
The following shows the API schema for "create a virtual machine instance".
Also, required API parameters were defined with "'required': True".

{
    'type' : 'object',
    'properties' : {
        'server': {
            'properties' : {
                'name': {'type': 'string', 'minLength': 1,
                         'maxLength': 255, 'required': True},
                'imageRef': {'type': 'intOrUuidOrUrl', 'required': True},
                'flavorRef': {'type': 'intOrUuidOrUrl', 'required': True},
                'min_count': {'type': 'integer', 'minimum': 1},
                'max_count': {'type': 'integer', 'minimum': 1},
                'accessIPv4': {'type': 'ipv4'},
                'accessIPv6': {'type': 'ipv6'},
[..]
            }
        }
    }
}

Migration plan

TBD

Future

I wish the following implementations after applying this framework to every Nova APIs.

  • Add the existence check of API parameter definition to Jenkins
    OpenStack utilizes a good development process through Jenkins which is used to verify the coding style and to ensure functionality, when a code change is proposed. We feel it is good to add the existence check of API parameter definition to this process. By doing so, we will be able to keep the APIs secure.
  • Apply API validation framework to other components
    By applying this framework to other components, we feel that the overall safety OpenStack APIs will be improved.
    If we can successfully apply this framework to Nova, which contains many APIs with complex API parameters, we will be able to apply it to other components as well.
    We hope to collaborate with many developers on this framework.

Appendix