Jump to: navigation, search

NovaApiValidationFramework

Revision as of 07:46, 25 March 2013 by Ken1ohmichi (talk | contribs) (Migration plan)

Nova API Validation Framework

Introduction

Nova is a HTTP service and it has RESTful APIs.
These APIs provide server capacity in the cloud.
For example, we can create a virtual machine instance by sending the following POST request to Nova's URI: /servers.

{
   "server" : {
       "name" : "new-server-test",
       "imageRef" : "http://servers.api.openstack.org/1234/images/52415800-8b69-11e0-9b19-734f6f006e54",
       "flavorRef" : 1
   }
}

Nova has many RESTful APIs, because of many features.
Moreover, each RESTful API has multiple parameters to control each feature flexibly.
For example, the above "create a virtual machine instance" API has 19 parameters which include "name", "imageRef", etc.

HTTP service should validate generally every API parameters in terms of acceptable types, minimum and maximum length and ranges.
The service can avoid unnecessary workload by validating API parameters before API operation.
Moreover, API validation is important feature from the viewpoint of HTTP service security.

Background

Figure 1 shows current API validation overview in Nova.

Figure 1

Figure 1

Nova validates API parameters in a method corresponding to the received API request.
There are validations and operations in each API method.
It is possible to execute some operation even if some parameter is invalid, because the operation is executed before some parameter validation.
Not all API parameters are completely validated, so many API operations are executed without API parameter validations.
Furthermore, it is difficult to handle error message on the API caller side because error message formats are not unified.

Objectives

Nova is one of HTTP services, and it should validate API parameters based on the following:

  • Validate every API parameters.
  • Return an error response before API operation, if API parameter is invalid.
  • Unify an error message format of the response, if the cause is the same.
    (ex) ".. is too short.", ".. is too long.", ".. is not integer."

Proposal

For the comprehensive validation, I'd like to propose API validation framework.

Validation point

This framework separates API validations from API methods, and executes API validations before each API method.
Figure 2 shows API validation of this framework.

  1. Select both API schema definition and API method corresponding to the received API request.
  2. Validate API parameters by the API schema definition.
  3. Execute API method if the API validation is succeeded.
Figure 2

Figure 2

Merits

The OpenStack community releases a new Nova every six months, and the Nova APIs are increasing version by version.
Now Nova has hundreds of its API.
We need a lot of cooperation for both the implementation of API schema definition and its review due to many APIs, but this framework is worth implementing because of the following merits.

  • Clarify the API parameter definitions.
    The API schema definitions clarify each type and acceptable range and length.
    By schema definitions of every APIs, developers will be able to create easily an application which uses the Nova APIs.
  • Clean up codes
    We can reduce Nova codes by merging validations and error response methods.
    For example, there are the following 8 validation lines for a parameter "min_count" of "create a virtual machine instance" API.
       try:
           min_count = int(str(min_count))
       except ValueError:
           msg = _('min_count must be an integer value')
           raise exc.HTTPBadRequest(explanation=msg)
       if min_count < 1:
           msg = _('min_count must be > 0')
           raise exc.HTTPBadRequest(explanation=msg)
By this framework, we will be able to merge these lines to the following 1 line.
       'min_count': {'type': 'integer', 'minimum': 1},

Implementation

I investigated API validation framework by creating a prototype with JSON Schema library.
The prototype validates "create a virtual machine instance" API, because the API is most popular
and it has many API parameters. I will implement the other API validations after discussions at the OpenStack community.

API parameter type

The default types of JSON Schema are basic types such as integer, float, string, and boolean.
Nova has many APIs, and we need additional types(uuid, url, etc.) for strict validation.
Current expected types are the following:

  • integer
    minimum, maximum
  • float
    minimum, maximum
  • string
    minLength, maxLength
  • uuid
  • url
  • ipv4 address
  • ipv6 address
  • base64 encoded data


On JSON Schema library, the type validation is implemented by isinstance() method.
isinstance() calls the meta class's __instancecheck__() method, which is specified as argument the type class.
Therefore, we can implement additional types by defining both meta class and type class like the following:

class ipv4Meta(type):
    def __instancecheck__(self, instance):
        return utils.is_valid_ipv4(instance)

class ipv4:
    __metaclass__ = ipv4Meta

We can validate API parameters with additional types by specifying the argument "types" of a Validator instance.

types={
    'ipv4': ipv4,
    'ipv6': ipv6,
    [..]
}

validator = jsonschema.Draft3Validator(schema, types=types)
try:
    validator.validate(action_args['body'])
except jsonschema.ValidationError as ex:
    msg = _("Invalid API parameter: %s") % ex.args[0]
    return Fault(webob.exc.HTTPBadRequest(explanation=msg))

API schema

On the prototype, I defined API schema definitions as JSON Schema.
Each API parameter is defined by additional types which are described in "3.1. API parameter type".
The following shows the schema of "create a virtual machine instance" API.
Moreover, necessary API parameters are defined with "'required': True".

{
    'type' : 'object',
    'properties' : {
        'server': {
            'properties' : {
                'name': {'type': 'string', 'minLength': 1,
                         'maxLength': 255, 'required': True},
                'imageRef': {'type': 'intOrUuidOrUrl', 'required': True},
                'flavorRef': {'type': 'intOrUuidOrUrl', 'required': True},
                'min_count': {'type': 'integer', 'minimum': 1},
                'max_count': {'type': 'integer', 'minimum': 1},
                'accessIPv4': {'type': 'ipv4'},
                'accessIPv6': {'type': 'ipv6'},
[..]
            }
        }
    }
}

On this prototype, API schema is defined near each API method in the same source file, because I feel it is easy to define each API schema.
On the other hand, it may be better to define each schema in different file from the source code because we can know all API schema definitions in one file.
It is better to reconsider this when API schema definitions increase.

Migration plan

TBD

Future

I wish the following implementations after applying this framework to every Nova APIs.

  • Add the existence check of API parameter definition to Jenkins
    OpenStack has a good development process by Jenkins which is used to continuously test to verify the coding style and to ensure functionality, when the code change is proposed. We feel it is good to add the existence check of API parameter definition to this process. By this process, we will be able to keep API security.
  • Apply API validation framework to other components
    By applying this framework to other components, we can make whole OpenStack secure.
    If we can succeed to apply this framework to Nova which has many APIs and complex API parameters, we will be able to apply it to other components.
    I expect cooperation of many developers.