Jump to: navigation, search

Oslo/blueprints/notification-structured

< Oslo
Revision as of 15:43, 2 October 2013 by JulienDanjou (talk | contribs)

State of Oslo notifications

Currently, Oslo provides code to send notification via the module openstack.common.notifier. Its submodule named openstack..common.notifier.api allows an application to use a Python API to send notifications. Notifications are sent by using a set of drivers included in openstack.common.notifier, such as log or rpc.

The prototype of the main notification function is openstack.common.notifier.api.notify(context, publisher_id, event_type, priority, payload).

Problems

Currently the notify() arguments include very few structured information. The argument payload can be anything, and is usually a Python dict containing a various amount of data whose fields cannot be known in advance.

Some of these fields are documented on a wiki page, but it has obviously a lot of drawbacks:

  • the list is easily outdated;
  • the list is not even complete;
  • the list doesn't help any sort of automatic testing around content events.

In Ceilometer, the main consumer of events in OpenStack, we've been dealing for a long time with all of that. Our unit test tries to mimic what is sent by the other OpenStack components by including copies of the payload we've seen at some point. As soon as the payload format is changed in an OpenStack component, our tests will still pass, but the code will not be able to cope with the events in a real deployment.

Having non structured events also implies a lot of problems in storage area. Storing unstructured events relies on storing EAV based data. While this isn't a problem in some database model (NoSQL) this is a terrible issue for others (SQL) and can be a sensible one in both cases in term of performance (e.g. indexing).

Proposed solution

The proposed solution is to build a new API call with a different prototype that would relies on Python objects rather than dict. This would solve all of the mentioned problems.

The function would something like openstack.common.notifier.notify(context, priority, event) where event would be a Python object. In such context event could be a simple object, that later could be translated to a simple dict before being sent to a driver that will publisher it over the wire.

# This is peudo code written in the wiki, I didn't run it
class Event(object):
    user_id = str
    project_id = str

  def __init__(self, **kwargs):
      fields = set(filter(lambda x: not x.startswith('_'), self.__dict__.keys()))
      if set(kwargs.keys()) != fields:
          raise Exception("Too much or missing fields given in this notification") # should also indicate what
      for attr in fields:
          setattr(self, getattr(self.__class__, attr)(kwargs[attr]))

    def as_dict(self):
       return self.__dict__ # + filter out private fields, or something like that

    @property
    def event(self):
        """Return the event type by converting the class name from CamelCase to dotted notation."""
        return re.sub('([A-Z]+)', r'.\1',self.__class__.__name__).lower().strip('.')
 
class ComputeInstanceCreateEvent(Event):
    instance_id = str
    instance_type_id = str
    display_name = str
    created_at = datetime.datetime
    launchad_at = datetime.datetime
    image_ref_url = str
    state = str
    memory_mb = int

class Notifier(object):
    def __init__(self, application, host=socket.gethostname(), driver=get_the_default_driver()):
        self.application = application
        self.host = host
        self.driver = driver

    def __call__(self, context, priority, event):
       self.driver.notify(context, priority, application, host, event.type, event.as_dict())

It's also possible for an application to introspect easily the event list and their fields by introspecting the module. This could be leveraged to automatically construct SQL schema for example, or validation code.

Using Python classes will also make sure that if any part of the code (e.g. event definition) is updated in Oslo, updating Oslo in e.g. Nova will break the code in Nova and the unit testing will show that easily enough so it'll be fixed. The same will go for consumer like Ceilometer, that will have to adapt to deal with the new events.

Specyfing type for attributes is useful to:

  1. Validate that the data given are good
  2. Build correct storage schema through possible introspection

This is heavily inspired by the approach that WSME has taken to provides automatically generated REST API.

The notifier object should get more first class parameter such as the application emitting the event. This is really needed to have good filtering possibilities