Jump to: navigation, search

Difference between revisions of "Oslo/blueprints/notification-structured"

Line 1: Line 1:
 
= State of Oslo notifications =
 
= State of Oslo notifications =
  
Currently, Oslo provides code to send notification via the module '''openstack.common.notifier'''. Its submodule named '''openstack..common.notifier.api''' allows an application to use a Python API to send notifications. Notifications are sent by using a set of drivers included in ''openstack.common.notifier'', such as ''log'' or ''rpc''.
+
Currently, Oslo provides code to send notifications via the module '''openstack.common.notifier'''. Its submodule named '''openstack..common.notifier.api''' allows an application to use a Python API to send notifications. Notifications are sent by using a set of drivers included in ''openstack.common.notifier'', such as ''log'' or ''rpc''.
  
 
The prototype of the main notification function is '''openstack.common.notifier.api.notify(context, publisher_id, event_type, priority, payload)'''.
 
The prototype of the main notification function is '''openstack.common.notifier.api.notify(context, publisher_id, event_type, priority, payload)'''.
Line 12: Line 12:
 
* the list is easily outdated;
 
* the list is easily outdated;
 
* the list is not even complete;
 
* the list is not even complete;
* the list doesn't help any sort of automatic testing around content events.
+
* the list doesn't help any sort of code generation, introspection and automatic testing around events content.
  
In Ceilometer, the main consumer of events in OpenStack, we've been dealing for a long time with all of that. Our unit test tries to mimic what is sent by the other OpenStack components by including copies of the payload we've seen at some point. As soon as the payload format is changed in an OpenStack component, our tests will still pass, but the code will not be able to cope with the events in a real deployment.
+
In Ceilometer, the main consumer of events in OpenStack, we've been dealing for a long time with all of that. Our unit test tries to mimic what is sent by the other OpenStack components by including copies of the payload we've seen at some point. As soon as the payload format is changed in an OpenStack component, our tests will still pass, but the code will not be able to cope with the events sent in a real deployment.
  
Having non structured events also implies a lot of problems in storage area. Storing unstructured events relies on storing [http://en.wikipedia.org/wiki/Entity%E2%80%93attribute%E2%80%93value_model EAV] based data. While this isn't a problem in some database model (NoSQL) this is a terrible issue for others (SQL) and can be a sensible one in both cases in term of performance (e.g. indexing).
+
Having non structured events also implies a lot of problems in storage area. Storing unstructured events relies on storing [http://en.wikipedia.org/wiki/Entity%E2%80%93attribute%E2%80%93value_model EAV] based data. While this isn't a problem in some database storage system (NoSQL) this is a terrible issue for others (SQL) and can be a sensible one in both cases in term of performance (e.g. indexing, querying…).
  
 
= Proposed solution =
 
= Proposed solution =
  
The proposed solution is to build a new API call with a different prototype that would relies on Python objects rather than dict. This would solve all of the mentioned problems.
+
The proposed solution is to build a new Python API with a different prototype that would relies on Python objects rather than dict. This would solve all of the mentioned problems. This could live side by side with the current one to allow an easy transition to the new notification mechanism.
  
The function would something like '''openstack.common.notifier.notify(context, priority, event)''' where ''event'' would be a Python object. In such context event could be a simple object, that later could be translated to a simple dict before being sent to a driver that will publisher it over the wire.
+
The first class object would be something like '''openstack.common.notifier.Notifier''' and would be able to send events. The ''event'' would be a well known structured Python object. It could later be translated to a simple dict before being sent to a driver, that will then publish it over the wire.
  
 
  # This is peudo code written in the wiki, I didn't run it
 
  # This is peudo code written in the wiki, I didn't run it
Line 63: Line 63:
 
         self.driver.notify(context, priority, application, host, event.type, event.as_dict())
 
         self.driver.notify(context, priority, application, host, event.type, event.as_dict())
  
It's also possible for an application to introspect easily the event list and their fields by introspecting the module. This could be leveraged to automatically construct SQL schema for example, or validation code.
+
With such design, it's possible for an application to introspect easily the event list and their fields by introspecting the module itself. This could be leveraged to automatically construct SQL schema for example, or validation code (unit tests). The driver would be in charge of the serialization and de-serialization of notifications.
 +
(Note that there is currently no de-serialization API as there is no high level API for consuming notification; this is something that could be solved also in this blueprint, even if not directly related).
  
 
Using Python classes will also make sure that if any part of the code (e.g. event definition) is updated in Oslo, updating Oslo in e.g. Nova will break the code in Nova and the unit testing will show that easily enough so it'll be fixed. The same will go for consumer like Ceilometer, that will have to adapt to deal with the new events.
 
Using Python classes will also make sure that if any part of the code (e.g. event definition) is updated in Oslo, updating Oslo in e.g. Nova will break the code in Nova and the unit testing will show that easily enough so it'll be fixed. The same will go for consumer like Ceilometer, that will have to adapt to deal with the new events.
Line 73: Line 74:
 
This is heavily inspired by the approach that [http://pythonhosted.org/WSME/ WSME] has taken to provides automatically generated REST API.
 
This is heavily inspired by the approach that [http://pythonhosted.org/WSME/ WSME] has taken to provides automatically generated REST API.
  
The notifier object should get more first class parameter such as the application emitting the event. This is really needed to have good filtering possibilities
+
The ''Notifier'' object gets more first class parameter such as the application emitting the event. This is really needed to have good filtering possibilities.

Revision as of 15:51, 2 October 2013

State of Oslo notifications

Currently, Oslo provides code to send notifications via the module openstack.common.notifier. Its submodule named openstack..common.notifier.api allows an application to use a Python API to send notifications. Notifications are sent by using a set of drivers included in openstack.common.notifier, such as log or rpc.

The prototype of the main notification function is openstack.common.notifier.api.notify(context, publisher_id, event_type, priority, payload).

Problems

Currently the notify() arguments include very few structured information. The argument payload can be anything, and is usually a Python dict containing a various amount of data whose fields cannot be known in advance.

Some of these fields are documented on a wiki page, but it has obviously a lot of drawbacks:

  • the list is easily outdated;
  • the list is not even complete;
  • the list doesn't help any sort of code generation, introspection and automatic testing around events content.

In Ceilometer, the main consumer of events in OpenStack, we've been dealing for a long time with all of that. Our unit test tries to mimic what is sent by the other OpenStack components by including copies of the payload we've seen at some point. As soon as the payload format is changed in an OpenStack component, our tests will still pass, but the code will not be able to cope with the events sent in a real deployment.

Having non structured events also implies a lot of problems in storage area. Storing unstructured events relies on storing EAV based data. While this isn't a problem in some database storage system (NoSQL) this is a terrible issue for others (SQL) and can be a sensible one in both cases in term of performance (e.g. indexing, querying…).

Proposed solution

The proposed solution is to build a new Python API with a different prototype that would relies on Python objects rather than dict. This would solve all of the mentioned problems. This could live side by side with the current one to allow an easy transition to the new notification mechanism.

The first class object would be something like openstack.common.notifier.Notifier and would be able to send events. The event would be a well known structured Python object. It could later be translated to a simple dict before being sent to a driver, that will then publish it over the wire.

# This is peudo code written in the wiki, I didn't run it
class Event(object):
    user_id = str
    project_id = str

  def __init__(self, **kwargs):
      fields = set(filter(lambda x: not x.startswith('_'), self.__dict__.keys()))
      if set(kwargs.keys()) != fields:
          raise Exception("Too much or missing fields given in this notification") # should also indicate what
      for attr in fields:
          setattr(self, getattr(self.__class__, attr)(kwargs[attr]))

    def as_dict(self):
       return self.__dict__ # + filter out private fields, or something like that

    @property
    def event(self):
        """Return the event type by converting the class name from CamelCase to dotted notation."""
        return re.sub('([A-Z]+)', r'.\1',self.__class__.__name__).lower().strip('.')
 
class ComputeInstanceCreateEvent(Event):
    instance_id = str
    instance_type_id = str
    display_name = str
    created_at = datetime.datetime
    launchad_at = datetime.datetime
    image_ref_url = str
    state = str
    memory_mb = int

class Notifier(object):
    def __init__(self, application, host=socket.gethostname(), driver=get_the_default_driver()):
        self.application = application
        self.host = host
        self.driver = driver

    def __call__(self, context, priority, event):
       self.driver.notify(context, priority, application, host, event.type, event.as_dict())

With such design, it's possible for an application to introspect easily the event list and their fields by introspecting the module itself. This could be leveraged to automatically construct SQL schema for example, or validation code (unit tests). The driver would be in charge of the serialization and de-serialization of notifications. (Note that there is currently no de-serialization API as there is no high level API for consuming notification; this is something that could be solved also in this blueprint, even if not directly related).

Using Python classes will also make sure that if any part of the code (e.g. event definition) is updated in Oslo, updating Oslo in e.g. Nova will break the code in Nova and the unit testing will show that easily enough so it'll be fixed. The same will go for consumer like Ceilometer, that will have to adapt to deal with the new events.

Specyfing type for attributes is useful to:

  1. Validate that the data given are good
  2. Build correct storage schema through possible introspection

This is heavily inspired by the approach that WSME has taken to provides automatically generated REST API.

The Notifier object gets more first class parameter such as the application emitting the event. This is really needed to have good filtering possibilities.