Jump to: navigation, search

Ceilometer/blueprints/configurable-event-definitions

Introduction

Ceilometer's Events are currently fairly limited, as the traits added are pretty minimal, and the conversion is hardcoded, which makes things difficult for people deploying Ceilometer, who may have different needs for what data they want to process and store.

In order to make it easier to allow users to extract what they need, we should add a method of converting Notifications to Events that can be driven from a configuration file. This will include descriptions of how to map fields in the notification body to Traits, and optional plugins for doing any programmatic translations (splitting a string, forcing case, etc.) The mapping of notification to events will be defined per event_type, which can be wildcarded. Traits are added to events if the corresponding fields in the notification exist and are non-null. (As a special case, an empty string is considered null for non-text traits. This is due to some openstack projects (mostly Nova) using empty string for null dates.)

Current behavior is preserved: If the definitions file is not present, a warning will be logged, but an empty set of definitions will be assumed. By default, any notifications that do not have an event definition in the definitions file for them will be converted to events with a set of minimal, default traits. This can be changed by setting a flag in the ceilometer.conf file. If this is set to True, then any notifications that don't have events defined for them in the file will be dropped. This can be what you want, the notification system is quite chatty by design (notifications philosophy is "tell us everything, we'll ignore what we don't need"), so you may want to ignore the noisier ones if you don't use them.

There is a set of default traits (all are TEXT type) that will be added to all events if the notification has the relevant data:

  • message_id: (All notifications should have this) notification's uuid
  • service: (All notifications should have this) notification's publisher
  • tenant_id
  • request_id

These do not have to be specified in the event definition, they are automatically added, but their definitions can be overridden for a given event_type.

Configuration Settings

There are two configuration settings in ceilometer.conf that affect event conversions:

  • allow_dropping_of_notifications: (Default: False) If set to True, then notifications with no matching event definition will be dropped. (Notifications will only be dropped if this is True)
  • event_definitions_cfg_file: (Default: "event_definitions.yaml") Name of event definitions config file (yaml format)

Definitions file format

The event definitions file is in YAML format. It consists of a list of event definitions, which are mappings. Order is significant, the list of definitions is scanned in reverse order (last definition in the file to the first), to find a definition which matches the notification's event_type. That definition will be used to generate the Event. The reverse ordering is done because it is common to want the have a more general wildcarded definition (such as compute.instance.* ) with a set of traits common to all of those events, with a few more specific event definitions (like compute.instance.exists) afterward that have all of the above traits, plus a few more. This lets you put the general definition first, followed by the specific ones, and use YAML mapping include syntax to avoid copying all of the trait definitions.

Event Definitions

Each event definition is a mapping with two keys (both required):

  • event_type: This is a list (or a string, which will be taken as a 1 element list) of event_types this definition will handle. These can be wildcarded with unix shell glob syntax. An exclusion listing (starting with a '!') will exclude any types listed from matching. If ONLY exclusions are listed, the definition will match anything not matching the exclusions.
  • traits: This is a mapping, the keys are the trait names, and the values are trait definitions.

Trait Definitions

Each trait definition is a mapping with the following keys:

  • type (optional): The data type for this trait. (as a string). Valid options are: text, int, float, and datetime. defaults to text if not specified.
  • fields: A path specification for the field(s) in the notification you wish to extract for this trait. Specifications can be written to match multiple possible fields, the value for the trait will be derived from the matching fields that exist and have a non-null values in the notification. By default the value will be the first such field. (plugins can alter that, if they wish). This is nomally a string, but, for convenience, it can be specified as a list of specifications, which will match the fields for all of them. (See Field Path Specifications for more info on this syntax.)
  • plugin (optional): This is a mapping (For convenience, this value can also be specified as a string, which is interpreted as a the name of a plugin, to be loaded with no parameters) with the following keys:
    • name: (string) name of a plugin to load
    • parameters: (optional) Mapping of keyword arguments to pass to the plugin on initialization. (See documentation on each plugin to see what arguments it accepts.)

Field Path Specifications

The path specifications define which fields in the JSON notification body are extracted to provide the value for a given trait. The paths can be specified with a dot syntax (e.g. payload.host). Square bracket syntax (e.g. payload[host]) is also supported. In either case, if the key for the field you are looking for contains special characters, like '.', it will need to be quoted (with double or single quotes) like so: 

         payload.image_meta.'org.openstack__1__architecture'

The syntax used for the field specification is a variant of JSONPath, and is fairly flexible. (see: https://github.com/kennknowles/python-jsonpath-rw for more info)

Example Definitions file

---
- event_type: compute.instance.*
  traits: &instance_traits
    user_id:
      fields: payload.user_id
    instance_id:
      fields: payload.instance_id
    host:
      fields: publisher_id
      plugin:
        name: split
        parameters:
          segment: 1
          max_split: 1
    service_name:
      fields: publisher_id
      plugin: split
    instance_type_id:
      type: int
      fields: payload.instance_type_id
    os_architecture:
      fields: payload.image_meta.'org.openstack__1__architecture'
    launched_at:
      type: datetime
      fields: payload.launched_at
    deleted_at:
      type: datetime
      fields: payload.deleted_at
- event_type: 
    - compute.instance.exists
    - compute.instance.update
  traits:
    <<: *instance_traits
    audit_period_beginning:
      type: datetime
      fields: payload.audit_period_beginning
    audit_period_ending:
      type: datetime
      fields: payload.audit_period_ending

Trait plugins

Trait plugins can be used to do simple programmatic conversions on the value in a notification field, like splitting a string, lowercasing a value, converting a screwball date into ISO format, or the like. They are initialized with the parameters from the trait definition, if any, which can customize their behavior for a given trait. They are called with a list of all matching fields from the notification, so they can derive a value from multiple fields. The plugin will be called even if there is no fields found matching the field path(s), this lets a plugin set a default value, if needed. A plugin can also reject a value by returning None, which will cause the trait not to be added. If the plugin returns anything other than None, the trait's value will be set from whatever the plugin returned (coerced to the appropriate type for the trait).

Retrieved from "https://wiki.openstack.org/w/index.php?title=Ceilometer/blueprints/configurable-event-definitions&oldid=28631"