Jump to: navigation, search

NotificationSystem

Summary

Nova has a need to make notifications available to users in as close to a real-time manner as is feasible. The proposed solution is to implement a PubSubHubbub (PSH) mechanism where Nova will create notifications in standard Atom 1.0 format and notify the PSH hub, and clients subscribe to the appropriate notifications to receive updates when their are notifications waiting.

PubSubHubbub (PSH) is an open protocol developed at Google and maintained at http://code.google.com/p/pubsubhubbub/.

Nova should not necessarily have to implement a hub from scratch, since there are several reference hubs available. However, Nova should make extensions/modifications to an existing hub to support access control for content. Specifically, if a cloud is being shared among multiple tenants (projects, customers), the administrator may not want notifications for one tenant being made available to other tenants. Nova will need to provide access controls so that one tenant (identified by an "account number" string) will not be able to access other tenants.

Release Note

The PubSubHubbub reference hub provided by Nova packages an open-source hub along with access controls that permit authentication to restrict access to certain data.

Rationale

By using PubSubHubbub (PSH), Nova can publish notifications (for example, exception logs or usage data) to a set of subscribers without needed to define interfaces for each specific use case. PubSubHubbub is a well-known, open-source standard that is based upon existing IEF technologies (Atom). Moreover, by ensuring that notifications are in a standard format, users of Nova will be able to access them via many existing technologies, including most modern web browsers. Finally, because PSH supports the concept of "ping" upon content creation, these notifications can be delivered in a near real-time basis.

User stories

As a Nova administrator, I need to receive notifications in a timely manner so that I can respond to emergencies.

As a Nova administrator, I need to support multiple clients for content easily so that I do not need to use my resources to develop individual interfaces.

As a systems integrator, I want to receive timely updates of usage data and disperse them to various customers (both internal and external) for billing, decision support, and analytic purposes.

Assumptions

Design

General Requirements

  1. Services or components that produce notifications MUST make those notifications available in Atom 1.0 format.
  2. There MUST be a central configuration setting that defines zero or more hubs to be used for notifications.
  3. If there is at least one hub defined for notifications, then the service or component Atom feed MUST contain the link to the hub: <link rel="hub" … />
  4. If there is at least one hub defined for notifications, then the service or component SHOULD notify the hub when new notification(s) are available. See http://code.google.com/p/pubsubhubbub/wiki/PublisherClients for example code.
  5. The PSH hub MUST behave according to the PSH specification (with the possible exception of the access controls defined below).
  6. The <atom:content> element SHOULD contain structured data in a standard format (technically, this is a requirement on the service producing the notification; however, the notification service should validate that the <content type=""> is valid).

General Notification Message Format

All messages are comprised of the following fields: message_id, publisher_id, event_type, priority and payload. A description of each follows.

  • message_id - A UUID generated at message creation for differentiating messages.
  • publisher_id - the source worker_type.host of the message.
  • event_type - the literal type of event (ex. Instance Creation.)
  • priority - patterned after the enumeration of Python logging levels in the set (DEBUG, WARN, INFO, ERROR, CRITICAL.)
  • payload - A python dictionary of attributes. Note that the payload can contain any python data type, and varies by the entry in the event_type field.
  • timestamp - the current system time, in UTC, at the moment of message creation.

The composite message will be constructed as a dictionary of the above attributes, which will then be sent via the transport mechanism defined by the driver.

Message example

    {'message_id': str(uuid.uuid4()),
     'publisher_id': 'compute.host1',
     'timestamp': utils.utcnow(),
     'priority': 'WARN',
     'event_type': 'compute.create_instance',
     'payload': {'instance_id': 12, ... }}


Hub Development

Nova should not have to develop a hub from scratch; a number of hubs have been implemented and are available in open-source format. This page lists some available hub implementations:

http://code.google.com/p/pubsubhubbub/wiki/Hubs

Access Control

While the PSH protocol does not formally support access control (it is, of course, intended to be a publication system), some users have implemented ACLs on top of it. Nova should implement one of those systems on top of its chosen hub.

The implementor should choose one of these methods that best fits the current Nova authentication pattern(s).

Dependencies

Multi-tenant accounting - Hub access control is performed via accountID as defined in the multi-tenant spec.

Implementation

In order to separate concerns, and accommodate users who may not need or want notifications traffic, nova emits notifications through a configurable notification driver. By default, nova uses the no_op driver, which simply ignores notifications. The driver may be selected with the --notification_driver flag.

Notifications may be sent to Nova's logging output by using the log_notifier driver (--notification_driver=nova.openstack.common.notifier.log_notifier)

For the full implementation of this specification, notifications are sent to a set of AMQP queues using the rabbit_notifier driver (--notification_driver=nova.openstack.common.notifier.rabbit_notifier). An external application, such as Yagi monitors these queues and takes care of formatting the notifications in Atom format, creating feeds, and optionally, pinging a PubSubHubub Hub, and/or publishing notifications to other systems via Atom Pub. Notifications are published in json format to topic queues under the "nova" exchange, under topics named like "notifications.info" where the first part is the topic prefix (by default "notifications", but this is settable via the --notification_topic flag), and the second part is the priority (one of "debug", "info", "warn", "critical" or "error" ). Publishing to multiple queues is done to prevent a large number of low-priority notifications from delaying processing of higher-priority ones.

Nova currently emits notifications about Errors, such as instance build failures, and SystemUsageData for usage tracking.

Test/Demo Plan

This need not be added or completed until the specification is nearing beta.

Unresolved issues

This should highlight any issues that should be addressed in further specifications, and not problems with the specification itself; since any specification with problems cannot be approved.

BoF agenda and discussion

Use this section to take notes during the BoF; if you keep it in the approved spec, use it for summarising what was discussed and note any options that were rejected.