Ceilometer/blueprints/StandardPaaSEventFormat

Introduction
Up until recently the focus of OpenStack has been on infrastructure level services (compute, storage, networking), but there are a large number of PaaS services that are currently in development and also a growing number of applications running on OpenStack systems. Many of these new applications will need to be metered for a variety of reasons: internal systems management, license management, end customer billing. Ceilometer currently supports collection of metering information for infrastructure level services. The general mechanism has been to either create a targeted agent or use the existing service APIs to meter these systems. While this approach has worked for the existing set of services, it will run into a several problems as the number of PaaS and SaaS services that need to be metered expand. The main issue is that it will not be viable to do custom integrations for the potentially hundreds (and eventually thousands) of services that may be hosted in OpenStack. This blueprint provides a proposal for standardizing the format of integrations so that Ceilometer can consume metering data from any conforming application without doing any custom integration or even a code change.

Database as a Service
Database as a Service (DBaaS), i.e. Trove, has an architecture where a service controller manages special Nova compute instances on behalf of the customer. From their perspective they are running instances of a database, not compute instances. Metering and billing will be based on hours that the database has been running, not necessarily how long the hosting instance has been run. This requires a unique set of metering records to be generated and stored to enable usage tracking and billing of the individual database instances.

DNS as a Service
DNS as a service runs on a similar service controller architecture to DBaaS. In this case, the meters that need to be measured are the existence of a DNS zone and the number of DNS queries that have been served. Once again these are application level meters that do not correlate directly to the host instances that are running. Queries could be served by a variety of hosts and a DNS zone's existence does not depend on a specific compute instance. Application level metrics must be tracked by the DNS service and reported out so that these systems can be tracked.

Load Balancing as a Service
A load balancer is a logical system that consists of some number of compute instances that host the load balancing software. Meters that must be measured would be things like the existence of a load balancer instance and the amount of data that is being transferred through the system. Once again, this does not directly correlate to the underlying infrastructure but must be reported at the application level.

Requirements
The previously listed uses cases show that PaaS and other application level metering have their own metering requirements that are similar to but distinct from the infrastructure level components such as compute. Each application or service may have its own unique meters that must be tracked and processed. Since there are a large number of potential services/applications, it will be difficult (if not impossible) to support these without some kind of standard integration model. A full integration model includes event format, field and metadata standards and delivery mechanism(s) required. This blueprint only addresses the event format. The following are the requirements for an event format to be able to generally support the service/application level metering:


 * 1) Uses a standard format that can easily be generated and consumed using commonly available libraries.
 * 2) Consistent structure of usage records to facilitate easy parsing
 * 3) Standardized field types and names, allows consumers of information to understand what each field refers to and facilitates processing of data.
 * 4) Support for reporting of the full range of meter types along with supporting metadata required to process them.

Event Format
PaaS Events and Usage Records will share the same format. The following table provides details of the fields that will be contained in the records. The format was derived from service manager events that are already generated by Infrastructure level services such as Nova and Cinder, so the overall format and any fields that overlap with existing events are essentially the same. Some additional metadata has been added to help with the processing of the events. The goal is to make the events as self-describing as possible and ensure there is no ambiguity about the data or its source.

Events
The event format listed above can be used to deliver two basic types of events: quantities and state events

State Events
These events describe the state of the metered service. They are very similar to the existing state events generated by Infrastructure. Generally there would be at least three types of events: create, exists and delete. Examples of these events for a DNS service are listed below.

dns.zone.create
This event is sent after a zone has been created

{ "event_type":"dns.zone.create", "time_stamp":"2013-04-07 22:56:30.026191", "message_id": 52232791371, "payload": { "instance_type":"type1", "availability_zone" :"az1", "instance_id" :"6accc078-81de-4567-894f-53af5653ac63", "audit_period_beginning":"2013-04-07 21:56:32.249876", "state":"active",  "audit_period_ending":"2013-04-07 22:56:32.249712", "service_id":"1abbb078-81cd-4758-974e-35fa5653ac63",        "version":"1.0",  "tenant_id":"12345", "instance_type_id": 1, "display_name":"example100.com", "message_id": 52232791371, "user_id":"6789", "state_description":"happy DNS" } }

dns.zone.exists
This event is sent every hour for existing zones

{ "event_type":"dns.zone.exists", "time_stamp":"2013-04-07 22:56:37.782573", "message_id": 52232791372, "payload": { "instance_type":"type1", "availability_zone" :"az1", "instance_id" :"6accc078-81de-4567-894f-53af5653ac63", "audit_period_beginning":"2013-04-07 21:56:37.783215", "state":"active", "audit_period_ending":"2013-04-07 22:56:37.783153", "service_id":"1abbb078-81cd-4758-974e-35fa5653ac63", "version":"1.0",  "tenant_id":"12345", "instance_type_id": 1, "display_name":"example100.com", "message_id": 52232791371, "user_id":"6789",   "state_description":"happy DNS" } }

dns.zone.delete
This event is sent when a zone is deleted

{ "event_type":"dns.zone.delete", "time_stamp":"2013-04-07 22:56:37.787774", "message_id": 52232791373, "payload": {"instance_type":"type1", "availability_zone" :"az1", "instance_id" :"6accc078-81de-4567-894f-53af5653ac63",  "audit_period_beginning":"2013-04-07 21:56:37.788177",  "state":"active",        "audit_period_ending":"2013-04-07 22:56:37.788144", "service_id":"1abbb078-81cd-4758-974e-35fa5653ac63", "version":"1.0", "tenant_id":"12345", "instance_type_id": 1, "display_name":"example100.com",    "message_id": 52232791371, "user_id":"6789", "state_description":"happy DNS" } }

Quantity Events
Quantity events have the same overall format, but additionally have a section called metrics which is an array of information about the meters that the event is reporting on. Each metric entry has a type, units, name and value. Since it is an array, multiple values can be reported in one event.

dns.zone.usage
Hourly event sent with usage for each zone instance { "event_type":"dns.zone.usage", "time_stamp":"2013-04-08 10:05:31.618074", "message_id": 52232791371, "payload": {"metrics": { "metric_type":"delta", "metric_value": 42, "metric_units":"hits", "metric_name":"queries" }, "instance_type":"type1", "availability_zone" :"az1", "instance_id" :"6accc078-81de-4567-894f-53af5653ac63", "audit_period_beginning":"2013-04-08 09:05:31.618204", "state":"active", "audit_period_ending":"2013-04-08 10:05:31.618191",  "service_id":"1abbb078-81cd-4758-974e-35fa5653ac63",       "version":"1.0", "tenant_id":"12345", "instance_type_id": 1, "display_name":"example100.com", "message_id": 52232791371, "user_id":"6789", "state_description":"happy DNS"    } }

© Copyright 2013 Hewlett-Packard Development Company, L.P.