Difference between revisions of "Ceilometer/blueprints/StandardPaaSEventFormat"

Revision as of 17:15, 9 September 2013

Introduction

Up until recently the focus of OpenStack has been on infrastructure level services (compute, storage, networking), but there are a large number of PaaS services that are currently in development and also a growing number of applications running on OpenStack systems. Many of these new applications will need to be metered for a variety of reasons: internal systems management, license management, end customer billing. Ceilometer currently supports collection of metering information for infrastructure level services. The general mechanism has been to either create a targeted agent or use the existing service APIs to meter these systems. While this approach has worked for the existing set of services, it will run into a several problems as the number of PaaS and SaaS services that need to be metered expand. The main issue is that it will not be viable to do custom integrations for the potentially hundreds (and eventually thousands) of services that may be hosted in OpenStack. This blueprint provides a proposal for standardizing the format of integrations so that Ceilometer can consume metering data from any conforming application without doing any custom integration or even a code change.

Example Use Cases

Database as a Service

Database as a Service (DBaaS), i.e. Trove, has an architecture where a service controller manages special Nova compute instances on behalf of the customer. From their perspective they are running instances of a database, not compute instances. Metering and billing will be based on hours that the database has been running, not necessarily how long the hosting instance has been run. This requires a unique set of metering records to be generated and stored to enable usage tracking and billing of the individual database instances.

DNS as a Service

DNS as a service runs on a similar service controller architecture to DBaaS. In this case, the meters that need to be measured are the existence of a DNS zone and the number of DNS queries that have been served. Once again these are application level meters that do not correlate directly to the host instances that are running. Queries could be served by a variety of hosts and a DNS zone's existence does not depend on a specific compute instance. Application level metrics must be tracked by the DNS service and reported out so that these systems can be tracked.

Load Balancing as a Service

A load balancer is a logical system that consists of some number of compute instances that host the load balancing software. Meters that must be measured would be things like the existence of a load balancer instance and the amount of data that is being transferred through the system. Once again, this does not directly correlate to the underlying infrastructure but must be reported at the application level.

Requirements

The previously listed uses cases show that PaaS and other application level metering have their own metering requirements that are similar to but distinct from the infrastructure level components such as compute. Each application or service may have its own unique meters that must be tracked and processed. Since there are a large number of potential services/applications, it will be difficult (if not impossible) to support these without some kind of standard integration model. A full integration model includes event format, field and metadata standards and delivery mechanism(s) required. This blueprint only addresses the event format. The following are the requirements for an event format to be able to generally support the service/application level metering:

Uses a standard format that can easily be generated and consumed using commonly available libraries.
Consistent structure of usage records to facilitate easy parsing
Standardized field types and names, allows consumers of information to understand what each field refers to and facilitates processing of data.
Support for reporting of the full range of meter types along with supporting metadata required to process them.

Event Format

PaaS Events and Usage Records will share the same format. The following table provides details of the fields that will be contained in the records. The format was derived from service manager events that are already generated by Infrastructure level services such as Nova and Cinder, so the overall format and any fields that overlap with existing events are essentially the same. Some additional metadata has been added to help with the processing of the events. The goal is to make the events as self-describing as possible and ensure there is no ambiguity about the data or its source.

Field	Sub-field	Sub-field2	Type	Description	Required	Notes
event_type			enumeration	for event type records, this describes the actual event that occurred	yes for events	depends on service, defaults to create, exists, deletetimestamp
timestamp			DateTime	timestamp for record	yes
message_id			String	unique identifier for event	yes
payload
	version		String	version of event format	yes
	audit_period_beginning		DateTime	timestamp of reporting period start	yes
	audit_period_ending		DateTime	timestamp of reporting period end	yes
	record_type		enumeration [event,quantity]	events describe some kind of state change in the service, quantity describes a usage metric value	yes
	tenant_id		String	identifies the owner of the service instance	yes
	user_id		String	identifies specific user	no
	service_id		UUID	uniquely identifies a service	yes
	instance_id		UUID	uniquely identifies an instance of the service	yes	assuming instance level reporting
	display_name		String	text description of the service	no	used if customer names instances
	instance_type_id		enumeration	used to describe variations of a service	yes	needed if variations of service have different prices or need to be broken out separately on the invoice
	instance_type		String	text description of service variations	no
	availability zone		String	where the service is deployed	no	required if service is deployed at an AZ level
	region		String	data center that the service is deployed in	no	required if service is deployed at a regional level
	state		enumeration	status of the service at the time of record generation	no	required for existence events
	state_description		String	text description of the state of service	no
	license_code		enumeration	value that describes a specific license model	no
	metrics	array
		metric_name	String	unique name for the metric that is represented in this record
		metric_type	enumeration	gauge, cumulative, delta		describes the behavior of the metric, from ceilometer
		metric_value	Float	value of metric for quantity type records	yes for quantities
		metric_units	enumeration	describes the units for the quantity	?

Events

The event format listed above can be used to deliver two basic types of events: quantities and state events

State Events

These events describe the state of the metered service. They are very similar to the existing state events generated by Infrastructure. Generally there would be at least three types of events: create, exists and delete. Examples of these events for a DNS service are listed below.

dns.zone.create

This event is sent after a zone has been created

{

"event_type":"dns.zone.create", "time_stamp":"2013-04-07 22:56:30.026191", "message_id": 52232791371,

"payload": { "instance_type":"type1", "availability_zone" :"az1", "instance_id" :"6accc078-81de-4567-894f-53af5653ac63", "audit_period_beginning":"2013-04-07 21:56:32.249876",  "state":"active",  "audit_period_ending":"2013-04-07 22:56:32.249712", "service_id":"1abbb078-81cd-4758-974e-35fa5653ac63",        "version":"1.0",  "tenant_id":"12345", "instance_type_id": 1, "display_name":"example100.com", "message_id": 52232791371, "user_id":"6789", "state_description":"happy DNS" }

}

dns.zone.exists

This event is sent every hour for existing zones

{

"event_type":"dns.zone.exists", "time_stamp":"2013-04-07 22:56:37.782573", "message_id": 52232791372,

"payload": { "instance_type":"type1", "availability_zone" :"az1", "instance_id" :"6accc078-81de-4567-894f-53af5653ac63", "audit_period_beginning":"2013-04-07 21:56:37.783215", "state":"active", "audit_period_ending":"2013-04-07 22:56:37.783153", "service_id":"1abbb078-81cd-4758-974e-35fa5653ac63",  "version":"1.0",  "tenant_id":"12345", "instance_type_id": 1, "display_name":"example100.com", "message_id": 52232791371, "user_id":"6789",   "state_description":"happy DNS" }

}

dns.zone.delete

This event is sent when a zone is deleted

{

"event_type":"dns.zone.delete", "time_stamp":"2013-04-07 22:56:37.787774", "message_id": 52232791373,

"payload": {"instance_type":"type1", "availability_zone" :"az1",  "instance_id" :"6accc078-81de-4567-894f-53af5653ac63",  "audit_period_beginning":"2013-04-07 21:56:37.788177",  "state":"active",        "audit_period_ending":"2013-04-07 22:56:37.788144", "service_id":"1abbb078-81cd-4758-974e-35fa5653ac63", "version":"1.0", "tenant_id":"12345", "instance_type_id": 1, "display_name":"example100.com",    "message_id": 52232791371, "user_id":"6789", "state_description":"happy DNS" }

}

Quantity Events

Quantity events have the same overall format, but additionally have a section called metrics which is an array of information about the meters that the event is reporting on. Each metric entry has a type, units, name and value. Since it is an array, multiple values can be reported in one event.

dns.zone.usage

Hourly event sent with usage for each zone instance

{

"event_type":"dns.zone.usage", "time_stamp":"2013-04-08 10:05:31.618074", "message_id": 52232791371,

"payload": {"metrics": { "metric_type":"delta", "metric_value": 42,  "metric_units":"hits", "metric_name":"queries" }, "instance_type":"type1", "availability_zone" :"az1", "instance_id" :"6accc078-81de-4567-894f-53af5653ac63", "audit_period_beginning":"2013-04-08 09:05:31.618204", "state":"active", "audit_period_ending":"2013-04-08 10:05:31.618191",  "service_id":"1abbb078-81cd-4758-974e-35fa5653ac63",       "version":"1.0", "tenant_id":"12345", "instance_type_id": 1, "display_name":"example100.com", "message_id": 52232791371, "user_id":"6789", "state_description":"happy DNS"    }

}