Ceilometer/blueprints/Ceilometer-specify-event-api

Introduction
This blueprint documents the API extensions to the Ceilometer v2 API for supporting events.

The API was modeled by reviewing the StackTach, Stacky, and current Ceilometer v2 API, at
 * http://docs.openstack.org/developer/ceilometer/webapi/v2.html

and Ceilometer measurements at
 * http://docs.openstack.org/developer/ceilometer/measurements.html.

There is a table below that shows the mapping from Stack/Stacky to the extensions to the Ceilometer API. All the details aren't defined yet as I want to make sure that the direction seems like a reasonable direction first. We can use the table to track that we have everything important covered.

At a high-level, we tried to make this fit into the current Ceilometer API as much as possible.


 * 1) There have been additions to the Measurements section, which currently imply the following. For each new event type, such as "compute.instance.create.start", there is a meter associated with it, that will be a counter. For new event pairs, such as (compute.instance.create.start, compute.instance.create.end) there is a meter that would be total cumulative time. If you are interested in the total time that it took for a specific instance to be started, then you would supply a query filter parameter with the resource_id specified. I've  only specified this page for the compute.instance.create measurements, but if this seems reasonable, the rest will be straight-forward to add.
 * 2) There is a new section for querying Events. The query filter parameter can be used for filtering on resource IDs, project Ids, similar to all the existing query filters. There are a couple of additions to the query filter section to model traits in StackTach, such as the event_type and request_id. Some examples have been provided.
 * 3) There have been some changes to the Meters section, such as "GET /v2/meters/statistics/" and how the current commands can be used for querying meters to simulate stacky commands.
 * 4) There are several new return types.
 * EventType
 * Event

If you compare this document to the links above you should see how this aligns with the current Ceilometer v2 API.

Stacky to Ceilometer Mapping
This section describe the proposed mapping from StackTach/Stacky to Ceilometer. This section was derived from reviewing the stacktack/urls.py and output from the stacky cli. Note, not all commands have a mapping yet, so a few more details need to be filled in.

Measurements
This section describes the list of new meters to be added to support StackTach capabilities by extending the table defined at [Measurements](http://docs.openstack.org/developer/ceilometer/measurements.html).

Compute (Nova)

 * NOTE: We will need support for a Time Delta meter from all .start and .end event pairs. Not sure how this will work in CM?
 * If we truncate the meter name to remove the .start and .end, but keep the rest of the data (request id, etc.) the same, then it would be possible to use the normal duration calculation in /v2/meters/(meter_id)/statistics/ to get the time delta. - dhellmann
 * They need to be correlated by Request ID and ordered by time ... is that possible? - sandy
 * Yes, to grouping by Request ID. If I understand group-by, we can group using "g[0]=request_id" or something along those lines. --Roland
 * Ordering by time is possible, but I'm just wondering if we need to add an "order-by" extension as well, similar to group-by. For example, "o[0]=timestamp". --Roland
 * We need to implement the "group by" query option as well. https://blueprints.launchpad.net/ceilometer/+spec/api-group-by - dhellmann
 * Agreed. --Roland
 * NOTE: We will also need support for Time Delta from "Service" == "api" -> *.end event pairs with same Request ID. Again, not sure how CM can deal with that yet?
 * I think we just need middleware in the other services to collect the event data, although I may be missing something about this question. - dhellmann
 * yeah, it's not a middleware problem, it's a time-ordering problem. We have a unifying Request ID across the events already. -sandy
 * OK, I think that's also handled by groupby and the existing meter stuff, then? - dhellmann
 * Probably a new meter will be added for each event type.
 * Note, that it will probably be the case that all events have at least one meter associated with it, such as in the case of a counter. However, events are not meters. Events may have more than one meter associated with it, and event pairs, as in the case of start/end pairs, may introduce additional events.
 * TBD: Complete the list of events based on the notifications.
 * Is this TBD "make a list of the notifications" or "make a list of the interesting notification"? Is there any reason not to collect them all? - dhellmann
 * we will be collecting all the events, the question is are there other metrics we should infer from them? -sandy
 * Got it. - dhellmann

GET /v2/resources/
Returns the definitions of all the resources. There are no new changes required for StackTach integration.

GET /v2/resources/(resource_id)/
Returns the details for the specified resource_id. There are no new changes required for StackTach integration.

Events
This is a new api section for StackTach integration. The main goals of this section are for querying events.

GET /v2/event_types/
Returns a list of all the event types. Examples of event types are as follows:


 * compute.instance.create.start
 * compute.instance.create.end

Return type:

 * list(EventType) - List of event types. Do we really need a EventType or can we just return list(String).

To simulate the stacky command "./stacky.py events" the http request GET "/v2/event_types/" will return the list of event types.

GET /v2/events/
Returns a list of all the events based on the query filter. Examples of events are as follows:


 * compute.instance.create.start
 * compute.instance.create.end

Using this query all the events for a specific resource, user, project, request ID, host, or time range can be requested.

To simulate the Stacky command "./stacky.py uuid c0c0e50b-cdee-4fe9-9e4c-95ce6cf3fd94" the http request "GET /v2/events" with a query filter of

{       "field": "resource_id", "op": "eq", "value": "bd9431c1-8d69-4ad3-803a-8d4a6b89fd36" }

can be used.

To simulate the Stack commane "./stacky.py request req-0809cdd8-21f3-46c3-8803-4a2d9aa45436" the http request "GET /v2/events" with a query filter of

{       "field": "request_id", "op": "eq", "value": "req-0809cdd8-21f3-46c3-8803-4a2d9aa45436" }

can be used.

Parameters:

 * q (list(Query)) - Filter rules for the event names to be returned. By specifying different query filters all the queries that StackTach supports can be replicated. For example, to get all the events associated with a specific resource ID the following query filter can be defined:

{       "field": "resource_id", "op": "eq", "value": "bd9431c1-8d69-4ad3-803a-8d4a6b89fd36" }


 * NOTE: I believe we need to make one modification to the query filter. As the traits table stores data based on the type of the object, I believe it must be specified in the query, ie "type":"t_int". Any other thoughts on how to get around this? -JohnH

Return type:

 * list(Events) - List of events.

GET /v2/events/(event_id)/
Returns detailed information about the specified event where event_id is the UUID of the event.

TBD. Fill in the details of an event.

To simulate the stacky command "./stacky.py show 1562333" the http request "GET /v2/events/1562333" can be used. Although the event ID will be a UUID"

There are two types of IDs Ceilometer for an Event, the Message ID and the Event ID (PK on Event table). Which should we really use here? There's also the concept of a Message ID for a sample, so that term seems a bit overloaded in Ceilometer. As I understand it, the Message ID attribute is universal to OpenStack to track a single message around OpenStack? I believe the UUID you're referring to is Message ID, in which case, we probably ought to expose it to the Event API, so people can query for it in the event-show (GET /v2/events/) API call. Thoughts?? -Thomas

GET /v2/events/traits/
Returns a list of all traits for events. Examples of traits include the following:


 * resource_id
 * project_id
 * user_id
 * request_id
 * host
 * event_type
 * timestamp

''This list might be expensive to compute. What's the use case for that data? - dhellmann''
 * Should only be a select distinct on the Traits table. Not sure how expensive that is under mongo? -Sandy
 * Not sure if this would be a performance issue in some DBs, such as mongo. I know with the DB that I use, Vertica, this sort of query assuming the data is properly organized, is nearly instantaneous. I hope we would be able to get similar performance from mongo. As a use case, it is useful to be able to query the system for the overall list of traits, but getting the list of traits for an event type, the next one down, is probably more useful. The only other alternative would be to document every trait in the system. If that was done then we could remove this request. --Roland
 * We could organize the data in Mongo in a way to avoid having to compute it (basically, keep a set of the names of all the traits). The bigger question is why is this information useful? How is it going to be used by a consumer of the API? - dhellmann

GET /v2/events/(event_type)/traits/
Return a list of all traits given the specified event type.

GET /v2/events/(event_type)/traits/(trait)/
Returns a list of all values for the specified event type and trait. For example, to model the Stacky command "stacky hosts", which returns a list of all hosts, use "GET /v2/events/compute.instance.create.start/traits/host/"

Parameters:

 * q (list(Query)) - Filter rules for the event names to be returned.

Meters
This section models the API documentation at, http://docs.openstack.org/developer/ceilometer/webapi/v2.html#meters. There are only one new commands, but in some cases we've included the section for the documentation to show how a function in StackTach can be achieved.

GET /v2/meters/
Returns all known meters, … There are no new changes required for StackTach/Stacky.

GET /v2/meters/statistics/
Returns statistics for meters. This is a new command for addressing StackTach integration.

Parameters:

 * q (list(Query)) - Filter rules for the event names to be returned. By specifying different query filters all the queries that StackTach supports can be replicated. For example, to get all the events associated with a specific resource ID the following query filter can be defined:

{       "field": "hostname", "op": "eq", "value": "hostname.domain" }

Return type:

 * list(Statistics) - List of statistics.

To simulate the stacky command "./stacky.py summary" the http request "GET v2/meters/statistics" can be used. This can be further refined by using query filters.

''How is this different from the existing /v2/meters/(meter_id)/statistics/ query? Just that you don't have to specify the meter you want summarized? Would the grouping by meter be automatic? - dhellmann''
 * Yes, that is what this query does. Grouping by meter would be automatic. This query isn't required. Would you like to leave it out? --Roland
 * If it is indeed redundant after we get the groupby blueprint implemented, then yes I think we should leave it out. - dhellmann

GET /v2/meters/(meter_id)/
Returns the samples for the specified meter. There are no new changes required for StackTach integration.

To simulate the Stacky command "./stacky.py timings compute.instance.create" the http request "GET /v2/events" with a query filter of

{       "field": "event_type", "op": "eq", "value": "compute.instance.create" }

If no query filter is specified a list of all meters will be returned for resources for which the specfied meter_id is valid.

GET /v2/meters/(meter_id)/statistics/
Returns the statistics for the specified meter, such as min, max, avg. There are no new changes required for StackTach integration.

Parameters:

 * q (list(Query)) - Filter rules for the event names to be returned. By specifying different query filters all the queries that StackTach supports can be replicated. For example, to get all the events associated with a specific resource ID the following query filter can be defined:

{       "field": "hostname", "op": "eq", "value": "hostname.com.com" }

Return type:

 * list(Statistics) - List of statistics.

Meter
One category of measurements.


 * request_id: An optional field that specifies the request ID if valid.

Sample
A single measurement for a given meter and resource.


 * request_id: An optional field that specifies the request ID if valid.

Statistics
Computed statistics for a query.


 * request_id: An optional field that specifies the request ID if valid.

Filtering Queries
There are no changes required for StackTach integration. The queries need to support the following fieldes for StackTach integration:
 * resource_id
 * project_id
 * user_id
 * host
 * timestamp

Several additional fields to address StackTach integration are as follows:
 * request_id
 * event_type

Events
This section is new for the StackTack integration.

EventType
A decription of an event type. TBD.


 * Do we really need a EventType or should we just use a String.
 * Currently, EventType is just a String. -Sandy

Event
A description of an event. Need to specify all the details of an Event. Note, events will can be very specific. For example VM lifecycle events have Task State, Build State, ...

NOTE: We will need to add a non-Trait attribute of Source ID to Event. Currently we are using the database ID as the unique identifier, but this will need to change to be externally defined so as to identify duplicate events (possible via AMQP rescheduling) and the db should block duplicate index entries on this column.