Jump to: navigation, search

Difference between revisions of "Ceilometer/blueprints/Ceilometer-specify-event-api"

(GET /v2/meters/)
(GET /v2/events/(event_id)/)
 
(31 intermediate revisions by 4 users not shown)
Line 31: Line 31:
 
| events || stacky events || GET /v2/event_types/ || Returns a list of all the event types.  
 
| events || stacky events || GET /v2/event_types/ || Returns a list of all the event types.  
 
|-
 
|-
| hosts || stacky hosts || GET /v2/events/traits/trait/. GET /v2/events/traits/host/ || Returns a list of values for the specified trait. Returns a list of all hosts.
+
| hosts || stacky hosts || GET /v2/events/traits/(trait)/. || Returns a list of values given the specified trait.
 
|-
 
|-
 
| uuids || stacky uuid c0c0e50b-cdee-4fe9-9e4c-95ce6cf3fd94 || GET /v2/events/ || Returns a list of all events.
 
| uuids || stacky uuid c0c0e50b-cdee-4fe9-9e4c-95ce6cf3fd94 || GET /v2/events/ || Returns a list of all events.
Line 69: Line 69:
 
|-
 
|-
 
|compute.instance.create || Cumulative || ns || inst ID || notification || Total time to create instances
 
|compute.instance.create || Cumulative || ns || inst ID || notification || Total time to create instances
 +
|-
 +
| TBD: Complete this table. || || || || ||
 
|}
 
|}
  
 +
* NOTE: We will need support for a Time Delta meter from all <foo>.start and <foo>.end event pairs. Not sure how this will work in CM?
 +
** If we truncate the meter name to remove the .start and .end, but keep the rest of the data (request id, etc.) the same, then it would be possible to use the normal duration calculation in /v2/meters/(meter_id)/statistics/ to get the time delta. - dhellmann
 +
*** They need to be correlated by Request ID and ordered by time  ... is that possible? - sandy
 +
**** Yes, to grouping by Request ID. If I understand group-by, we can group using "g[0]=request_id" or something along those lines. --Roland
 +
**** Ordering by time is possible, but I'm just wondering if we need to add an "order-by" extension as well, similar to group-by. For example, "o[0]=timestamp". --Roland
 +
*** We need to implement the "group by" query option as well. https://blueprints.launchpad.net/ceilometer/+spec/api-group-by - dhellmann
 +
**** Agreed. --Roland
 +
* NOTE: We will also need support for Time Delta from "Service" == "api" -> *.end event pairs with same Request ID. Again, not sure how CM can deal with that yet?
 +
** I think we just need middleware in the other services to collect the event data, although I may be missing something about this question. - dhellmann
 +
*** yeah, it's not a middleware problem, it's a time-ordering problem. We have a unifying Request ID across the events already. -sandy
 +
*** OK, I think that's also handled by groupby and the existing meter stuff, then? - dhellmann
 
* Probably a new meter will be added for each event type.
 
* Probably a new meter will be added for each event type.
* Additionally, a new meter will be added for events that have start/end pairs.
 
 
* Note, that it will probably be the case that all events have at least one meter associated with it, such as in the case of a counter. However, events are not meters. Events may have more than one meter associated with it, and event pairs, as in the case of start/end pairs, may introduce additional events.  
 
* Note, that it will probably be the case that all events have at least one meter associated with it, such as in the case of a counter. However, events are not meters. Events may have more than one meter associated with it, and event pairs, as in the case of start/end pairs, may introduce additional events.  
* TBD: Complete the list of events based on the notificactions.
+
* TBD: Complete the list of events based on the notifications.
 +
** Is this TBD "make a list of the notifications" or "make a list of the ''interesting'' notification"? Is there any reason not to collect them all? - dhellmann
 +
*** we will be collecting all the events, the question is are there other metrics we should infer from them? -sandy
 +
*** Got it. - dhellmann
  
 
==Resources==
 
==Resources==
Line 83: Line 98:
 
===GET /v2/resources/(resource_id)/===
 
===GET /v2/resources/(resource_id)/===
 
Returns the details for the specified resource_id. There are no new changes required for StackTach integration.
 
Returns the details for the specified resource_id. There are no new changes required for StackTach integration.
 
==Hosts==
 
This is a new api section for StackTach integration.
 
  
 
==Events==
 
==Events==
Line 139: Line 151:
 
         "value": "bd9431c1-8d69-4ad3-803a-8d4a6b89fd36"
 
         "value": "bd9431c1-8d69-4ad3-803a-8d4a6b89fd36"
 
     }
 
     }
 +
 +
 +
* NOTE: I believe we need to make one modification to the query filter. As the traits table stores data based on the type of the object, I believe it must be specified in the query, ie "type":"t_int". Any other thoughts on how to get around this? -JohnH
  
 
====Return type:====
 
====Return type:====
Line 145: Line 160:
  
 
===GET /v2/events/(event_id)/===
 
===GET /v2/events/(event_id)/===
Returns detailed information about the specified event.
+
Returns detailed information about the specified event where event_id is the UUID of the event.
  
 
TBD. Fill in the details of an event.
 
TBD. Fill in the details of an event.
  
To simulate the stacky command "./stacky.py show 1562333" the http request "GET /v2/events/1562333" can be used."
+
To simulate the stacky command "./stacky.py show 1562333" the http request "GET /v2/events/1562333" can be used. Although the event ID will be a UUID"
 +
 
 +
There are two types of IDs Ceilometer for an Event, the Message ID and the Event ID (PK on Event table). Which should we really use here? There's also the concept of a Message ID for a sample, so that term seems a bit overloaded in Ceilometer. As I understand it, the Message ID attribute is universal to OpenStack to track a single message around OpenStack? I believe the UUID you're referring to is Message ID, in which case, we probably ought to expose it to the Event API, so people can query for it in the event-show (GET /v2/events/<message_id>) API call. Thoughts?? -Thomas
  
 
===GET /v2/events/traits/===
 
===GET /v2/events/traits/===
Line 162: Line 179:
 
* timestamp
 
* timestamp
  
===GET /v2/events/traits/(trait)/===
+
''This list might be expensive to compute. What's the use case for that data? - dhellmann''
Returns a list of all values for the specified trait. For example, to model the Stacky command "stacky hosts", which returns a list of all hosts, use "GET /v2/events/traits/host/"
+
*Should only be a select distinct on the Traits table. Not sure how expensive that is under mongo? -Sandy
 +
*Not sure if this would be a performance issue in some DBs, such as mongo. I know with the DB that I use, Vertica, this sort of query assuming the data is properly organized, is nearly instantaneous. I hope we would be able to get similar performance from mongo. As a use case, it is useful to be able to query the system for the overall list of traits, but getting the list of traits for an event type, the next one down, is probably more useful. The only other alternative would be to document every trait in the system. If that was done then we could remove this request. --Roland
 +
* We could organize the data in Mongo in a way to avoid having to compute it (basically, keep a set of the names of all the traits). The bigger question is why is this information useful? How is it going to be used by a consumer of the API? - dhellmann
 +
 
 +
===GET /v2/events/(event_type)/traits/===
 +
Return a list of all traits given the specified event type.
 +
 
 +
===GET /v2/events/(event_type)/traits/(trait)/===
 +
Returns a list of all values for the specified event type and trait. For example, to model the Stacky command "stacky hosts", which returns a list of all hosts, use "GET /v2/events/compute.instance.create.start/traits/host/"
  
 
====Parameters:====
 
====Parameters:====
Line 169: Line 194:
 
* q (list(Query)) - Filter rules for the event names to be returned.
 
* q (list(Query)) - Filter rules for the event names to be returned.
  
===GET /v2/events/(event_type)/traits/===
+
==Meters==
Return a list of all traits given the specified event type.
+
This section models the API documentation at, http://docs.openstack.org/developer/ceilometer/webapi/v2.html#meters. There are only one new commands, but in some cases we've included the section for the documentation to show how a function in StackTach can be achieved.
  
==Meters==
 
 
 
===GET /v2/meters/===
 
===GET /v2/meters/===
 
Returns all known meters, … There are no new changes required for StackTach/Stacky.
 
Returns all known meters, … There are no new changes required for StackTach/Stacky.
Line 195: Line 218:
  
 
To simulate the stacky command "./stacky.py summary" the http request "GET v2/meters/statistics" can be used. This can be further refined by using query filters.
 
To simulate the stacky command "./stacky.py summary" the http request "GET v2/meters/statistics" can be used. This can be further refined by using query filters.
 +
 +
''How is this different from the existing /v2/meters/(meter_id)/statistics/ query? Just that you don't have to specify the meter you want summarized? Would the grouping by meter be automatic? - dhellmann''
 +
* Yes, that is what this query does. Grouping by meter would be automatic. This query isn't required. Would you like to leave it out? --Roland
 +
* If it is indeed redundant after we get the groupby blueprint implemented, then yes I think we should leave it out. - dhellmann
  
 
===GET /v2/meters/(meter_id)/===
 
===GET /v2/meters/(meter_id)/===
Line 225: Line 252:
  
 
* list(Statistics) - List of statistics.
 
* list(Statistics) - List of statistics.
 
  
 
==Samples and Statistics==
 
==Samples and Statistics==
Line 256: Line 282:
 
* event_type
 
* event_type
  
==Hosts==
 
 
==Events==
 
==Events==
 
This section is new for the StackTack integration.
 
This section is new for the StackTack integration.
Line 264: Line 289:
  
 
* Do we really need a EventType or should we just use a String.
 
* Do we really need a EventType or should we just use a String.
 +
** Currently, EventType is just a String. -Sandy
  
 
===Event===
 
===Event===
 
A description of an event. Need to specify all the details of an Event. Note, events will can be very specific. For example VM lifecycle events have Task State, Build State, ...
 
A description of an event. Need to specify all the details of an Event. Note, events will can be very specific. For example VM lifecycle events have Task State, Build State, ...
 +
 +
NOTE: We will need to add a non-Trait attribute of Source ID to Event. Currently we are using the database ID as the unique identifier, but this will need to change to be externally defined so as to identify duplicate events (possible via AMQP rescheduling) and the db should block duplicate index entries on this column.
  
 
==Issues==
 
==Issues==

Latest revision as of 14:30, 5 November 2013

Introduction

This blueprint documents the API extensions to the Ceilometer v2 API for supporting events.

The API was modeled by reviewing the StackTach, Stacky, and current Ceilometer v2 API, at

http://docs.openstack.org/developer/ceilometer/webapi/v2.html

and Ceilometer measurements at

http://docs.openstack.org/developer/ceilometer/measurements.html.

There is a table below that shows the mapping from Stack/Stacky to the extensions to the Ceilometer API. All the details aren't defined yet as I want to make sure that the direction seems like a reasonable direction first. We can use the table to track that we have everything important covered.

At a high-level, we tried to make this fit into the current Ceilometer API as much as possible.

  1. There have been additions to the Measurements section, which currently imply the following. For each new event type, such as "compute.instance.create.start", there is a meter associated with it, that will be a counter. For new event pairs, such as (compute.instance.create.start, compute.instance.create.end) there is a meter that would be total cumulative time. If you are interested in the total time that it took for a specific instance to be started, then you would supply a query filter parameter with the resource_id specified. I've only specified this page for the compute.instance.create measurements, but if this seems reasonable, the rest will be straight-forward to add.
  2. There is a new section for querying Events. The query filter parameter can be used for filtering on resource IDs, project Ids, similar to all the existing query filters. There are a couple of additions to the query filter section to model traits in StackTach, such as the event_type and request_id. Some examples have been provided.
  3. There have been some changes to the Meters section, such as "GET /v2/meters/statistics/" and how the current commands can be used for querying meters to simulate stacky commands.
  4. There are several new return types.
  • EventType
  • Event

If you compare this document to the links above you should see how this aligns with the current Ceilometer v2 API.

Stacky to Ceilometer Mapping

This section describe the proposed mapping from StackTach/Stacky to Ceilometer. This section was derived from reviewing the stacktack/urls.py and output from the stacky cli. Note, not all commands have a mapping yet, so a few more details need to be filled in.

StackTach Stacky Ceilometer Description
deployments stacky deployments NA Lists the environment names, such as rnd-a
events stacky events GET /v2/event_types/ Returns a list of all the event types.
hosts stacky hosts GET /v2/events/traits/(trait)/. Returns a list of values given the specified trait.
uuids stacky uuid c0c0e50b-cdee-4fe9-9e4c-95ce6cf3fd94 GET /v2/events/ Returns a list of all events.
timings stacky timings compute.instance.create GET /v2/meters/(meter_id)/ Returns statistics for meters.
timings/instance_uuid Returns meters for instance ID.
summary stacky summary GET /v2/meters/statistics/ Lists the statistics, such as num, min, max, and avg for meters
requests stacky request req-0809cdd8-21f3-46c3-8803-4a2d9aa45436 GET /v2/events/ Lists the events associated with the specified resuest ID.
reports
reports/id
show/event_id stacky show 1562333 GET /v2/events/(event_id)/ Returns the details for the specified event ID.
watch/deployment_id
kpi/project_id
kpi/tenant_id

Measurements

This section describes the list of new meters to be added to support StackTach capabilities by extending the table defined at [Measurements](http://docs.openstack.org/developer/ceilometer/measurements.html).

Compute (Nova)

Name Type Unit Resource Origin Note
compute.instance.create.start Cumulative ns inst ID notification Number of create start notifications
compute.instance.create.end Cumulative ns inst ID notification Number of create end notifications
compute.instance.create Cumulative ns inst ID notification Total time to create instances
TBD: Complete this table.
  • NOTE: We will need support for a Time Delta meter from all <foo>.start and <foo>.end event pairs. Not sure how this will work in CM?
    • If we truncate the meter name to remove the .start and .end, but keep the rest of the data (request id, etc.) the same, then it would be possible to use the normal duration calculation in /v2/meters/(meter_id)/statistics/ to get the time delta. - dhellmann
      • They need to be correlated by Request ID and ordered by time ... is that possible? - sandy
        • Yes, to grouping by Request ID. If I understand group-by, we can group using "g[0]=request_id" or something along those lines. --Roland
        • Ordering by time is possible, but I'm just wondering if we need to add an "order-by" extension as well, similar to group-by. For example, "o[0]=timestamp". --Roland
      • We need to implement the "group by" query option as well. https://blueprints.launchpad.net/ceilometer/+spec/api-group-by - dhellmann
        • Agreed. --Roland
  • NOTE: We will also need support for Time Delta from "Service" == "api" -> *.end event pairs with same Request ID. Again, not sure how CM can deal with that yet?
    • I think we just need middleware in the other services to collect the event data, although I may be missing something about this question. - dhellmann
      • yeah, it's not a middleware problem, it's a time-ordering problem. We have a unifying Request ID across the events already. -sandy
      • OK, I think that's also handled by groupby and the existing meter stuff, then? - dhellmann
  • Probably a new meter will be added for each event type.
  • Note, that it will probably be the case that all events have at least one meter associated with it, such as in the case of a counter. However, events are not meters. Events may have more than one meter associated with it, and event pairs, as in the case of start/end pairs, may introduce additional events.
  • TBD: Complete the list of events based on the notifications.
    • Is this TBD "make a list of the notifications" or "make a list of the interesting notification"? Is there any reason not to collect them all? - dhellmann
      • we will be collecting all the events, the question is are there other metrics we should infer from them? -sandy
      • Got it. - dhellmann

Resources

GET /v2/resources/

Returns the definitions of all the resources. There are no new changes required for StackTach integration.

GET /v2/resources/(resource_id)/

Returns the details for the specified resource_id. There are no new changes required for StackTach integration.

Events

This is a new api section for StackTach integration. The main goals of this section are for querying events.

GET /v2/event_types/

Returns a list of all the event types. Examples of event types are as follows:

  • compute.instance.create.start
  • compute.instance.create.end

Return type:

  • list(EventType) - List of event types. Do we really need a EventType or can we just return list(String).

To simulate the stacky command "./stacky.py events" the http request GET "/v2/event_types/" will return the list of event types.

GET /v2/events/

Returns a list of all the events based on the query filter. Examples of events are as follows:

  • compute.instance.create.start
  • compute.instance.create.end

Using this query all the events for a specific resource, user, project, request ID, host, or time range can be requested.

To simulate the Stacky command "./stacky.py uuid c0c0e50b-cdee-4fe9-9e4c-95ce6cf3fd94" the http request "GET /v2/events" with a query filter of

   {
       "field": "resource_id",
       "op": "eq",
       "value": "bd9431c1-8d69-4ad3-803a-8d4a6b89fd36"
   }

can be used.

To simulate the Stack commane "./stacky.py request req-0809cdd8-21f3-46c3-8803-4a2d9aa45436" the http request "GET /v2/events" with a query filter of

   {
       "field": "request_id",
       "op": "eq",
       "value": "req-0809cdd8-21f3-46c3-8803-4a2d9aa45436"
   }

can be used.

Parameters:

  • q (list(Query)) - Filter rules for the event names to be returned. By specifying different query filters all the queries that StackTach supports can be replicated. For example, to get all the events associated with a specific resource ID the following query filter can be defined:
   {
       "field": "resource_id",
       "op": "eq",
       "value": "bd9431c1-8d69-4ad3-803a-8d4a6b89fd36"
   }


  • NOTE: I believe we need to make one modification to the query filter. As the traits table stores data based on the type of the object, I believe it must be specified in the query, ie "type":"t_int". Any other thoughts on how to get around this? -JohnH

Return type:

  • list(Events) - List of events.

GET /v2/events/(event_id)/

Returns detailed information about the specified event where event_id is the UUID of the event.

TBD. Fill in the details of an event.

To simulate the stacky command "./stacky.py show 1562333" the http request "GET /v2/events/1562333" can be used. Although the event ID will be a UUID"

There are two types of IDs Ceilometer for an Event, the Message ID and the Event ID (PK on Event table). Which should we really use here? There's also the concept of a Message ID for a sample, so that term seems a bit overloaded in Ceilometer. As I understand it, the Message ID attribute is universal to OpenStack to track a single message around OpenStack? I believe the UUID you're referring to is Message ID, in which case, we probably ought to expose it to the Event API, so people can query for it in the event-show (GET /v2/events/<message_id>) API call. Thoughts?? -Thomas

GET /v2/events/traits/

Returns a list of all traits for events. Examples of traits include the following:

  • resource_id
  • project_id
  • user_id
  • request_id
  • host
  • event_type
  • timestamp

This list might be expensive to compute. What's the use case for that data? - dhellmann

  • Should only be a select distinct on the Traits table. Not sure how expensive that is under mongo? -Sandy
  • Not sure if this would be a performance issue in some DBs, such as mongo. I know with the DB that I use, Vertica, this sort of query assuming the data is properly organized, is nearly instantaneous. I hope we would be able to get similar performance from mongo. As a use case, it is useful to be able to query the system for the overall list of traits, but getting the list of traits for an event type, the next one down, is probably more useful. The only other alternative would be to document every trait in the system. If that was done then we could remove this request. --Roland
  • We could organize the data in Mongo in a way to avoid having to compute it (basically, keep a set of the names of all the traits). The bigger question is why is this information useful? How is it going to be used by a consumer of the API? - dhellmann

GET /v2/events/(event_type)/traits/

Return a list of all traits given the specified event type.

GET /v2/events/(event_type)/traits/(trait)/

Returns a list of all values for the specified event type and trait. For example, to model the Stacky command "stacky hosts", which returns a list of all hosts, use "GET /v2/events/compute.instance.create.start/traits/host/"

Parameters:

  • q (list(Query)) - Filter rules for the event names to be returned.

Meters

This section models the API documentation at, http://docs.openstack.org/developer/ceilometer/webapi/v2.html#meters. There are only one new commands, but in some cases we've included the section for the documentation to show how a function in StackTach can be achieved.

GET /v2/meters/

Returns all known meters, … There are no new changes required for StackTach/Stacky.

GET /v2/meters/statistics/

Returns statistics for meters. This is a new command for addressing StackTach integration.

Parameters:

  • q (list(Query)) - Filter rules for the event names to be returned. By specifying different query filters all the queries that StackTach supports can be replicated. For example, to get all the events associated with a specific resource ID the following query filter can be defined:
   {
       "field": "hostname",
       "op": "eq",
       "value": "hostname.domain"
   }

Return type:

  • list(Statistics) - List of statistics.

To simulate the stacky command "./stacky.py summary" the http request "GET v2/meters/statistics" can be used. This can be further refined by using query filters.

How is this different from the existing /v2/meters/(meter_id)/statistics/ query? Just that you don't have to specify the meter you want summarized? Would the grouping by meter be automatic? - dhellmann

  • Yes, that is what this query does. Grouping by meter would be automatic. This query isn't required. Would you like to leave it out? --Roland
  • If it is indeed redundant after we get the groupby blueprint implemented, then yes I think we should leave it out. - dhellmann

GET /v2/meters/(meter_id)/

Returns the samples for the specified meter. There are no new changes required for StackTach integration.

To simulate the Stacky command "./stacky.py timings compute.instance.create" the http request "GET /v2/events" with a query filter of

   {
       "field": "event_type",
       "op": "eq",
       "value": "compute.instance.create"
   }

If no query filter is specified a list of all meters will be returned for resources for which the specfied meter_id is valid.

GET /v2/meters/(meter_id)/statistics/

Returns the statistics for the specified meter, such as min, max, avg. There are no new changes required for StackTach integration.

Parameters:

  • q (list(Query)) - Filter rules for the event names to be returned. By specifying different query filters all the queries that StackTach supports can be replicated. For example, to get all the events associated with a specific resource ID the following query filter can be defined:
   {
       "field": "hostname",
       "op": "eq",
       "value": "hostname.com.com"
   }

Return type:

  • list(Statistics) - List of statistics.

Samples and Statistics

Meter

One category of measurements.

  • request_id: An optional field that specifies the request ID if valid.

Sample

A single measurement for a given meter and resource.

  • request_id: An optional field that specifies the request ID if valid.

Statistics

Computed statistics for a query.

  • request_id: An optional field that specifies the request ID if valid.

Filtering Queries

There are no changes required for StackTach integration. The queries need to support the following fieldes for StackTach integration:

  • resource_id
  • project_id
  • user_id
  • host
  • timestamp


Several additional fields to address StackTach integration are as follows:

  • request_id
  • event_type

Events

This section is new for the StackTack integration.

EventType

A decription of an event type. TBD.

  • Do we really need a EventType or should we just use a String.
    • Currently, EventType is just a String. -Sandy

Event

A description of an event. Need to specify all the details of an Event. Note, events will can be very specific. For example VM lifecycle events have Task State, Build State, ...

NOTE: We will need to add a non-Trait attribute of Source ID to Event. Currently we are using the database ID as the unique identifier, but this will need to change to be externally defined so as to identify duplicate events (possible via AMQP rescheduling) and the db should block duplicate index entries on this column.

Issues