Difference between revisions of "Ceilometer/ComplexFilterExpressionsInAPIQueries"

Revision as of 19:27, 10 February 2014

Architecture

The purpose of the improvement is to support rich query functionality in Ceilometer API. The current query support is limited, it's grammar cannot be extended easily.

REST Resources

The new solution includes new REST resources for provide complex queries for both the meter and alarm related data:

As new resources were defined, the simple query support remained as it is now, which means that the API is backward compatible and also a new query grammar could be defined without limitations.

POST request

The new solution uses POST request, as the functionality of the query part of the GET request is very limited. The body of the POST request contains the query parameters in JSON format, which makes the processing of the query easier on the API level.

Filter Expression Language

 filter := expression
 expression := simple_expression| simple_in_expression | complex_expression
 simple_expression := {simple_operator: {field_name: value}}
 simple_in_expression := {in: {field_name: [value, value, ...]}}
 simple_operator := = | != | < | <= | > | >= | in
 complex_expression := {complex_operator: [expression, expression, ...]} | not_expression
 not_expression := {not: expression}
 complex_operator := and | or

Where

• field_name is the name of a field of the Sample, Alarm or AlarmChange object
  • currently directly mapped to db column name
  • validated against the db models
• value can be
  • boolean
  • int
  • float
  • string
  • iso date string for timestamp and state_timestamp fields
• metadata fields of the Sample object can be used in the query with metadata.<field_name> syntax. For example: {"=": {"metadata.display_name" : "server1"}}
  • metadata field names are not validated and therefore filtering on a nonexistent metadata field does not return an error but an empty result instead

DB engine differences

The not operator has different meaning in Mongo DB and in SQL. If the not operator is applied on a nonexistent metadata field then the result depends on the DB engine. For example if {"not": {"metadata.nonexistent_field" : "some value"}} filter is used in a query the Mongo DB will return every Sample object from its DB as "not" operator evaluated true for every Sample where the given field does not exists. See more in the Mongod DB doc. In the other hand SqlAlchemy DB engine will return empty result as the SQL join operation on the metadata table will return zero row as the on clause of the join which tries to match on the metadata field name is never fulfilled.

Orderby

List of field_name and sort instruction pairs:

orderby: [{"field_name1": "ASC"}, {"field_name2": "DESC"}, {"field_name3": "ASC"}, ...]

Limit

Number of rows returned, defined value should be a positive int.

Supported DB drivers

• SQLAlchmey
• MongoDB

Further improvements

• Statistics
  • Support complex filter functionality for Statistics
  • Separate blueprint
• Stored query
  • The architecture of this new REST resource provides the possibility to implement stored query support
  • Future blueprint

API example

Filter expression

Check for cpu_util samples reported between 18:00-18:15 or between 18:30 - 18:45 where the utilization is between 23 and 26 percent.

cpu_util > 23% AND cpu_util < 26% AND ((timestamp < 2013-12-01T18:15:00 AND timestamp > 2013-12-01T18:00:00) OR (timestamp < 2013-12-02T18:45:00 AND timestamp > 2013-12-02T18:30:00))

Pretty printed json filter:

{"and":
 [{"and":
  [{"=": {"counter_name": "cpu_util"}},
   {">": {"counter_volume": 0.23}},
   {"<": {"counter_volume": 0.26}}]},
  {"or":
  [{"and":
   [{">": {"timestamp": "2013-12-01T18:00:00"}},
    {"<": {"timestamp": "2013-12-01T18:15:00"}}]},
   {"and":
    [{">": {"timestamp": "2013-12-01T18:30:00"}},
     {"<": {"timestamp": "2013-12-01T18:45:00"}}]}]}]}

Filter expression in the POST request body after converted from json to string:

{
"filter": "{\"and\": [{\"and\": [{\"=\": {\"counter_name\": \"cpu_util\"}}, {\">\": {\"counter_volume\": 0.23}}, {\"<\": {\"counter_volume\": 0.26}}]}, {\"or\": [{\"and\": [{\">\": {\"timestamp\": \"2013-12-01T18:00:00\"}}, {\"<\": {\"timestamp\": \"2013-12-01T18:15:00\"}}]}, {\"and\": [{\">\": {\"timestamp\": \"2013-12-01T18:30:00\"}}, {\"<\": {\"timestamp\": \"2013-12-01T18:45:00\"}}]}]}]}"
}

Samples query

The current implementation supports to retrieve limited number of values and using multiple orderby fields with the possibility of specifying the ordering direction too. An example query can be to request the first 4 samples with the filter specified above, with ordering the result by the values of the counter_volume field with ascending order and then by the timestamp with descending order.

POST request body of the query:

{
"filter" : "{\"and\":[{\"and\": [{\"=\": {\"counter_name\": \"cpu_util\"}}, {\">\": {\"counter_volume\": 0.23}}, {\"<\": {\"counter_volume\": 0.26}}]}, [{\"or\": [{\"and\": [{\">\": {\"timestamp\": \"2013-12-01T18:00:00\"}}, {\"<\": {\"timestamp\": \"2013-12-01T18:15:00\"}}]}, {\"and\": [{\">\": {\"timestamp\": \"2013-12-01T18:30:00\"}}, {\"<\": {\"timestamp\": \"2013-12-01T18:45:00\"}}]}]}]]}",
"orderby" : "[{\"counter_volume\": \"ASC\"}, {\"timestamp\": \"DESC\"}]",
"limit" : 4
}

After saving the request body into a file, e.g: body.txt, the POST request will be the following:

curl -X POST -H 'X-Auth-Token: <inserttokenhere>' -H 'Content-Type: application/json' -d @body.txt \
  http://localhost:8777/v2/query/samples