Ceilometer/ComplexFilterExpressionsInAPIQueries
Contents
Architecture
The purpose of the improvement is to support rich query functionality in Ceilometer API. The current query support is limited, it's grammar cannot be extended easily.
REST Resources
The new solution includes new REST resources for provide complex queries for both the meter and alarm related data:
| REST Resource | Functionality | Returned Object Type |
|---|---|---|
| '/query/samples' | retrieve samples for meter(s) | Sample |
| '/query/alarms' | retrieve alarms | Alarm |
| '/query/alarms/history' | retrieve alarm history | AlarmChange |
As new resources were defined, the simple query support remained as it is now, which means that the API is backward compatible and also a new query grammar could be defined without limitations.
POST request
The new solution uses POST request, as the functionality of the query part of the GET request is very limited. The body of the POST request contains the query parameters in JSON format, which makes the processing of the query easier on the API level.
Filter Expression Language
filter := expression
expression := simple_expression| simple_in_expression | complex_expression
simple_expression := {simple_operator: {field_name: value}}
simple_in_expression := {in: {field_name: [value, value, ...]}}
simple_operator := = | != | < | <= | > | >= | in
complex_expression := {complex_operator: [expression, expression, ...]} | not_expression
not_expression := {not: expression}
complex_operator := and | or
Where
- field_name is the name of a field of the Sample, Alarm or AlarmChange object
- currently directly mapped to db column name
- validated against the db models
- value can be
- boolean
- int
- float
- string
- iso date string for timestamp and state_timestamp fields
- metadata fields of the Sample object can be used in the query with metadata.<field_name> syntax. For example: {"=": {"metadata.display_name" : "server1"}}
- metadata field names are not validated and therefore filtering on a nonexistent metadata field does not return an error but an empty result instead
DB engine differences
The not operator has different meaning in Mongo DB and in SQL. If the not operator is applied on a nonexistent metadata field then the result depends on the DB engine. For example if {"not": {"metadata.nonexistent_field" : "some value"}} filter is used in a query the Mongo DB will return every Sample object from its DB as "not" operator evaluated true for every Sample where the given field does not exists. See more in the Mongod DB doc. In the other hand SqlAlchemy DB engine will return empty result as the SQL join operation on the metadata table will return zero row as the on clause of the join which tries to match on the metadata field name is never fulfilled.
Orderby
List of field_name and sort instruction pairs:
orderby: [{"field_name1": "ASC"}, {"field_name2": "DESC"}, {"field_name3": "ASC"}, ...]
Limit
Number of rows returned, defined value should be a positive int.
Supported DB drivers
- SQLAlchmey
- MongoDB
Further improvements
- Statistics
- Support complex filter functionality for Statistics
- Separate blueprint
- Stored query
- The architecture of this new REST resource provides the possibility to implement stored query support
- Future blueprint
API example
Filter expression
Check for cpu_util samples reported between 18:00-18:15 or between 18:30 - 18:45 where the utilization is between 23 and 26 percent.
cpu_util > 23% AND cpu_util < 26% AND ((timestamp < 2013-12-01T18:15:00 AND timestamp > 2013-12-01T18:00:00) OR (timestamp < 2013-12-02T18:45:00 AND timestamp > 2013-12-02T18:30:00))
Pretty printed json filter:
{"and":
[{"and":
[{"=": {"counter_name": "cpu_util"}},
{">": {"counter_volume": 0.23}},
{"<": {"counter_volume": 0.26}}]},
{"or":
[{"and":
[{">": {"timestamp": "2013-12-01T18:00:00"}},
{"<": {"timestamp": "2013-12-01T18:15:00"}}]},
{"and":
[{">": {"timestamp": "2013-12-01T18:30:00"}},
{"<": {"timestamp": "2013-12-01T18:45:00"}}]}]}]}
Filter expression in the POST request body after converted from json to string:
{
"filter": "{\"and\": [{\"and\": [{\"=\": {\"counter_name\": \"cpu_util\"}}, {\">\": {\"counter_volume\": 0.23}}, {\"<\": {\"counter_volume\": 0.26}}]}, {\"or\": [{\"and\": [{\">\": {\"timestamp\": \"2013-12-01T18:00:00\"}}, {\"<\": {\"timestamp\": \"2013-12-01T18:15:00\"}}]}, {\"and\": [{\">\": {\"timestamp\": \"2013-12-01T18:30:00\"}}, {\"<\": {\"timestamp\": \"2013-12-01T18:45:00\"}}]}]}]}"
}
Samples query
The current implementation supports to retrieve limited number of values and using multiple orderby fields with the possibility of specifying the ordering direction too. An example query can be to request the first 4 samples with the filter specified above, with ordering the result by the values of the counter_volume field with ascending order and then by the timestamp with descending order.
POST request body of the query:
{
"filter" : "{\"and\":[{\"and\": [{\"=\": {\"counter_name\": \"cpu_util\"}}, {\">\": {\"counter_volume\": 0.23}}, {\"<\": {\"counter_volume\": 0.26}}]}, {\"or\": [{\"and\": [{\">\": {\"timestamp\": \"2013-12-01T18:00:00\"}}, {\"<\": {\"timestamp\": \"2013-12-01T18:15:00\"}}]}, {\"and\": [{\">\": {\"timestamp\": \"2013-12-01T18:30:00\"}}, {\"<\": {\"timestamp\": \"2013-12-01T18:45:00\"}}]}]}]}",
"orderby" : "[{\"counter_volume\": \"ASC\"}, {\"timestamp\": \"DESC\"}]",
"limit" : 4
}
After saving the request body into a file, e.g: body.txt, the POST request will be the following:
curl -X POST -H 'X-Auth-Token: <inserttokenhere>' -H 'Content-Type: application/json' -d @body.txt \ http://localhost:8777/v2/query/samples
