Difference between revisions of "Ceilometer/blueprints/api-group-by"

Revision as of 09:53, 16 January 2013

Summary

The API should allow to do GROUP BY type operation.

User stories

I had an instance running for 6h. It started as a m1.tiny flavor during the first 2 hours and then grew up to a m1.large flavor for the next 4 hours. I need to get this two durations so I can bill them with different rates.

Design

Enhance API v2 so it implements a new arguments to do GROUP BY operations.

For example add:

g[]=<field name>

That solves the user story above with:

/v2/meters/instance/statistics?
 q[0].field=resource&
 q[0].op=eq&
 q[0].value=<my-resource-id>&
 q[1].field=timestamp&
 q[1].op=lt&
 q[1].value=<now>&
 q[2].field=timestamp&
 q[2].op=gt&
 q[2].value=<now - 6 hours>&
 g[0]=metadata.flavor&
 period=360

Would return

{[
  { "m1.tiny": { min: 1, max: 1, avg: 1, sum: 1 },
  { "m1.tiny": { min: 1, max: 1, avg: 1, sum: 1 } },
  { "m1.large": { min: 1, max: 1, avg: 1, sum: 1 } },
  { "m1.large": { min: 1, max: 1, avg: 1, sum: 1 } },
  { "m1.large": { min: 1, max: 1, avg: 1, sum: 1 } },
  { "m1.large": { min: 1, max: 1, avg: 1, sum: 1 } },
]}

Further more, dropping the q[0] request that narrows the search to only one resource allows to retrieve this information for all instances over that period of time:

/v2/meters/instance/statistics?
 q[0].field=timestamp&
 q[0].op=lt&
 q[0].value=<now>&
 q[1].field=timestamp&
 q[1].op=gt&
 q[1].value=<now - 6 hours>&
 g[0]=metadata.flavor&
 g[1]=resource&
 period=360

If there was another large instance, that would return:

{[
  { "m1.tiny": { min: 1, max: 1, avg: 1, sum: 1 }, "m1.large": { min: 1, max: 1, avg: 1, sum: 1 } },
  { "m1.tiny": { min: 1, max: 1, avg: 1, sum: 1 }, "m1.large": { min: 1, max: 1, avg: 1, sum: 1 } },
  { "m1.large": { min: 1, max: 1, avg: 1, sum: 2 } },
  { "m1.large": { min: 1, max: 1, avg: 1, sum: 2 } },
  { "m1.large": { min: 1, max: 1, avg: 1, sum: 2 } },
  { "m1.large": { min: 1, max: 1, avg: 1, sum: 2 } },
]}

Angus's comments/ramblings

Just to update with current impl.

/v2/meters/instance/statistics?
 q[0].field=resource&
 q[0].op=eq&
 q[0].value=<my-resource-id>&
 q[1].field=timestamp&
 q[1].op=lt&
 q[1].value=<now>&
 q[2].field=timestamp&
 q[2].op=gt&
 q[2].value=<now - 6 hours>&
 g[0]=metadata.flavor&
 period=360

1) I assume we can't group by more than one field? If so this should be (not an array):

groupby=metadata.flavor&

2) period is not yet impl. - I'd better get on that ;)

3) Currently we return:

{[
  { "min": 1,
    "max": 1,
    "avg": 1,
    "sum": 1,
    "count": 1,
    "duration": 1,
  },
]}

To show the groupby we could return the following:

{[
  { "min": 1,
    "max": 1,
    "avg": 1,
    "sum": 1,
    "count": 1,
    "duration": 1,
    "groupby": "m1.tiny",
  },
]}

If there is no groupby that can just be None.

4) from an impl. pov (mongo) we have:

   MAP_STATS = bson.code.Code("""
	    function () {
-	        emit('statistics', { min : this.counter_volume,
+	        emit(groupby_field, { min : this.counter_volume,
	                             max : this.counter_volume,
	                             qty : this.counter_volume,
	                             count : 1,
	                             timestamp_min : this.timestamp,
	                             timestamp_max : this.timestamp } )
	    }
	    """)

If we can pass in the groupby field into the above function then this will be super easy. Can we generate this bcode dynamically?

@@ Line 23: / Line 23: @@
 <pre><nowiki>
-/v2/meters/instance?
+/v2/meters/instance/statistics?
   q[0].field=resource&
   q[0].op=eq&
@@ Line 34: / Line 34: @@
   q[2].value=<now - 6 hours>&
   g[0]=metadata.flavor&
- c[0]=sum&
   period=360
 </nowiki></pre>
@@ Line 43: / Line 42: @@
 <pre><nowiki>
 {[
-   { "m1.tiny": 1 },
+   { "m1.tiny": { min: 1, max: 1, avg: 1, sum: 1 },
-   { "m1.tiny": 1 },
+   { "m1.tiny": { min: 1, max: 1, avg: 1, sum: 1 } },
-   { "m1.large": 1 },
+   { "m1.large": { min: 1, max: 1, avg: 1, sum: 1 } },
-   { "m1.large": 1 },
+   { "m1.large": { min: 1, max: 1, avg: 1, sum: 1 } },
-   { "m1.large": 1 },
+   { "m1.large": { min: 1, max: 1, avg: 1, sum: 1 } },
-   { "m1.large": 1 },
+   { "m1.large": { min: 1, max: 1, avg: 1, sum: 1 } },
 ]}
 </nowiki></pre>
-NOTES:
-* Each return value should include the time range for the accounting systems to record. It may be simpler to just extend the statistics API we have (without the "c" argument), and adding a "g" field with the grouping column names and values so the recipient can figure out what each result means. -- dhellmann
-* That's a problem that APIv2 needs to solve, not this blueprint -- jd
 Further more, dropping the q[0] request that narrows the search to only one resource allows to retrieve this information for all instances over that period of time:
 <pre><nowiki>
-/v2/meters/instance?
+/v2/meters/instance/statistics?
   q[0].field=timestamp&
   q[0].op=lt&
@@ Line 69: / Line 64: @@
   g[0]=metadata.flavor&
   g[1]=resource&
- c[0]=sum&
   period=360
 </nowiki></pre>
@@ Line 79: / Line 73: @@
 <pre><nowiki>
 {[
-   { "m1.tiny": 1, "m1.large": 1 },
+   { "m1.tiny": { min: 1, max: 1, avg: 1, sum: 1 }, "m1.large": { min: 1, max: 1, avg: 1, sum: 1 } },
-   { "m1.tiny": 1, "m1.large": 1 },
+   { "m1.tiny": { min: 1, max: 1, avg: 1, sum: 1 }, "m1.large": { min: 1, max: 1, avg: 1, sum: 1 } },
-   { "m1.large": 2 },
+   { "m1.large": { min: 1, max: 1, avg: 1, sum: 2 } },
-   { "m1.large": 2 },
+   { "m1.large": { min: 1, max: 1, avg: 1, sum: 2 } },
-   { "m1.large": 2 },
+   { "m1.large": { min: 1, max: 1, avg: 1, sum: 2 } },
-   { "m1.large": 2 },
+   { "m1.large": { min: 1, max: 1, avg: 1, sum: 2 } },
 ]}
 </nowiki></pre>
-Be careful that specifying a c[] is '''mandatory''' in a GROUP BY operation (as in SQL).
 == Angus's comments/ramblings ==