EfficientMetering/GrizzlySummit/BeyondMetering - Revision history

Ryan Lane: Text replace - "NOTOC" to ""

2013-02-17T23:29:20Z

Text replace - "__NOTOC__" to ""

Asalkeld at 00:36, 22 November 2012

2012-11-22T00:36:40Z

Asalkeld at 23:27, 21 November 2012

2012-11-21T23:27:32Z

Asalkeld at 10:58, 8 November 2012

2012-11-08T10:58:34Z

Asalkeld at 05:24, 8 November 2012

2012-11-08T05:24:46Z

Asalkeld at 05:14, 8 November 2012

2012-11-08T05:14:24Z

Asalkeld at 03:58, 8 November 2012

2012-11-08T03:58:06Z

Asalkeld at 05:09, 13 October 2012

2012-10-13T05:09:44Z

Doug at 20:03, 10 October 2012

2012-10-10T20:03:47Z

Nijaba at 09:50, 8 October 2012

2012-10-08T09:50:04Z

← Older revision		Revision as of 23:29, 17 February 2013
Line 1:		Line 1:
−	~~__NOTOC__~~	+
	= Beyond Metering - Extending Ceilometer =		= Beyond Metering - Extending Ceilometer =

← Older revision		Revision as of 00:36, 22 November 2012
Line 90:		Line 90:

	== User Rest API ==		== User Rest API ==
		+
		+	=== Other API to take into account: ===
		+	* http://docs.amazonwebservices.com/AmazonCloudWatch/latest/APIReference/Welcome.html
		+	* http://dev.librato.com/v1/post/metrics
		+	* http://dmtf.org/sites/default/files/standards/documents/DSP0263_1.0.1.pdf page ~ 150

	=== Metrics: ===		=== Metrics: ===

← Older revision		Revision as of 23:27, 21 November 2012
Line 124:		Line 124:

	Some are already collected in ceilometer in the counter as resource_id or metadata		Some are already collected in ceilometer in the counter as resource_id or metadata
−
−	~~== Horizontal Scale ==~~
−
−	~~So we either _require_ Cassandra or do something like the below.~~
−
−	~~......~~
−
−	~~Hash on the resource_id to access the storage for a particular metric.~~
−	~~But how do we know where a user's metrics are stored?~~
−	~~ask nova/cinder/quantum/... ?~~
−	~~And how do we know where the data is for an autoscaling group?~~
−
−	~~Re: grouping/dimensions (an idea)~~
−	~~# make a virtual resource for each dimension that we don't explicitly measure (it gets hashed and stored in a known place)~~
−	~~# create an event like for instance create/resize/destroy that we use to modify the db entry to point to the actual resource~~
−	~~# when queried we just look for the resource "autoscaling_group_4" and we can then "see" how to find the statistics for that dimension~~
−
−	~~so when we see an instance create we need to modify image_id and instance_type virtual resources~~
−	~~also heat will need to create rpc events like nova does but for scaling groups (create/resize/destroy essentially membership)~~
−
−	~~If we do this, I think we can do better that CW regarding querying based on dimensions:~~
−	~~http://docs.amazonwebservices.com/AmazonCloudWatch/latest/DeveloperGuide/cloudwatch_concepts.html#Di...~~
−
−	~~We could also make the user a virtual resource to make finding the location of the user's resources easier.~~
−	~~Basically we end up with a number of fields that we can use to locate the metric data.~~
−
−	~~So logically we hash "image-04" and get db-on-host-1, then we hash each of the children to go find the hosts~~
−	~~where each resource's data is stored.~~
−
−
−	~~<pre><nowiki>~~
−	~~db-on-host-1:~~
−
−	~~identifiers:~~
−	~~id: image-04~~
−	~~type: image_id~~
−	~~children: {inst-094, inst-092}~~
−
−	~~id: autogroup-02~~
−	~~type: autoscale_group~~
−	~~children: {inst-094, inst-092, user-023, project-021}~~
−
−	~~db-on-host-2:~~
−
−	~~data:~~
−	~~name: cpu_util~~
−	~~value:~~
−	~~units:~~
−	~~timestamp:~~
−	~~resource_id: inst-094~~
−	~~</nowiki></pre>~~
−

	== Alarms ==		== Alarms ==

← Older revision		Revision as of 10:58, 8 November 2012
Line 126:		Line 126:

	== Horizontal Scale ==		== Horizontal Scale ==
		+
		+	So we either _require_ Cassandra or do something like the below.
		+
		+	......

	Hash on the resource_id to access the storage for a particular metric.		Hash on the resource_id to access the storage for a particular metric.

@@ Line 146: / Line 146: @@
 Basically we end up with a number of fields that we can use to locate the metric data.
-So logically we hash "image-num-4" and get db-on-host-1, then we hash each of the children to go find the hosts
+So logically we hash "image-04" and get db-on-host-1, then we hash each of the children to go find the hosts
 where each resource's data is stored.
@@ Line 153: / Line 153: @@
 db-on-host-1:
-   resources:
+   identifiers:
-     id: image-num-4
+     id: image-04
      type: image_id
-     children: {inst-094, bla, bla}
+     children: {inst-094, inst-092}
 db-on-host-2:
@@ Line 168: / Line 172: @@
 </nowiki></pre>
 == Alarms ==

← Older revision		Revision as of 05:14, 8 November 2012
Line 78:		Line 78:
	* have one vibrant community rather that two that are less so		* have one vibrant community rather that two that are less so
	* code reuse		* code reuse
		+
		+	= Requirements =
		+
		+	== Metrics ==
		+
		+	Kinds of metrics to monitor
		+	http://docs.amazonwebservices.com/AmazonCloudWatch/latest/DeveloperGuide/CW_Support_For_AWS.html
		+
		+	Sample at between 10s to 60s
		+	Transmit at between 1min and 5min
		+
		+	== User Rest API ==
		+
		+	=== Metrics: ===
		+	* list metrics
		+
		+	=== Data: ===
		+	* get stats
		+	* put custom metric data
		+
		+	=== Alarms: ===
		+	* create alarm
		+	* delete alarm
		+	* show alarm
		+	* show alarm history
		+	* enable/disable alarm
		+	* set alarm state (tempory override)
		+
		+	=== What fields are needed? ===
		+
		+	* namespace.name
		+	* dimensions (resource_id, ++)
		+	* time stamp
		+	* units
		+	* user_id
		+	* project_id
		+
		+	== Groupings (aws dimensions) ==
		+
		+	Some examples:
		+	* Autoscaling groupname
		+	* image id
		+	* instance id (resource_id)
		+	* instance type
		+
		+	Some are already collected in ceilometer in the counter as resource_id or metadata
		+
		+	== Horizontal Scale ==
		+
		+	Hash on the resource_id to access the storage for a particular metric.
		+	But how do we know where a user's metrics are stored?
		+	ask nova/cinder/quantum/... ?
		+	And how do we know where the data is for an autoscaling group?
		+
		+	Re: grouping/dimensions (an idea)
		+	# make a virtual resource for each dimension that we don't explicitly measure (it gets hashed and stored in a known place)
		+	# create an event like for instance create/resize/destroy that we use to modify the db entry to point to the actual resource
		+	# when queried we just look for the resource "autoscaling_group_4" and we can then "see" how to find the statistics for that dimension
		+
		+	so when we see an instance create we need to modify image_id and instance_type virtual resources
		+	also heat will need to create rpc events like nova does but for scaling groups (create/resize/destroy essentially membership)
		+
		+	If we do this, I think we can do better that CW regarding querying based on dimensions:
		+	http://docs.amazonwebservices.com/AmazonCloudWatch/latest/DeveloperGuide/cloudwatch_concepts.html#Di...
		+
		+	We could also make the user a virtual resource to make finding the location of the user's resources easier.
		+	Basically we end up with a number of fields that we can use to locate the metric data.
		+
		+	So logically we hash "image-num-4" and get db-on-host-1, then we hash each of the children to go find the hosts
		+	where each resource's data is stored.
		+
		+
		+	<pre><nowiki>
		+	db-on-host-1:
		+
		+	resources:
		+	id: image-num-4
		+	type: image_id
		+	children: {inst-094, bla, bla}
		+
		+	db-on-host-2:
		+
		+	data:
		+	name: cpu_util
		+	value:
		+	units:
		+	timestamp:
		+	resource_id: inst-094
		+	</nowiki></pre>
		+
		+
		+	As an extension we could allow the user to create user-groupings of their own and alarm on those.
		+
		+	== Alarms ==
		+
		+	Initially this can be poll based and run on the node that the resource is hashed to (including virtual resources/dimensions)
		+
		+	== Instrumentation ==
		+
		+	How does this fit in with Ceilometer? - we will need to modify the agents to send the info we need at the rate we need.
		+	The agent would need to hash the resource_id and find out where to send the metric data.
		+
		+	== Optimisations (reducing network activity) ==
		+
		+	Assume data aggregation, but have an api to turn that off.
		+
		+	What does that mean?
		+	The agent could locally (in memory) aggregate multiple samples into one data point (sum=31,min=2,max=5,sample_count=10) and only send that at a much higher period. An alternative is to still send all the data points (but in bulk) just delayed.
		+
		+	We could automatically turn this off when we have alarms attached to metrics.

@@ Line 41: / Line 41: @@
 === Some examples ===
-"Tell me when the maximum disk utilization exceeds 90%"
+* "Tell me when the maximum disk utilization exceeds 90%"
-"Tell me when the average CPU utilization exceeds 80% over 120 seconds"
+* "Tell me when the average CPU utilization exceeds 80% over 120 seconds"
-"Tell me when my web app is becoming unresponsive" (loadbalancer latency meter)
+* "Tell me when my web app is becoming unresponsive" (loadbalancer latency meter)
-"Tell me when my httpd daemon dies" (custom user script that checks daemon health)
+* "Tell me when my httpd daemon dies" (custom user script that checks daemon health)
 == Why integrate them into one project? ==

@@ Line 48: / Line 48: @@
 == Why integrate them into one project? ==
-* reuse the data collectors / pollsters
+* one statistics api
 * have one vibrant community rather that two that are less so
 * code reuse

@@ Line 1: / Line 1: @@
 __NOTOC__
 = Beyond Metering - Extending Ceilometer =
 Over the past few months, numerous request have been made to the Ceilometer project to extend its scope from just metering to more monitoring or alerting.  This causes quite a few challenges but as we are instrumenting more and more openstack component to extract data from them, it makes sense to think on how we could extend our capabilities over time.