OpenStack - User contributions [en]

Ceilometer/blueprints/transportation-pipeline-monitoring-metrics

2013-04-07T09:24:04Z

Shengjie min: /* Alternative for metrics pipeline */

== Summary ==
Compared to metering data, the monitoring metrics need to be collected in a more frequent fashion and the amount of data is way bigger. Does the monitoring metrics necessarily need to go through the message bus as metering data does. Or the option worth being considered is to write them directly to data store.

==The current metering pipeline==
<pre>
---
-
name: meter_pipeline
interval: 60
counters:
- "*"
transformers:
publishers:
- meter_publisher
</pre>

==Alternative for metrics pipeline==
<pre>
---
-
name: metrics_pipeline
interval: 5 // more frequent polling might be needed by monitoring
counters:
- "*"
transformers:
publishers:
- metrics_publisher // metrics publisher might directly write to backend storage (avoid MQ)
- metrics_notifier(publisher) // inline pre-processing metrics notification
</pre>

[[Image:ceilometer-metrics-pipeline.png]]

=== Direct write to data store ===
Storage backend has record_metering_message() to store the metering message(converted from counter) to backend storage via RPC(message queue). Another record_monitoring_metrics() method might just store metrics directly to the storage back end.

=== Pre-processing/In-line calculation ===
Transformer might be used to do some inline pre-processing before the metrics get stored in data store. This allows some critical metrics or calculated metrics being notified in a real-time manner before persistence.

"real-time single metric", when it hits some thresholds or matches some specific values, the notifications can be triggered immediately. eg. The capacity reaches the high-end limit.

"calculated metric", some metrics might sequentially be kept in memory by transformers (similar to AccumulatorTransformer) for some in-line calculation. When the PublishContext exits, the calculated result might get flushed to data store as new type of metric. If it meets some criteria and it can trigger some notifications too. eg. a few drastically increasing storage metrics might indicate a sharp usage rise in a short period of time.

=== Post-processing/analytics ===
Post-processing, analytics and mapred tasks might be carried out periodically against data store directly or via monitoring API, this is more for time-consuming analytics tasks.

File:Ceilometer-metrics-pipeline.png

2013-04-07T09:22:20Z

Shengjie min:

Ceilometer/blueprints/transportation-pipeline-monitoring-metrics

2013-04-07T09:19:30Z

Shengjie min: /* Alternative for metrics pipeline */

Ceilometer/blueprints/transportation-pipeline-monitoring-metrics

2013-04-07T09:15:35Z

Shengjie min: /* Alternative for metrics pipeline */

Ceilometer/blueprints/transportation-pipeline-monitoring-metrics

2013-04-07T09:15:13Z

Shengjie min: /* Alternative for metrics pipeline */

Ceilometer/blueprints/transportation-pipeline-monitoring-metrics

2013-04-07T09:11:05Z

Shengjie min: Created page with "== Summary == Compared to metering data, the monitoring metrics need to be collected in a more frequent fashion and the amount of data is way bigger. Does the monitoring metri..."

Ceilometer/blueprints/monitoring-metrics-object

2013-04-07T08:57:04Z

Shengjie min: /* Monitoring Metrics */

== Meters ==
In Ceilometer, the meters are polled in the form of Counter(namedtuple) as below, do we need two different peer objects or a common lower level object for both meters and metrics?
<pre>
Counter = collections.namedtuple('Counter',
' '.join([
'name',
'type',
'unit',
'volume',
'user_id',
'project_id',
'resource_id',
'timestamp',
'resource_metadata',
]))
</pre>

== Monitoring Metrics ==
As the monitoring metrics, take ganglia metric as an example(Gmetad XML dump):
<pre>
-Host
-Cluster Name
-METRIC NAME="swap_free"
-VAL="2097148"
-TYPE="float"
-UNITS="KB"
-TN="138" - timestamp
-TMAX="180" (like a leasetime - freshness)
-DMAX="0" (how long to retain old records)
-SLOPE="both" (+ = counter, others = gauge)
</pre>

== Mapping ==
{| class="wikitable"
|-
! Ceilometer Counter !! Ganglia Metric
|-
| name || METRIＣNAME
|-
|Type(gauge|delta|cumulative) || SLOPE(zero|positive|negative|both) more from metadata
|-
| unit|| UNITS
|-
| volume || VAL
|-
| user_id || None (appropriate for a meter, not monitor)
|-
| project_id || None (appropriate for a meter, not monitor)
|-
| resource_id(swift: tenant_id, compute:instance_id, glance:image_id)|| Host (FQDN)
|-
| timestamp|| Ganglia TN (current time - TN = timestamp when metric published)
|-
| resource_metadata || NONE
|}

Ceilometer/blueprints/monitoring-metrics-object

2013-04-07T08:56:27Z

Shengjie min: /* Meters */

== Meters ==
In Ceilometer, the meters are polled in the form of Counter(namedtuple) as below, do we need two different peer objects or a common lower level object for both meters and metrics?
<pre>
Counter = collections.namedtuple('Counter',
' '.join([
'name',
'type',
'unit',
'volume',
'user_id',
'project_id',
'resource_id',
'timestamp',
'resource_metadata',
]))
</pre>

== Monitoring Metrics ==
As the monitoring metrics, take ganglia metric as an example(Gmetad XML dump):

:-Host
: -Cluster Name
: -METRIC NAME="swap_free"
: -VAL="2097148"
: -TYPE="float"
: -UNITS="KB"
: -TN="138" - timestamp
: -TMAX="180" (like a leasetime - freshness)
: -DMAX="0" (how long to retain old records)
: -SLOPE="both" (+ = counter, others = gauge)

== Mapping ==
{| class="wikitable"
|-
! Ceilometer Counter !! Ganglia Metric
|-
| name || METRIＣNAME
|-
|Type(gauge|delta|cumulative) || SLOPE(zero|positive|negative|both) more from metadata
|-
| unit|| UNITS
|-
| volume || VAL
|-
| user_id || None (appropriate for a meter, not monitor)
|-
| project_id || None (appropriate for a meter, not monitor)
|-
| resource_id(swift: tenant_id, compute:instance_id, glance:image_id)|| Host (FQDN)
|-
| timestamp|| Ganglia TN (current time - TN = timestamp when metric published)
|-
| resource_metadata || NONE
|}

Ceilometer/blueprints/monitoring-metrics-object

2013-04-07T08:55:25Z

Shengjie min: /* Meters */

== Meters ==
In Ceilometer, the meters are polled in the form of Counter(namedtuple) as below, do we need two different peer objects or a common lower level object for both meters and metrics?
:Counter = collections.namedtuple('Counter',
: ' '.join([
: 'name',
: 'type',
: 'unit',
: 'volume',
: 'user_id',
: 'project_id',
: 'resource_id',
: 'timestamp',
: 'resource_metadata',
: ]))

== Monitoring Metrics ==
As the monitoring metrics, take ganglia metric as an example(Gmetad XML dump):

:-Host
: -Cluster Name
: -METRIC NAME="swap_free"
: -VAL="2097148"
: -TYPE="float"
: -UNITS="KB"
: -TN="138" - timestamp
: -TMAX="180" (like a leasetime - freshness)
: -DMAX="0" (how long to retain old records)
: -SLOPE="both" (+ = counter, others = gauge)

== Mapping ==
{| class="wikitable"
|-
! Ceilometer Counter !! Ganglia Metric
|-
| name || METRIＣNAME
|-
|Type(gauge|delta|cumulative) || SLOPE(zero|positive|negative|both) more from metadata
|-
| unit|| UNITS
|-
| volume || VAL
|-
| user_id || None (appropriate for a meter, not monitor)
|-
| project_id || None (appropriate for a meter, not monitor)
|-
| resource_id(swift: tenant_id, compute:instance_id, glance:image_id)|| Host (FQDN)
|-
| timestamp|| Ganglia TN (current time - TN = timestamp when metric published)
|-
| resource_metadata || NONE
|}

Ceilometer/blueprints/monitoring-metrics-object

2013-04-07T08:55:02Z

Shengjie min: Created page with "== Meters == In Ceilometer, the meters are polled in the form of Counter(namedtuple) as below, do we need two different peer objects or a common lower level object for both me..."

== Meters ==
In Ceilometer, the meters are polled in the form of Counter(namedtuple) as below, do we need two different peer objects or a common lower level object for both meters and metrics?

:Counter = collections.namedtuple('Counter',
: ' '.join([
: 'name',
: 'type',
: 'unit',
: 'volume',
: 'user_id',
: 'project_id',
: 'resource_id',
: 'timestamp',
: 'resource_metadata',
: ]))

== Monitoring Metrics ==
As the monitoring metrics, take ganglia metric as an example(Gmetad XML dump):

:-Host
: -Cluster Name
: -METRIC NAME="swap_free"
: -VAL="2097148"
: -TYPE="float"
: -UNITS="KB"
: -TN="138" - timestamp
: -TMAX="180" (like a leasetime - freshness)
: -DMAX="0" (how long to retain old records)
: -SLOPE="both" (+ = counter, others = gauge)

== Mapping ==
{| class="wikitable"
|-
! Ceilometer Counter !! Ganglia Metric
|-
| name || METRIＣNAME
|-
|Type(gauge|delta|cumulative) || SLOPE(zero|positive|negative|both) more from metadata
|-
| unit|| UNITS
|-
| volume || VAL
|-
| user_id || None (appropriate for a meter, not monitor)
|-
| project_id || None (appropriate for a meter, not monitor)
|-
| resource_id(swift: tenant_id, compute:instance_id, glance:image_id)|| Host (FQDN)
|-
| timestamp|| Ganglia TN (current time - TN = timestamp when metric published)
|-
| resource_metadata || NONE
|}