Difference between revisions of "Monasca/Instrumentation"
< Monasca
Timothyb89 (talk | contribs) (Created page with "== Self-instrumenting Monasca == Goal: instrument Monasca components in the same way end users might instrument their own applications. === Example Metrics === === Options...") |
Timothyb89 (talk | contribs) |
||
| Line 1: | Line 1: | ||
== Self-instrumenting Monasca == | == Self-instrumenting Monasca == | ||
| − | Goal: instrument Monasca components in the same way end users might instrument their own applications. | + | Goal: instrument Monasca components in the same way end users might instrument their own applications, with an emphasis on usability in Kubernetes environments. |
=== Example Metrics === | === Example Metrics === | ||
| + | * API latency per endpoint | ||
| + | * API batch insert time, per message | ||
| + | * API metric insert rate | ||
| + | * Persister metric read and write rates | ||
| + | * Kafka partition lag | ||
| + | * Influx per-metric write latency | ||
| Line 11: | Line 17: | ||
* Already part of Monasca: https://github.com/openstack/monasca-statsd | * Already part of Monasca: https://github.com/openstack/monasca-statsd | ||
* Good developer API | * Good developer API | ||
| + | |||
Cons: | Cons: | ||
| Line 16: | Line 23: | ||
** i.e., language-specific libraries are needed. The only Monasca-compatible library is for Python (monasca-statsd) | ** i.e., language-specific libraries are needed. The only Monasca-compatible library is for Python (monasca-statsd) | ||
* In a k8s environment, a sidecar would need to run with every pod being instrumented | * In a k8s environment, a sidecar would need to run with every pod being instrumented | ||
| + | * Requires authenticated write to Monasca, i.e. configuration required | ||
| + | |||
==== prometheus ==== | ==== prometheus ==== | ||
Pros: | Pros: | ||
* Simple data format: easy to write simple metrics endpoint from scratch in any language | * Simple data format: easy to write simple metrics endpoint from scratch in any language | ||
| − | * | + | * Can take advantage of existing client libraries for most languages |
| − | * Can be scraped with zero configuration or authentication | + | * Can be scraped with zero configuration or authentication (in k8s environments) |
| + | * Can easily inspect the current application state at runtime (just browse to /metrics) | ||
| + | |||
Cons: | Cons: | ||
| − | * | + | * May require sidecar in certain situations, new code needed (see below) |
| + | |||
See blueprint: https://blueprints.launchpad.net/monasca/+spec/prometheus-instrumentation | See blueprint: https://blueprints.launchpad.net/monasca/+spec/prometheus-instrumentation | ||
| Line 30: | Line 42: | ||
===== Sidecar ===== | ===== Sidecar ===== | ||
Prometheus monitoring with multiprocessing apps, like in the Python implementations of monasca-api and monasca-persister, leads to issues: | Prometheus monitoring with multiprocessing apps, like in the Python implementations of monasca-api and monasca-persister, leads to issues: | ||
| − | # Multiprocessing support in the python prometheus_client is | + | # Multiprocessing support in the python prometheus_client is not ideal |
| + | #* Requires large modifications to gunicorn, and keystone middleware, plus custom Falcon endpoints... | ||
# Overloads '/metrics' endpoint (api) | # Overloads '/metrics' endpoint (api) | ||
| + | |||
| + | |||
| + | To work around these limitations, a sidecar server would: | ||
| + | * Ingest metrics from application processes using language-native Prometheus clients | ||
| + | ** Clients would use either a small client plugin (supported in the Python client) or alternative registration method | ||
| + | * Perform simple aggregations to hide individual subprocesses/workers | ||
| + | * Publish aggregated metrics as standard Prometheus /metrics | ||
| + | * Have a minimal system footprint | ||
| + | |||
See blueprint: https://blueprints.launchpad.net/monasca/+spec/monasca-sidecar | See blueprint: https://blueprints.launchpad.net/monasca/+spec/monasca-sidecar | ||
| + | |||
| + | See PoC: https://github.com/timothyb89/monasca-sidecar | ||
Revision as of 22:47, 24 January 2017
Contents
Self-instrumenting Monasca
Goal: instrument Monasca components in the same way end users might instrument their own applications, with an emphasis on usability in Kubernetes environments.
Example Metrics
- API latency per endpoint
- API batch insert time, per message
- API metric insert rate
- Persister metric read and write rates
- Kafka partition lag
- Influx per-metric write latency
Options
monasca-statsd
Pros:
- Already part of Monasca: https://github.com/openstack/monasca-statsd
- Good developer API
Cons:
- statsd dialects are not compatible between backends, e.g. monasca, prometheus, datadog, statsd proper
- i.e., language-specific libraries are needed. The only Monasca-compatible library is for Python (monasca-statsd)
- In a k8s environment, a sidecar would need to run with every pod being instrumented
- Requires authenticated write to Monasca, i.e. configuration required
prometheus
Pros:
- Simple data format: easy to write simple metrics endpoint from scratch in any language
- Can take advantage of existing client libraries for most languages
- Can be scraped with zero configuration or authentication (in k8s environments)
- Can easily inspect the current application state at runtime (just browse to /metrics)
Cons:
- May require sidecar in certain situations, new code needed (see below)
See blueprint: https://blueprints.launchpad.net/monasca/+spec/prometheus-instrumentation
Sidecar
Prometheus monitoring with multiprocessing apps, like in the Python implementations of monasca-api and monasca-persister, leads to issues:
- Multiprocessing support in the python prometheus_client is not ideal
- Requires large modifications to gunicorn, and keystone middleware, plus custom Falcon endpoints...
- Overloads '/metrics' endpoint (api)
To work around these limitations, a sidecar server would:
- Ingest metrics from application processes using language-native Prometheus clients
- Clients would use either a small client plugin (supported in the Python client) or alternative registration method
- Perform simple aggregations to hide individual subprocesses/workers
- Publish aggregated metrics as standard Prometheus /metrics
- Have a minimal system footprint
See blueprint: https://blueprints.launchpad.net/monasca/+spec/monasca-sidecar
