Difference between revisions of "Monasca/Instrumentation"
< Monasca
Timothyb89 (talk | contribs) (Created page with "== Self-instrumenting Monasca == Goal: instrument Monasca components in the same way end users might instrument their own applications. === Example Metrics === === Options...") |
Timothyb89 (talk | contribs) |
||
Line 1: | Line 1: | ||
== Self-instrumenting Monasca == | == Self-instrumenting Monasca == | ||
− | Goal: instrument Monasca components in the same way end users might instrument their own applications. | + | Goal: instrument Monasca components in the same way end users might instrument their own applications, with an emphasis on usability in Kubernetes environments. |
=== Example Metrics === | === Example Metrics === | ||
+ | * API latency per endpoint | ||
+ | * API batch insert time, per message | ||
+ | * API metric insert rate | ||
+ | * Persister metric read and write rates | ||
+ | * Kafka partition lag | ||
+ | * Influx per-metric write latency | ||
Line 11: | Line 17: | ||
* Already part of Monasca: https://github.com/openstack/monasca-statsd | * Already part of Monasca: https://github.com/openstack/monasca-statsd | ||
* Good developer API | * Good developer API | ||
+ | |||
Cons: | Cons: | ||
Line 16: | Line 23: | ||
** i.e., language-specific libraries are needed. The only Monasca-compatible library is for Python (monasca-statsd) | ** i.e., language-specific libraries are needed. The only Monasca-compatible library is for Python (monasca-statsd) | ||
* In a k8s environment, a sidecar would need to run with every pod being instrumented | * In a k8s environment, a sidecar would need to run with every pod being instrumented | ||
+ | * Requires authenticated write to Monasca, i.e. configuration required | ||
+ | |||
==== prometheus ==== | ==== prometheus ==== | ||
Pros: | Pros: | ||
* Simple data format: easy to write simple metrics endpoint from scratch in any language | * Simple data format: easy to write simple metrics endpoint from scratch in any language | ||
− | * | + | * Can take advantage of existing client libraries for most languages |
− | * Can be scraped with zero configuration or authentication | + | * Can be scraped with zero configuration or authentication (in k8s environments) |
+ | * Can easily inspect the current application state at runtime (just browse to /metrics) | ||
+ | |||
Cons: | Cons: | ||
− | * | + | * May require sidecar in certain situations, new code needed (see below) |
+ | |||
See blueprint: https://blueprints.launchpad.net/monasca/+spec/prometheus-instrumentation | See blueprint: https://blueprints.launchpad.net/monasca/+spec/prometheus-instrumentation | ||
Line 30: | Line 42: | ||
===== Sidecar ===== | ===== Sidecar ===== | ||
Prometheus monitoring with multiprocessing apps, like in the Python implementations of monasca-api and monasca-persister, leads to issues: | Prometheus monitoring with multiprocessing apps, like in the Python implementations of monasca-api and monasca-persister, leads to issues: | ||
− | # Multiprocessing support in the python prometheus_client is | + | # Multiprocessing support in the python prometheus_client is not ideal |
+ | #* Requires large modifications to gunicorn, and keystone middleware, plus custom Falcon endpoints... | ||
# Overloads '/metrics' endpoint (api) | # Overloads '/metrics' endpoint (api) | ||
+ | |||
+ | |||
+ | To work around these limitations, a sidecar server would: | ||
+ | * Ingest metrics from application processes using language-native Prometheus clients | ||
+ | ** Clients would use either a small client plugin (supported in the Python client) or alternative registration method | ||
+ | * Perform simple aggregations to hide individual subprocesses/workers | ||
+ | * Publish aggregated metrics as standard Prometheus /metrics | ||
+ | * Have a minimal system footprint | ||
+ | |||
See blueprint: https://blueprints.launchpad.net/monasca/+spec/monasca-sidecar | See blueprint: https://blueprints.launchpad.net/monasca/+spec/monasca-sidecar | ||
+ | |||
+ | See PoC: https://github.com/timothyb89/monasca-sidecar |
Revision as of 22:47, 24 January 2017
Contents
Self-instrumenting Monasca
Goal: instrument Monasca components in the same way end users might instrument their own applications, with an emphasis on usability in Kubernetes environments.
Example Metrics
- API latency per endpoint
- API batch insert time, per message
- API metric insert rate
- Persister metric read and write rates
- Kafka partition lag
- Influx per-metric write latency
Options
monasca-statsd
Pros:
- Already part of Monasca: https://github.com/openstack/monasca-statsd
- Good developer API
Cons:
- statsd dialects are not compatible between backends, e.g. monasca, prometheus, datadog, statsd proper
- i.e., language-specific libraries are needed. The only Monasca-compatible library is for Python (monasca-statsd)
- In a k8s environment, a sidecar would need to run with every pod being instrumented
- Requires authenticated write to Monasca, i.e. configuration required
prometheus
Pros:
- Simple data format: easy to write simple metrics endpoint from scratch in any language
- Can take advantage of existing client libraries for most languages
- Can be scraped with zero configuration or authentication (in k8s environments)
- Can easily inspect the current application state at runtime (just browse to /metrics)
Cons:
- May require sidecar in certain situations, new code needed (see below)
See blueprint: https://blueprints.launchpad.net/monasca/+spec/prometheus-instrumentation
Sidecar
Prometheus monitoring with multiprocessing apps, like in the Python implementations of monasca-api and monasca-persister, leads to issues:
- Multiprocessing support in the python prometheus_client is not ideal
- Requires large modifications to gunicorn, and keystone middleware, plus custom Falcon endpoints...
- Overloads '/metrics' endpoint (api)
To work around these limitations, a sidecar server would:
- Ingest metrics from application processes using language-native Prometheus clients
- Clients would use either a small client plugin (supported in the Python client) or alternative registration method
- Perform simple aggregations to hide individual subprocesses/workers
- Publish aggregated metrics as standard Prometheus /metrics
- Have a minimal system footprint
See blueprint: https://blueprints.launchpad.net/monasca/+spec/monasca-sidecar