Difference between revisions of "MagnetoDB/specs/requestmetrics"
Charles Wang (talk | contribs) |
Charles Wang (talk | contribs) (→Proposed Change) |
||
Line 20: | Line 20: | ||
=== Proposed Change === | === Proposed Change === | ||
− | Request metrics can be reported to StatsD | + | Request metrics can be reported to StatsD using a sample rate for API calls. Request metrics are either counters or timing data (in units of milliseconds). |
<br /> | <br /> | ||
Line 26: | Line 26: | ||
<br /> | <br /> | ||
− | Middleware seems a natural place to collect the request metrics data. | + | Middleware seems a natural place to collect the request metrics data. However, if admin user needs visibility into requests to each API endpoints, we will need to capture metrics before/after each call to the API endpoints. |
<br /> | <br /> | ||
− | We propose to introduce a new middleware to all API node services, including | + | We propose to introduce a new middleware to all API node services, including REST API and streaming. |
<br /> | <br /> | ||
− | Initially we will focus on request measurements on API | + | Initially we will focus on request measurements on API node wide metrics, API endpoints and WSGI processing delays on each API node. Later on, we will expand to cover Cassandra request metrics. |
<br /> | <br /> | ||
− | Each API | + | Each REST API call made to MagnetoDB will have the following request metrics: |
<br /> | <br /> | ||
Line 47: | Line 47: | ||
* response send bytes | * response send bytes | ||
<br /> | <br /> | ||
+ | |||
+ | It can be further broken down to API endpoints. | ||
+ | |||
+ | |||
WSGI will have the following metrics: | WSGI will have the following metrics: | ||
Line 57: | Line 61: | ||
Cassandra metrics is supported by Cassandra python driver and can be enabled. But metrics data is supported through Scales. | Cassandra metrics is supported by Cassandra python driver and can be enabled. But metrics data is supported through Scales. | ||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− |
Revision as of 00:33, 3 December 2014
Request Real Time Metrics
Real time request metrics including latency/count/etc.
Specification status
Draft
Problem Description
To proactively address MagnetoDB operational issues, admin user needs real time visibility to request metrics data on each API node. Including:
- number of requests
- number of failures
- number of errors
- average latency
- median latency
- minimum latency
- maximum latency
- requests per second
- distribution of request latency for each type of REST API call, such as "50%","66%","75%","80%","90%","95%","98%","99%","100%"
Proposed Change
Request metrics can be reported to StatsD using a sample rate for API calls. Request metrics are either counters or timing data (in units of milliseconds).
StatsD can be utilized to expand timing data to min, max, avg, count, and 90th percentile per timing metric.
Middleware seems a natural place to collect the request metrics data. However, if admin user needs visibility into requests to each API endpoints, we will need to capture metrics before/after each call to the API endpoints.
We propose to introduce a new middleware to all API node services, including REST API and streaming.
Initially we will focus on request measurements on API node wide metrics, API endpoints and WSGI processing delays on each API node. Later on, we will expand to cover Cassandra request metrics.
Each REST API call made to MagnetoDB will have the following request metrics:
- request receive time
- request receive error
- request receive timeout
- request receive bytes
- response send time
- response send error
- response send timeout
- response send bytes
It can be further broken down to API endpoints.
WSGI will have the following metrics:
- backlog
- waits
- request processing time
- response processing time
- dispatch time
Cassandra metrics is supported by Cassandra python driver and can be enabled. But metrics data is supported through Scales.