Compared to metering data, the monitoring metrics need to be collected in a more frequent fashion and the amount of data is way bigger. Does the monitoring metrics necessarily need to go through the message bus as metering data does. Or the option worth being considered is to write them directly to data store.
The current metering pipeline
--- - name: meter_pipeline interval: 60 counters: - "*" transformers: publishers: - meter_publisher
Alternative for metrics pipeline
--- - name: metrics_pipeline interval: 5 // more frequent polling might be needed by monitoring counters: - "*" transformers: publishers: - metrics_publisher // metrics publisher might directly write to backend storage (avoid MQ) - metrics_notifier(publisher) // inline pre-processing metrics notification
Direct write to data store
Storage backend has record_metering_message() to store the metering message(converted from counter) to backend storage via RPC(message queue). Another record_monitoring_metrics() method might just store metrics directly to the storage back end.
Transformer might be used to do some inline pre-processing before the metrics get stored in data store. This allows some critical metrics or calculated metrics being notified in a real-time manner before persistence.
"real-time single metric", when it hits some thresholds or matches some specific values, the notifications can be triggered immediately. eg. The capacity reaches the high-end limit.
"calculated metric", some metrics might sequentially be kept in memory by transformers (similar to AccumulatorTransformer) for some in-line calculation. When the PublishContext exits, the calculated result might get flushed to data store as new type of metric. If it meets some criteria and it can trigger some notifications too. eg. a few drastically increasing storage metrics might indicate a sharp usage rise in a short period of time.
Post-processing, analytics and mapred tasks might be carried out periodically against data store directly or via monitoring API, this is more for time-consuming analytics tasks.