Difference between revisions of "Monasca"
(→Technologies) |
m |
||
| (151 intermediate revisions by 14 users not shown) | |||
| Line 1: | Line 1: | ||
| − | == | + | == Overview == |
| − | + | Monasca is a open-source multi-tenant, highly scalable, performant, fault-tolerant monitoring-as-a-service solution that integrates with OpenStack. It uses a REST API for high-speed metrics processing and querying and has a streaming alarm engine and notification engine. | |
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | + | [[File:OpenStack_Project_Monasca_vertical.png|thumb|right]] | |
| − | |||
| − | |||
| − | + | ==== Contribution ==== | |
| − | * | + | * [https://storyboard.openstack.org/#!/project_group/59 Project Group on StoryBoard] |
| − | * | + | * [https://storyboard.openstack.org/#!/board/111 Important activities in Stein release] |
| − | * | + | * [https://storyboard.openstack.org/#!/board/141 Important activities in Train release] |
| − | * | + | * [https://docs.openstack.org/monasca-api/latest/contributor/index.html Contribution Guidelines] |
| − | |||
| − | * | + | ==== Communication and Meetings ==== |
| − | ** | + | * IRC: #openstack-monasca on OFTC |
| − | ** | + | * Weekly Meetings: |
| + | ** http://eavesdrop.openstack.org/#Monasca_Team_Meeting | ||
| + | ** [[Meetings/Monasca]] | ||
| + | * Use [Monasca] tag on [http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-discuss openstack-discuss mailing list]. | ||
| − | * | + | ===== PTG meetings ===== |
| + | * [[MonascaSteinPTG|Stein PTG Summary (Denver)]] | ||
| + | * [[MonascaTrainPTG|Train PTG Summary (Denver)]] | ||
| + | * [https://etherpad.opendev.org/p/monasca-planning-ussuri Ussuri planning meeting] | ||
| + | * [https://etherpad.opendev.org/p/monasca-ptg-victoria Victoria PTG meeting] | ||
| − | + | === Documentation === | |
| − | + | : Monasca API (and links to other documents): https://docs.openstack.org/monasca-api/latest/ | |
| + | : Monasca command line interface: https://docs.openstack.org/python-monascaclient/latest/ | ||
| + | : Monasca API Specification: https://github.com/openstack/monasca-api/blob/master/docs/monasca-api-spec.md | ||
| + | : Agent Documentation: https://github.com/openstack/monasca-agent | ||
| − | + | ==== Presentations ==== | |
| − | + | There have been many interesting and excellent presentations on Monasca, many given at the OpenStack or OpenInfra Summits. | |
| + | : [[Monasca/Presentations]] (updated with Denver 2019 links) | ||
| − | |||
| − | |||
| − | + | ==== Repositories ==== | |
| − | + | : '''Code''' | |
| − | + | :: Core: https://git.openstack.org/cgit/?q=monasca (old git link) | |
| + | :: API: https://opendev.org/openstack/monasca-api | ||
| − | + | : '''Deployment''' | |
| − | + | : The following repositories are available for deploying Monasca: | |
| + | :: Docker: https://github.com/monasca/monasca-docker | ||
| + | :: Kubernetes: https://github.com/monasca/monasca-helm | ||
| + | :: Ansible: https://github.com/search?utf8=%E2%9C%93&q=ansible-monasca | ||
| + | :: Puppet: https://git.openstack.org/openstack/puppet-monasca | ||
| − | |||
| − | |||
| − | + | ==== Bugs ==== | |
| − | |||
| − | + | ===== All open bugs ===== | |
| + | : https://storyboard.openstack.org/#!/worklist/213 | ||
| + | |||
| + | ===== In Stein triaged bugs ===== | ||
| + | : https://storyboard.openstack.org/#!/worklist/467 | ||
| + | |||
| + | ===== Bug-fixing progress ===== | ||
| + | : https://storyboard.openstack.org/#!/board/114 | ||
| + | |||
| + | |||
| + | ==== Requirements ==== | ||
| + | [[Monasca/Requirements]] from design in 2015. | ||
| + | |||
| + | See also https://opendev.org/openstack/monasca-specs/ | ||
== Features == | == Features == | ||
| Line 67: | Line 83: | ||
* Compound alarms described using a simple expressive grammar composed of alarm sub-expressions and logical operators. | * Compound alarms described using a simple expressive grammar composed of alarm sub-expressions and logical operators. | ||
| − | * Monitoring agent that supports a number of built-in system and service checks and also supports Nagios checks and statsd. | + | * Monitoring agent that supports a number of built-in system and service checks and also supports Nagios checks and statsd, and support for scraping endpoints as Prometheus does. |
* Open-source monitoring solution built on open-source technologies. | * Open-source monitoring solution built on open-source technologies. | ||
| + | |||
| + | |||
| + | Monasca API can be integrated with other reporting, visualization, or billing systems, such as [https://wiki.openstack.org/wiki/CloudKitty CloudKitty] or Grafana. | ||
| + | |||
| + | Monasca community is also involved with the [https://wiki.openstack.org/wiki/Self-healing_SIG Self Healing] and [https://wiki.openstack.org/wiki/Auto-scaling_SIG Auto-Scaling] OpenStack SIGs. | ||
| + | |||
| + | === Comparisons to alternatives === | ||
| + | [[Monasca/Comparison]] (to be written) | ||
== Architecture == | == Architecture == | ||
| Line 77: | Line 101: | ||
** Metrics: Store and query massive amounts of metrics in real-time. | ** Metrics: Store and query massive amounts of metrics in real-time. | ||
** Statistics: Query statistics for metrics. | ** Statistics: Query statistics for metrics. | ||
| − | ** | + | ** Alarm Definitions: Create, update, query and delete alarm definitions. |
| + | ** Alarms: Query and delete the alarm history. | ||
***Simple expressive grammar for creating compound alarms composed of alarm subexpressions and logical operators. | ***Simple expressive grammar for creating compound alarms composed of alarm subexpressions and logical operators. | ||
***Alarm severities can be associated with alarms. | ***Alarm severities can be associated with alarms. | ||
***The complete alarm state transition history is stored and queryable which allows for subsequent root cause analysis (RCA) or advanced analytics. | ***The complete alarm state transition history is stored and queryable which allows for subsequent root cause analysis (RCA) or advanced analytics. | ||
**Notification Methods: Create and delete notification methods and associate them with alarms, such as email. Supports the ability to notify users directly via email when an alarm state transitions occur. | **Notification Methods: Create and delete notification methods and associate them with alarms, such as email. Supports the ability to notify users directly via email when an alarm state transitions occur. | ||
| − | * Persister (monasca-persister): Consumes metrics and alarm state transitions from the MessageQ and stores them in the Metrics and Alarms database. | + | ** The Monasca API has both Java and Python implementations avaialble. |
| − | * Transform and Aggregation Engine: Transform metric names and values, such as delta or time-based derivative calculations, and creates new metrics that are published to the Message Queue. The Transform Engine is not available yet. | + | * Persister (monasca-persister): Consumes metrics and alarm state transitions from the MessageQ and stores them in the Metrics and Alarms database. |
| − | * Anomaly and Prediction Engine: Evaluates prediction and anomalies and generates predicted metrics as well as anomaly likelihood and anomaly scores. | + | ** The Persister has both Java and Python implementations. |
| + | * Transform and Aggregation Engine (monasca-transform): Transform metric names and values, such as delta or time-based derivative calculations, and creates new metrics that are published to the Message Queue. The Transform Engine is not available yet. | ||
| + | * Anomaly and Prediction Engine: Evaluates prediction and anomalies and generates predicted metrics as well as anomaly likelihood and anomaly scores. The Anomaly and Prediction Engine is currently in a prototype status. | ||
* Threshold Engine (monasca-thresh): Computes thresholds on metrics and publishes alarms to the MessageQ when exceeded. Based on Apache Storm a free and open distributed real-time computation system. | * Threshold Engine (monasca-thresh): Computes thresholds on metrics and publishes alarms to the MessageQ when exceeded. Based on Apache Storm a free and open distributed real-time computation system. | ||
* Notification Engine (monasca-notification): Consumes alarm state transition messages from the MessageQ and sends notifications, such as emails for alarms. The Notification Engine is Python based. | * Notification Engine (monasca-notification): Consumes alarm state transition messages from the MessageQ and sends notifications, such as emails for alarms. The Notification Engine is Python based. | ||
| + | * Analytics Engine (monasca-analytics): Consumes alarm state transisitions and metrics from the MessageQ and does anomaly detection and alarm clustering/correlation. | ||
* Message Queue: A third-party component that primarily receives published metrics from the Monitoring API and alarm state transition messages from the Threshold Engine that are consumed by other components, such as the Persister and Notification Engine. The Message Queue is also used to publish and consume other events in the system. Currently, a Kafka based MessageQ is supported. Kafka is a high performance, distributed, fault-tolerant, and scalable message queue with durability built-in. We will look at other alternatives, such as RabbitMQ and in-fact in our previous implementation RabbitMQ was supported, but due to performance, scale, durability and high-availability limitiations with RabbitMQ we have moved to Kafka. | * Message Queue: A third-party component that primarily receives published metrics from the Monitoring API and alarm state transition messages from the Threshold Engine that are consumed by other components, such as the Persister and Notification Engine. The Message Queue is also used to publish and consume other events in the system. Currently, a Kafka based MessageQ is supported. Kafka is a high performance, distributed, fault-tolerant, and scalable message queue with durability built-in. We will look at other alternatives, such as RabbitMQ and in-fact in our previous implementation RabbitMQ was supported, but due to performance, scale, durability and high-availability limitiations with RabbitMQ we have moved to Kafka. | ||
| − | * Metrics and Alarms Database: A third-party component that primarily stores metrics and the alarm state history. Currently, Vertica | + | * Metrics and Alarms Database: A third-party component that primarily stores metrics and the alarm state history. Currently, Vertica, InfluxDB, and Cassandra are supported. |
| − | * Config Database: A third-party component that stores a lot of the configuration and other information in the system. Currently, MySQL is supported. | + | * Config Database: A third-party component that stores a lot of the configuration and other information in the system. Currently, MySQL is supported. Support for PostgreSQL is in progress. |
* Monitoring Client (python-monascaclient): A Python command line client and library that communicates and controls the Monitoring API. The Monitoring Client was written using the OpenStack Heat Python client as a framework. The Monitoring Client also has a Python library, "monascaclient" similar to the other OpenStack clients, that can be used to quickly build additional capabilities. The Monitoring Client library is used by the Monitoring UI, Ceilometer publisher, and other components. | * Monitoring Client (python-monascaclient): A Python command line client and library that communicates and controls the Monitoring API. The Monitoring Client was written using the OpenStack Heat Python client as a framework. The Monitoring Client also has a Python library, "monascaclient" similar to the other OpenStack clients, that can be used to quickly build additional capabilities. The Monitoring Client library is used by the Monitoring UI, Ceilometer publisher, and other components. | ||
| − | |||
* Monitoring UI: A Horizon dashboard for visualizing the overall health and status of an OpenStack cloud. | * Monitoring UI: A Horizon dashboard for visualizing the overall health and status of an OpenStack cloud. | ||
* Ceilometer publisher: A multi-publisher plugin for Ceilometer, not shown, that converts and publishes samples to the Monitoring API. | * Ceilometer publisher: A multi-publisher plugin for Ceilometer, not shown, that converts and publishes samples to the Monitoring API. | ||
Most of the components are described in their respective repositories. However, there aren't any repositories for the third-party components used, so we describe some of the relevant details here. | Most of the components are described in their respective repositories. However, there aren't any repositories for the third-party components used, so we describe some of the relevant details here. | ||
| − | + | Further historical context for the architecture can be found at [[Monasca/Architecture Details]] | |
| − | |||
| − | === | + | === Further Reading === |
| − | There are several messages that are published and consumed by various components in | + | * [[Monasca/Message Schema]] |
| − | + | * Message Schema: A distributed, performant, scalable, HA message queue for distributing metrics, alarms and events in the monitoring system. Currently, based on Kafka. | |
| − | | | + | * Messages: There are several messages that are published and consumed by various components in Monasca via the MessageQ. See [[Monasca/Message Schema|Message Schema]]. |
| − | + | * Metrics and Alarms Database: see [[Monasca/Architecture Details]] | |
| − | + | * Config Database: see [[Monasca/Architecture Details]] | |
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | + | * Events: Support for real-time event stream processing in Monasca is in progress. For more details see the link at, [[Monasca/Events]]. | |
| − | + | * Logging: Support for logging in Monasca is under discussion. For more details see the link at, [[Monasca/Logging]]. | |
| − | + | * Transform and Aggregation Engine: For more details see the link at, [[Monasca/Transform]]. | |
| − | * | + | * Analytics: Support for anomaly detection and alarm clustering/correlation is in progress. For more details see the link at, [[Monasca/Analytics]]. |
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | + | * Monitoring: Enablement and usage for monitoring the status of Monasca is under discussion. For more details see the link at, [[Monasca/Monitoring_Of_Monasca]] | |
| − | |||
| − | + | * UI/UX Support: Adding more support for common UI/UX queries is under discussion. For more details see the link at, [[Monasca/UI_UX_Support]] | |
| − | * | + | * Post Metric Sequence: see [[Monasca/Architecture Details]] |
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | + | * Alarm Managers: see [[Monasca/Architecture Details]] and the official documentation at docs.openstack.org | |
| − | + | == Keystone Requirements == | |
| + | Monasca relies on keystone for running and there are requirements about which keystone configuration must exist. | ||
| − | + | * The endpoint for the api must be registered in keystone as the 'monasca' service. | |
| + | * The api must have an admin token to use in verifying the keystone tokens it receives. | ||
| + | * For each project which uses Monasca two users must exist, one will be in the 'monasca-agent' role and be used by the monasca-agent's running on machines. The other should not be in that role and can be used logging into the UI, using the CLI or for direct queries against the API. | ||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
= Development Environment = | = Development Environment = | ||
| − | * | + | * Monasca DevStack Plugin: |
| + | ** DevStack is the primary developmement environment for OpenStack. | ||
| + | ***See http://docs.openstack.org/developer/devstack/ | ||
| + | ** The Monasca DevStack plugin installs the Monasca Service, Agent, Horizon Monitoring Panel, and Grafana | ||
| + | ** README at, https://github.com/openstack/monasca-api/tree/master/devstack | ||
| + | ** Best way to get started is to use Vagrant with the Vagrantfile at, https://github.com/openstack/monasca-api/blob/master/devstack/Vagrantfile. | ||
| − | * | + | * Project and Bug tracking |
| + | ** Monasca on Storyboard https://storyboard.openstack.org/#!/project/list?q=monasca | ||
| + | ** Monasca on LaunchPad. https://launchpad.net/monasca (history only, Storyboard is the official tracking since 2017) | ||
| − | * Monasca | + | * Monasca projects source code. https://git.openstack.org/cgit/?q=monasca |
| − | |||
= Coding Standards = | = Coding Standards = | ||
* Python: All Python code conforms to the OpenStack standards at, http://docs.openstack.org/developer/hacking/. | * Python: All Python code conforms to the OpenStack standards at, http://docs.openstack.org/developer/hacking/. | ||
| + | ** Note, all components in Monasca, except for the Threshold Engine, have been ported to Python. | ||
| − | * Java: OpenStack does not have any Java coding standards. We've adopted the Google Java Style at, https://google | + | * Java: Several of the Monasca components are available as Java. OpenStack does not have any Java coding standards. We've adopted the Google Java Style at, https://google.github.io/styleguide/javaguide.html. |
** The standard says either 80 or 100 length lines. We've adopted 100. | ** The standard says either 80 or 100 length lines. We've adopted 100. | ||
| − | |||
| − | |||
= Technologies = | = Technologies = | ||
| − | + | Monasca uses a number of third-party technologies: | |
| − | * | + | * Internal Processing and Middleware |
| + | ** Apache Kafka (http://kafka.apache.org): Apache Kafka is publish-subscribe messaging rethought as a distributed commit log. Kafka is a highly performant, distributed, fault-tolerant, and scalable message queue with durability built-in. | ||
| + | ** Apache Storm (http://storm.incubator.apache.org/): Apache Storm is a free and open source distributed realtime computation system. Storm makes it easy to reliably process unbounded streams of data, doing for realtime processing what Hadoop did for batch processing. | ||
| + | ** ZooKeeper (http://zookeeper.apache.org/): Used by Kafka and Storm. | ||
| + | ** Apache Spark: Used by Monasca Transform as an aggregation engine. | ||
| − | * | + | * Configuration database: |
| − | + | ** MySQL: MySQL is supported as a Config Database. | |
| − | * | + | ** PostgreSQL: Support for POSTgres, via Hibernate and SQLAlchemy, for the Config Database. |
| − | |||
| − | |||
| − | |||
| − | |||
* Vagrant (http://www.vagrantup.com/): Vagrant provides easy to configure, reproducible, and portable work environments built on top of industry-standard technology and controlled by a single consistent workflow to help maximize the productivity and flexibility of you and your team. | * Vagrant (http://www.vagrantup.com/): Vagrant provides easy to configure, reproducible, and portable work environments built on top of industry-standard technology and controlled by a single consistent workflow to help maximize the productivity and flexibility of you and your team. | ||
| Line 238: | Line 199: | ||
* Dropwizard (https://dropwizard.github.io/dropwizard/): Dropwizard pulls together stable, mature libraries from the Java ecosystem into a simple, light-weight package that lets you focus on getting things done. Dropwizard has out-of-the-box support for sophisticated configuration, application metrics, logging, operational tools, and much more, allowing you and your team to ship a production-quality web service in the shortest time possible. | * Dropwizard (https://dropwizard.github.io/dropwizard/): Dropwizard pulls together stable, mature libraries from the Java ecosystem into a simple, light-weight package that lets you focus on getting things done. Dropwizard has out-of-the-box support for sophisticated configuration, application metrics, logging, operational tools, and much more, allowing you and your team to ship a production-quality web service in the shortest time possible. | ||
| − | * InfluxDB (http://influxdb.com/): An open-source distributed time series database with no external dependencies. | + | * Time Series Database: |
| + | ** InfluxDB (http://influxdb.com/): An open-source distributed time series database with no external dependencies. InfluxDB is supported for the Metrics Database. | ||
| + | ** Vertica (http://www.vertica.com): A commercial Enterprise class SQL analytics database that is highly scalable. It offers built-in automatic high-availability and excels at in-database analytics and compressing and storing massive amounts of data. A free community version of Vertica is available that can store up to 1 TB of data with no time-limit is available at, https://my.vertica.com/community/. Vertica is supported for the Metrics Database, though no longer commonly used. | ||
| + | ** Cassandra(https://cassandra.apache.org): Cassandra is supported for the Metrics Database. | ||
| + | |||
= License = | = License = | ||
| − | Copyright (c) 2014 Hewlett-Packard Development Company, L.P. | + | Copyright (c) 2014, 2015 Hewlett-Packard Development Company, L.P. |
| + | |||
| + | (C) Copyright 2019 SUSE LLC | ||
Licensed under the Apache License, Version 2.0 (the "License"); | Licensed under the Apache License, Version 2.0 (the "License"); | ||
| Line 256: | Line 223: | ||
See the License for the specific language governing permissions and | See the License for the specific language governing permissions and | ||
limitations under the License. | limitations under the License. | ||
| + | |||
| + | |||
| + | [[File:yklogo.png|200px|thumb|left|Monasca uses YourKit Profiler for Java development]] | ||
| + | |||
| + | <br /> | ||
| + | <br /> | ||
| + | <br /> | ||
| + | <br /> | ||
| + | |||
| + | [https://www.yourkit.com/ Visit YourKit website for more information] | ||
Latest revision as of 13:15, 1 June 2021
Contents
Overview
Monasca is a open-source multi-tenant, highly scalable, performant, fault-tolerant monitoring-as-a-service solution that integrates with OpenStack. It uses a REST API for high-speed metrics processing and querying and has a streaming alarm engine and notification engine.
Contribution
- Project Group on StoryBoard
- Important activities in Stein release
- Important activities in Train release
- Contribution Guidelines
Communication and Meetings
- IRC: #openstack-monasca on OFTC
- Weekly Meetings:
- Use [Monasca] tag on openstack-discuss mailing list.
PTG meetings
Documentation
- Monasca API (and links to other documents): https://docs.openstack.org/monasca-api/latest/
- Monasca command line interface: https://docs.openstack.org/python-monascaclient/latest/
- Monasca API Specification: https://github.com/openstack/monasca-api/blob/master/docs/monasca-api-spec.md
- Agent Documentation: https://github.com/openstack/monasca-agent
Presentations
There have been many interesting and excellent presentations on Monasca, many given at the OpenStack or OpenInfra Summits.
- Monasca/Presentations (updated with Denver 2019 links)
Repositories
- Code
- Core: https://git.openstack.org/cgit/?q=monasca (old git link)
- API: https://opendev.org/openstack/monasca-api
- Deployment
- The following repositories are available for deploying Monasca:
Bugs
All open bugs
In Stein triaged bugs
Bug-fixing progress
Requirements
Monasca/Requirements from design in 2015.
See also https://opendev.org/openstack/monasca-specs/
Features
This section describes the overall features.
- A highly performant, scalable, reliable and fault-tolerant Monitoring as a Service (MONaaS) solution that scales to service provider metrics levels of metrics throughput. Performance, scalability and high-availability have been designed in from the start. Can process 100s of thousands of metrics/sec as well as offer data retention periods of greater than a year with no data loss while still processing interactive queries.
- Rest API for storing and querying metrics and historical information. Most monitoring solution use special transports and protocols, such as CollectD or NSCA (Nagios). In our solution, http is the only protocol used. This simplifies the overall design and also allows for a much richer way of describing the data via dimensions.
- Multi-tenant and authenticated. Metrics are submitted and authenticated using Keystone and stored associated with a tenant ID.
- Metrics defined using a set of (key, value) pairs called dimensions.
- Real-time thresholding and alarming on metrics.
- Compound alarms described using a simple expressive grammar composed of alarm sub-expressions and logical operators.
- Monitoring agent that supports a number of built-in system and service checks and also supports Nagios checks and statsd, and support for scraping endpoints as Prometheus does.
- Open-source monitoring solution built on open-source technologies.
Monasca API can be integrated with other reporting, visualization, or billing systems, such as CloudKitty or Grafana.
Monasca community is also involved with the Self Healing and Auto-Scaling OpenStack SIGs.
Comparisons to alternatives
Monasca/Comparison (to be written)
Architecture
- Monitoring Agent (monasca-agent): A modern Python based monitoring agent that consists of several sub-components and supports system metrics, such as cpu utilization and available memory, Nagios plugins, statsd and many built-in checks for services such as MySQL, RabbitMQ, and many others.
- Monitoring API (monasca-api): A well-defined and documented RESTful API for monitoring that is primarily focused on the following concepts and areas:
- Metrics: Store and query massive amounts of metrics in real-time.
- Statistics: Query statistics for metrics.
- Alarm Definitions: Create, update, query and delete alarm definitions.
- Alarms: Query and delete the alarm history.
- Simple expressive grammar for creating compound alarms composed of alarm subexpressions and logical operators.
- Alarm severities can be associated with alarms.
- The complete alarm state transition history is stored and queryable which allows for subsequent root cause analysis (RCA) or advanced analytics.
- Notification Methods: Create and delete notification methods and associate them with alarms, such as email. Supports the ability to notify users directly via email when an alarm state transitions occur.
- The Monasca API has both Java and Python implementations avaialble.
- Persister (monasca-persister): Consumes metrics and alarm state transitions from the MessageQ and stores them in the Metrics and Alarms database.
- The Persister has both Java and Python implementations.
- Transform and Aggregation Engine (monasca-transform): Transform metric names and values, such as delta or time-based derivative calculations, and creates new metrics that are published to the Message Queue. The Transform Engine is not available yet.
- Anomaly and Prediction Engine: Evaluates prediction and anomalies and generates predicted metrics as well as anomaly likelihood and anomaly scores. The Anomaly and Prediction Engine is currently in a prototype status.
- Threshold Engine (monasca-thresh): Computes thresholds on metrics and publishes alarms to the MessageQ when exceeded. Based on Apache Storm a free and open distributed real-time computation system.
- Notification Engine (monasca-notification): Consumes alarm state transition messages from the MessageQ and sends notifications, such as emails for alarms. The Notification Engine is Python based.
- Analytics Engine (monasca-analytics): Consumes alarm state transisitions and metrics from the MessageQ and does anomaly detection and alarm clustering/correlation.
- Message Queue: A third-party component that primarily receives published metrics from the Monitoring API and alarm state transition messages from the Threshold Engine that are consumed by other components, such as the Persister and Notification Engine. The Message Queue is also used to publish and consume other events in the system. Currently, a Kafka based MessageQ is supported. Kafka is a high performance, distributed, fault-tolerant, and scalable message queue with durability built-in. We will look at other alternatives, such as RabbitMQ and in-fact in our previous implementation RabbitMQ was supported, but due to performance, scale, durability and high-availability limitiations with RabbitMQ we have moved to Kafka.
- Metrics and Alarms Database: A third-party component that primarily stores metrics and the alarm state history. Currently, Vertica, InfluxDB, and Cassandra are supported.
- Config Database: A third-party component that stores a lot of the configuration and other information in the system. Currently, MySQL is supported. Support for PostgreSQL is in progress.
- Monitoring Client (python-monascaclient): A Python command line client and library that communicates and controls the Monitoring API. The Monitoring Client was written using the OpenStack Heat Python client as a framework. The Monitoring Client also has a Python library, "monascaclient" similar to the other OpenStack clients, that can be used to quickly build additional capabilities. The Monitoring Client library is used by the Monitoring UI, Ceilometer publisher, and other components.
- Monitoring UI: A Horizon dashboard for visualizing the overall health and status of an OpenStack cloud.
- Ceilometer publisher: A multi-publisher plugin for Ceilometer, not shown, that converts and publishes samples to the Monitoring API.
Most of the components are described in their respective repositories. However, there aren't any repositories for the third-party components used, so we describe some of the relevant details here.
Further historical context for the architecture can be found at Monasca/Architecture Details
Further Reading
- Monasca/Message Schema
- Message Schema: A distributed, performant, scalable, HA message queue for distributing metrics, alarms and events in the monitoring system. Currently, based on Kafka.
- Messages: There are several messages that are published and consumed by various components in Monasca via the MessageQ. See Message Schema.
- Metrics and Alarms Database: see Monasca/Architecture Details
- Config Database: see Monasca/Architecture Details
- Events: Support for real-time event stream processing in Monasca is in progress. For more details see the link at, Monasca/Events.
- Logging: Support for logging in Monasca is under discussion. For more details see the link at, Monasca/Logging.
- Transform and Aggregation Engine: For more details see the link at, Monasca/Transform.
- Analytics: Support for anomaly detection and alarm clustering/correlation is in progress. For more details see the link at, Monasca/Analytics.
- Monitoring: Enablement and usage for monitoring the status of Monasca is under discussion. For more details see the link at, Monasca/Monitoring_Of_Monasca
- UI/UX Support: Adding more support for common UI/UX queries is under discussion. For more details see the link at, Monasca/UI_UX_Support
- Post Metric Sequence: see Monasca/Architecture Details
- Alarm Managers: see Monasca/Architecture Details and the official documentation at docs.openstack.org
Keystone Requirements
Monasca relies on keystone for running and there are requirements about which keystone configuration must exist.
- The endpoint for the api must be registered in keystone as the 'monasca' service.
- The api must have an admin token to use in verifying the keystone tokens it receives.
- For each project which uses Monasca two users must exist, one will be in the 'monasca-agent' role and be used by the monasca-agent's running on machines. The other should not be in that role and can be used logging into the UI, using the CLI or for direct queries against the API.
Development Environment
- Monasca DevStack Plugin:
- DevStack is the primary developmement environment for OpenStack.
- The Monasca DevStack plugin installs the Monasca Service, Agent, Horizon Monitoring Panel, and Grafana
- README at, https://github.com/openstack/monasca-api/tree/master/devstack
- Best way to get started is to use Vagrant with the Vagrantfile at, https://github.com/openstack/monasca-api/blob/master/devstack/Vagrantfile.
- Project and Bug tracking
- Monasca on Storyboard https://storyboard.openstack.org/#!/project/list?q=monasca
- Monasca on LaunchPad. https://launchpad.net/monasca (history only, Storyboard is the official tracking since 2017)
- Monasca projects source code. https://git.openstack.org/cgit/?q=monasca
Coding Standards
- Python: All Python code conforms to the OpenStack standards at, http://docs.openstack.org/developer/hacking/.
- Note, all components in Monasca, except for the Threshold Engine, have been ported to Python.
- Java: Several of the Monasca components are available as Java. OpenStack does not have any Java coding standards. We've adopted the Google Java Style at, https://google.github.io/styleguide/javaguide.html.
- The standard says either 80 or 100 length lines. We've adopted 100.
Technologies
Monasca uses a number of third-party technologies:
- Internal Processing and Middleware
- Apache Kafka (http://kafka.apache.org): Apache Kafka is publish-subscribe messaging rethought as a distributed commit log. Kafka is a highly performant, distributed, fault-tolerant, and scalable message queue with durability built-in.
- Apache Storm (http://storm.incubator.apache.org/): Apache Storm is a free and open source distributed realtime computation system. Storm makes it easy to reliably process unbounded streams of data, doing for realtime processing what Hadoop did for batch processing.
- ZooKeeper (http://zookeeper.apache.org/): Used by Kafka and Storm.
- Apache Spark: Used by Monasca Transform as an aggregation engine.
- Configuration database:
- MySQL: MySQL is supported as a Config Database.
- PostgreSQL: Support for POSTgres, via Hibernate and SQLAlchemy, for the Config Database.
- Vagrant (http://www.vagrantup.com/): Vagrant provides easy to configure, reproducible, and portable work environments built on top of industry-standard technology and controlled by a single consistent workflow to help maximize the productivity and flexibility of you and your team.
- Dropwizard (https://dropwizard.github.io/dropwizard/): Dropwizard pulls together stable, mature libraries from the Java ecosystem into a simple, light-weight package that lets you focus on getting things done. Dropwizard has out-of-the-box support for sophisticated configuration, application metrics, logging, operational tools, and much more, allowing you and your team to ship a production-quality web service in the shortest time possible.
- Time Series Database:
- InfluxDB (http://influxdb.com/): An open-source distributed time series database with no external dependencies. InfluxDB is supported for the Metrics Database.
- Vertica (http://www.vertica.com): A commercial Enterprise class SQL analytics database that is highly scalable. It offers built-in automatic high-availability and excels at in-database analytics and compressing and storing massive amounts of data. A free community version of Vertica is available that can store up to 1 TB of data with no time-limit is available at, https://my.vertica.com/community/. Vertica is supported for the Metrics Database, though no longer commonly used.
- Cassandra(https://cassandra.apache.org): Cassandra is supported for the Metrics Database.
License
Copyright (c) 2014, 2015 Hewlett-Packard Development Company, L.P.
(C) Copyright 2019 SUSE LLC
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
