OpenStack - User contributions [en]

Monasca/Transform

2020-10-16T03:09:40Z

Joseph Davis: Mon Trans Deprecation

= NOTICE: Monasca Transform has been deprecated =

While Monasca Transform has been a useful utility, its user base has declined and engineers for maintenance have been pulled in other directions. To make this clear, the monasca-transform repository has been marked as deprecated and content removed from the Wallaby release. If you still need a release of Monasca Transform, use the Ussuri or older branch.

Reference the commit to clear the content for Wallaby release: https://opendev.org/openstack/monasca-transform/commit/811acd76c9029e175c0a43302386bc7732fcc48e

If you find Monasca Transform to be useful and have a desire to take on maintenance, please post a message to the #openstack-monasca IRC channel, as described on http://eavesdrop.openstack.org/#Monasca_Team_Meeting .

The following content is for historical purposes and describes some of the original motivation and development of Monasca Transform.

== monasca-transform: A transformation and aggregation engine for Monasca ==

Original blueprint - https://blueprints.launchpad.net/monasca/+spec/monasca-transform
Original Proposal - [[Monasca/Transform-proposal]]

* monasca-transform is a data driven aggregation engine which collects, groups and aggregates existing individual Monasca metrics according to business requirements and publishes new transformed (derived) metrics to the Monasca Kafka queue.
* Since the new transformed metrics are published as any other metric in Monasca, alarms can be set and triggered on the transformed metric.

=== Problem description ===

There are several use cases which can be addressed by transforming, aggregating and publishing new metrics in Monasca:
* Aggregate individual metrics
Metrics that are currently available on a per host or per VM basis, for example mem.total_mb or vm.mem.used_mb, might have to be added up for all hosts or VMs so that cloud operator can know the the total memory available across all hosts and total memory being allocated to all VMs for the entire cloud.
* Combine multiple metrics to derive a new metric
There might be a need to derive metrics from one or more metrics, for example cpu.total_logical_cores and cpu.idle_perc are two metrics for a host which can be used to derive a new metric which will give total utilized CPUs. Also these per host utilized CPUs can be further summed up to find the number of CPUs being utilized across the entire cloud.
* Find the rate at which a metric changed over a time range
It might be necessary to calculate the rate at which a metric value is changing over time. For example, to find the rate at which swift disk capacity is increasing over a time range, one would have to find the value of the disk usage at the start of the time interval (say start of the hour) then find the disk usage at the end of the time interval (say end of the hour) in order to determine the rate at which the disk usage increased or decreased during that hour.
* Deriving metrics using complex business logic
There might be a need to apply complex business logic on a set of metrics and derive a new metric. For example consider the periodic heart beat metrics which come from a VM indicating what state the VM was in. One might have to look at how the state of the VM changed over the target time interval and then use some business logic to come up with a new metric which would indicate the amount of time that the VM was in a usable state.

=== How Monasca Transform Operates ===

The monasca-transform component framework fits into the microservices architecture of Monasca. At its core, it collects a set of metrics from Apache Kafka[1] over a configurable time period, then using grouping operations, runs a series of transformation (or aggregation) operations which can consist of simple mathematical operations like sum/max/min/avg or complex operations such as iterating through group using some business rules. The resulting transformed metric or set of transformed metrics are published back to Apache Kafka.

Some design considerations that were essential based on past experience when building a similar aggregation framework to process vast amounts of raw log records from public cloud services:

* Scalable, Fault Tolerant and Highly Available
Processing of metrics has to be scalable since we will have to collect, aggregate and publish vast amounts of metrics. The processing of data should be able to withstand any hardware failures such as CPU, memory or disk errors. The monasca-transform process is designed to be Highly Available. In case the monasca-transform daemon process goes down on a server because of any software or hardware problem the framework will start processing metrics on a another server.
* Data driven transformations
The transformation framework has to be data driven and it should be possible to add new transformations for any new metrics by configuration changes.
* Reusable components and reusable transformation routines
The transformation requirements of various metrics can be similar, so once a transformation routine has been implemented it should be possible to re-use the same transformation components and routines for a different set of metrics.
* Easy to add components
Adding new transformation components should be easy and straightforward.
* Flexible to change and adapt
Over a period of time business requirements and needs can change. To meet those requirements new transformation components will have to be developed and added or existing ones updated. Monasca Transform makes it possible to add and update new components with minimum effort.
* Unit Testing
Though the transformation routines can be complex, there are unit tests for each individual transformation component to be confident of the accuracy of the component.

==== Considered Alternatives ====

To avoid re-inventing the wheel it is desirable to use a framework or tool that provides the ability to process data across a cluster of commodity machines and that features work distribution, fault tolerance and the capability to seamlessly withstand node failures.

Hadoop with MapReduce/Yarn satisfies these requirements but requiring some kind of stable storage, HDFS and relying on temporary files to save intermediate processing states, they can be slow. They also require a number of components to be configured, installed, monitored and maintained across a cluster which is often a challenging and daunting task. Also it's highly likely that other associated tools like Hive (for sql-like processing of data) and Pig (which would faciliate the expression map reduce jobs as high level constructs) would be entailed. Using additional tools adds to the resource utilization burden as well as increasing development time.

Spark allows jobs to be expressed in Python consequently siting well with the openstack toolset. It has demonstrated much higher processing rates than Hadoop based tools yet allows existing domain knowledge of mapreduce type tools to be re-purposed. Spark is gaining momentum and adoption generally and this upward trajectory can be capitalised upon. It also provides support for streaming data, again meaning that we can concentrate on the task at hand rather than plumbing. For these reasons it has been selected to provide the processing framework.

There is a non-OpenStack project for those who want a light weight aggregation engine written in Go - [https://github.com/monasca/monasca-aggregator monasca-aggregator]. monasca-aggregator does not have feature parity with monasca-transform and does not have unit tests in place, but is open under the Apache License 2.0 for anyone who wishes to contribute.

=== Architecture ===

monasca-transform is a data driven data processing framework implemented in python and based on Apache Spark[2]. Apache Spark[2] is a highly scalable, fast, in-memory, fault tolerant and parallel data processing framework. All monasca-transform components are implemented in Python and use Spark's Python API to interact with Spark.

To process data, monasca-transform components make use of Spark's transformation operations like groupByKey(), filter(), map(), reduceByKey() on Spark's Resilient Distributed Datasets (also commonly referred to as RDDs). RDD is Spark's basic abstraction for distributed memory.

monasca-transform uses the Spark Steaming direct API to retrieve Monasca metrics from the Kafka queue. monasca-transform uses Spark Data Frames for processing whenever possible. Spark Data Frames use a new query planner and analyzer called Catalyst and is more efficient and optimized than comparable RDD operations.

Data Frames provide a more SQL-like data manipulation capability which also makes it easy to use from a development point of view. It is possible to convert from a Spark RDD to a Data Frame and vice versa. Also, Spark provides a common equivalent API for manipulating both an RDD as well as a Data Frame.

In addition to Apache Spark, monasca-transform also relies on MySQL database tables to store processing and configuration information:

* processing information: Kafka offsets from Spark batch, used to resume processing after a failure or a restart
* configuration information: Runtime parameters and transformation specifications, used to drive transformation and aggregation of metrics

It is possible to run multiple monasca-transform processes on multiple nodes. Multiple monasca-transform processes will use zookeeper's leader election capability to elect a master. In case a node on which monasca-transform process is running as a master goes down, another monasca-transform process running on an another node will be elected a leader and start processing data.

Here are the components of monasca-transform:

[[File:ArchitectureComponents.jpg]]

For an Enterprise deployment, it is recommended to deploy Spark cluster in standalone cluster mode alongside other Monasca components. In the minimal configuration, say for example in a devstack deployment, both master and worker can be deployed on a single node. Spark can be also deployed on multiple nodes, with multiple masters and workers running on each of the nodes.

=== Logical processing data flow ===

The logical data flow within monasca-transform can be broadly divided into two distinct flows, namely conversion of input metrics into a record store format and transforming of the record store data using series of generic transformation components to derive transformed metric data.

==== Conversion to record store format ====

[[File:LogicalEventBasedFlow1.jpg]]

Conversion of input metrics into the record store format consists of following steps:

'''Identification''': Identify and filter out unnecessary metrics. This is done by comparing input metric names with a list of metrics to be transformed in the configuration datastore. If an input metric is not recognized, the metric is ignored in further processing.

'''Validation and Field Extraction''': Input data is validated by checking for presence of expected fields in the metric JSON (expected fields are retrieved from the configuration datastore). Assuming the format is valid, the relevant fields are extracted.

'''Record Generator''': Extracted data is then used to generate one or more new internal processing metrics and the associated metadata, this is referred to as Record Store Data. Multiple internal metrics can be generated in the case where the input raw metric has to be aggregated in more than one way.

An advantage of converting the input metrics into internal record store data is that the rest of the processing does not depend on the input metric format. In the future if there is a need to transform and aggregate data coming in new format, new code will have to written to convert the new data format into record store, but the rest of the processing pipeline will be unchanged.

==== Generic transformation (aggregation) to derive and publish new metrics ====

[[File:LogicalEventBasedFlow2.jpg]]

This is the second half of the processing data flow. Record store data is routed to the appropriate processing component based on internal metric type and specifications in the configuration datastore. Additional processing is then performed. Processing includes grouping the data in the record store with various user-defined parameters, setting of the final transformed metric name, and setting appropriate dimensions in the new transformed metric. All these operations are driven from configuration parameters. The resulting new transformed metrics are then published back to Apache Kafka.

=== Data model design ===

[[File:DatastoreSchema.jpg]]

pre_transform_specifications and transform_specifications tables are used in conversion of input metrics into record store format and conversion of record store data into final transformed metrics to be published to Apache Kafka.

kafka_offsets table stores the offset information retrieved by Spark streaming direct Kafka API, when it starts processing a batch. Only if the processing data in a batch succeeds, the latest offset information is updated in kafka_offset table. The last saved offset information is used by the Spark driver on initialization to start processing from where it left off.

* pre_transform_specifications table
** event_type: name of the metric to process
** pre_transform_spec: Specification in JSON format to aid in processing of data to record store format. The JSON specification consists of following fields:
*** event_type: name of the metric to
*** event_processing_params: key-value pairs dictionary of parameters to aid in conversion of input metric to record store format. Examples include set_default_region
** intermediate_id_list: list of internal metrics, represented by their identifiers, that the input metric should be converted into. During record store generation script multiple internal metrics will be generated from the same input metric. Each internal metric can be transformed with a different processing pipeline.
** required_raw_fields_list: list of JSON path strings, that will be used during validation to check if they exist and are not empty.
** service_id: service identifier, identifies which service this input metric belongs to.

* transform_specs table
** metric_id: Identifier representing internal processing metric
** transform_spec: Specification in JSON format which defines the processing pipeline and runtime parameters to be used by processing pipeline components. The JSON specification consists of following fields:
*** aggregation_params_map: key-value pairs which represent the processing pipeline and runtime parameters.
*** aggregation_pipeline: JSON representation of processing pipeline specification and any run time parameters used by components in processing pipeline e.g. aggregation_group_by_list
* kafka_offsets table
** topic: Kafka topic for which this offsets are being collected and stored
** until_offset: end of the offset range
** from_offset: start of the offset range
** app_name: name of the application, on behalf of which these offsets are being stored.
** partition: partition identifier

=== Design of generic transformation components ===

Generic transformation components will be reusable components that can be assembled into a complex transformation pipeline. Each generic component will be implemented as a python class and can invoke multiple Spark transformations. Generic components of a same type will implement functions which take standard set of arguments and return a standard Spark RDD. Source, Usage, Setter and Insert are few generic component types.

All usage components will implement a python function called 'usage' which takes 'transform_context' and 'record_store_dataframe' as arguments and returns an 'instance_usage_dataframe'. Similarly all setter components will implement a function called 'setter' which takes 'transform_context' and 'instance_usage_dataframe' as arguments and returns a modified 'instance_usage_dataframe. Component interfaces are standardized by input and output type to allow plug-ability.

Generic transformation components are extensible and new components can be added when necessary. For example if there is a need for some transformation routine to lookup a value by making a REST API call, a new setter component can be developed which implements the standard "setter" function interface. Since the new setter component adheres to the Setter component interface, it is possible to add this new component after or before any previous setter component. Complex transformation pipelines can be built by using series of these pluggable components via the transformation specification.

{| class="wikitable"
|-
! Component Type !! Component Short Name !! Component Description !! Arguments !! Returns
|-
| Usage || fetch_quantity || Groups record_store_data_frame record represented by intermediate metric id by usage_group_by_list, and then performs operation indicated by usage_fetch_operation parameter. || transform_context, record_store_data_frame || instance_usage_data_frame
|-
|   || fetch_quantity_util || Groups record_store_data_frame records for quantity and utilization percentage by usage_group_by_list and usage_fetch operation to find the utilized quantity. || transform_context, record_store_data_frame || instance_usage_data_frame
|-
| Setter || rollup_quantity || Groups instance_usage_data_frame data, by setter_group_by_list parameter and then performs operation as indicated by setter_fetch_operation parameter || transform_context, instance_usage_data_frame || instance_usage_data_frame
|-
|   || set_aggregated_metric_name || Iterate through instance_usage_data_frame data and set metric name, indicated by aggregation_uom parameter || transform_context, instance_usage_data_frame || instance_usage_data_frame
|-
| Insert || insert_data || Prepares final final transformed metric json by setting dimensions as indicated by setter_dimension_list parameter and publishes the data to Kafka "metrics" topic || transform_context, instance_usage_data_frame || instance_usage_data_frame
|}
Generic transform components are easy to write and maintain. Besides being re-usable, these generic transform components can be combined to create complex transformation routines or pipelines. Unit tests in python can be written to test each component individually and also to test a complex transformation routines.

=== Example transformation of input metric ===

Following table lists transformation steps for a input metric
{| class="wikitable"
|-
! Step !! Component !! Description !! Spark Transformation !! Notes
|-
| 1 || Retrieve Input Metric 
<pre>
('583d49ab6aca482da653dd838deaf538load.avg_5_minhostnamedevstack',
'{"metric":{"name":"mem.total_mb",
"dimensions":{"service":"monitoring",
"hostname":"mini-mon"
},
"timestamp":1453308000000,
"value":5969.0},
"meta":{"tenantId":"583d49ab6aca482da653dd838deaf538",
"region":"useast"},
"creation_time":1453308005}')
</pre>
|| Incoming Monasca metric tuple pulled from Kafka || -- ||  
|-
| 2 || Call Input Metric to Record Store Converter || Converts incoming metric into a record store dataframe ||
<pre>
//pseudo code
KafkaUtils.createDirectStream //Spark Streaming API get discretized stream from kafka
DStream.transform() //to pull offsets
raw_metrics_df = inputDStreamRDD.map() //to extract the JSON metric from the tuple, and convert to dataframe
pre_transform_df = pre_transform_specs.jsonRDD() //convert to pre_transform_specs into data frame
filtered_metrics_df = raw_metrics_df.join(pre_transform_df).where(event_type=metric_name) //join raw data with pre transform specs df to filter out unwanted raw metrics
validated_df = filtered_metrics_df.validate() //validate metrics, check for required fields from pre_transform_spec for this event type
record_store_df = validated_df.explode() //generate multiple records for each intermediate metrics from pre_transform spec's intermediate_metric_id_list
</pre>
|| Pulls kafka offset information from Kafka DStream, filters, validates and generates records in record store dataframe format
|-
| 3 || Process Metrics in record store data || Finds the intermediate metrics from the record store dataframe, and for each metric finds the transformation pipeline from transform specs and invokes each component represented by aggregation_pipeline parameter in the transform spec || -- || Creates a transform_context object, which is contains Kafka offset information, and any configuration properties
|-
| 4 || Call fetch quantity usage component || fetch_quantity (latest) finds the latest quantity for an hour ||
<pre>
fetch_latest(transform_context, record_store_data) {

//pseudo code

grouped_sorted_df = record_store_df.groupBy("host", "event_date", "event_hour") and sort group by timestamp
latest_grouped_sorted_df = grouped_sorted_df: get latest in each group
instance_usage_df = convert latest_grouped_sorted_df to instance_usage_df
return instance_usage_df
}
</pre>
||  
|-
| 5 || Call Set Aggregated Metric Name Component || Sets final aggregated metric name ||
<pre>
set_aggregated_metric_metric_name(transform_context, instance_usage_df) {
//pseudo code
instance_usage_df = instance_usage_df.map(transform_context.transform_spec.aggregated_uom)
return instance_usage_df
}
</pre> ||  
|-
| 6 || Call Insert Kafka Component || Converts instance_usage_df to metrics json and publish to Kafka ||
<pre>
insert_data(transform_context, instance_usage_df) {
//pseudo code
//convert to json and set proper dimensions
instance_usage_json_list = convert_to_metrics_json(instance_usage_df,
transform_context.transform_specs.aggregaton_dimensions)
kafka_publisher.publish(instance_usage_json_list) //publish metrics to kafka
}
</pre>
||  
|-
|}

=== Impact of deploying Monasca Transform ===
==== Security impact ====
monasca-transform uses Spark Streaming to connect to the "metrics" topic in Kafka to retrieve metrics for processing. monasca-transform writes transformed metrics back to Kafka on behalf of the tenant which can be set in a configuration file. The tenant should have appropriate monasca-admin role so that the data gets persisted in the Monasca datastore. The Spark Web user interface and connections between master and worker nodes can be secured using ACL and encryption[4]. Connection to Kafka and encryption is not yet supported, but work is currently underway to provide such a support in Kafka[4].

==== Performance/Scalability Impacts ====

* Since new transformed metrics will be written back to the "metrics" topic in Kafka, it will have some impact in terms of increasing the metrics to be persisted. This should not have any significant impact as transformed/aggregated metrics would necessarily be significantly fewer in number compared to the overall number of incoming metrics being persisted.
* Given that transformed metrics will be significantly fewer in number they should not have any appreciable impact on data retention policy.
* Spark master and worker has to be configured and installed on nodes where monasca-transform is deployed (most likely alongside other Monasca components though it's not necessary) and thus has to be allocated resources like CPU and Memory for the Spark master, worker and executor processes. Also, monasca-transform python components have to be configured and deployed. Deploying Spark and monasca-transform will have some impact in terms of increasing load on nodes where it runs. This impact will have to be evaluated.
* In terms of scalability, Spark worker nodes can be scaled horizontally across additional nodes in case the amount of metrics that have to be processed increases.

==== Other deployer impact ====

monasca-transform will have to be configured via configuration file(s). Configuration options include pointers to Zookeeper, batching interval used by Spark Streaming, tenant id on behalf of which metrics will be submitted to Kafka and Spark Master information so that the monasca-transform can submit jobs to Spark cluster. There will also be configuration options to access the MySQL database to pull in information from the driver tables.

In addition, the deployer which installs monasca-transform will have to install start/stop scripts for the Spark worker, Spark master and monasca-transform processes. The deployer will also have to insert pre-transformation specifications and transformation specifications in the MySQL driver tables.

monasca-transform will be an optional component in Monasca and users can elect to either install it or not. A devstack plugin is being authored which will provide the means to install monasca-transform and Spark and also a tempest test. This will enable verification that monasca-transform is working. This can be used in CI/CD processing.

==== Developer impact ====

Developers who want to write their own transformation routines for aggregating new metrics will have to write new pre-transformation specifications in the transformation specifications JSON and add them to driver tables. To add new transformation components, the developers have to write python modules which implement the appropriate interface such as usage, setter, insert, etc. and use Spark transformations to manipulate and/or write data back to Kafka. New components will also have to be added to the setup.cfg so that new components can be loaded and utilized by monasca-transform.

=== Dependencies ===

* Apache Spark 1.6.3 (originally developed against 1.6.0)
* Zookeeper
* Relational database - tested with MariaDB 10.2
* Other Monasca services, including Kafka as used in monasca-common

=== Testing ===
Python unit tests are written to test each component of the overall transformation pipeline individually, as well as to test a part of or whole transformation pipeline. A devstack plugin was developed to install monasca-transform and Spark, allowing a simple way to try out monasca-transform.

=== References ===

1 Apache Kafka [[https://kafka.apache.org]]

2 Apache Spark [[https://spark.apache.org]]

3 Spark Security [[https://spark.apache.org/docs/latest/security.html]]

4 Kafka Security [[https://cwiki.apache.org/confluence/display/KAFKA/Security]]

Ceilosca

2020-06-02T07:41:29Z

Joseph Davis: /* Overview */ deprecation

= DEPRECATION =
Ceilosca is DEPRECATED as of the Ussuri OpenStack release.

The functionality of Ceilosca is [https://opendev.org/openstack/ceilometer/commit/126350c0ae609c5d35d54556883da2476e81e30e now found in Ceilometer] as the Monasca Publisher option. This closer integration will make it easier to deploy and connect Ceilometer to Monasca. To support older versions and follow OpenStack deprecation policies, this repository will continue to be maintained, but new deployments are encouraged to determine if the metrics required are available directly in Monasca, or use [https://docs.openstack.org/ceilometer/latest/admin/telemetry-data-pipelines.html#pipeline-configuration the Monasca Publisher].

= Overview =
Ceilosca is a combination of [[Ceilometer]] and [[Monasca]]. In Ceilosca data collected using the Ceilometer Agents (Notification Agent, Central/Compute Agent) is sent to the Monasca API and retrieved back using the Ceilometer V2 API for backwards compatibility.

Note that the Telemetry project has deprecated the Ceilometer v2 API starting in Newton and the API has been removed in the Queens release of Ceilometer. Ceilosca continues to support the Ceilometer v2 API through Pike, though users are encouraged to switch to using the Monasca API going forward. Data gathered through the Ceilometer agents will still continue to be published and stored in Monasca.

=== News ===
(Oct 2019) With the Ussuri OpenStack Release, Ceilosca will be integrated in the Ceilometer codebase as an official publisher. This will make it even easier to deploy and use. Watch commit https://review.opendev.org/#/c/562400/ for the latest status.

= Presentations =
:'''Tokyo Summit 2015'''
::Ceilometer+Monasca=Ceilosca
::Video: https://www.openstack.org/summit/tokyo-2015/videos/presentation/ceilometer-monascaceilosca
::Slides: https://www.slideshare.net/FabioGiannetti/ceilosca

= Features =
* Makes use of existing Ceilometer sample gathering functionality through OpenStack services
* Leverages the Monasca monitoring setup, thus reducing overall complexity in cloud deployment
* Integrated with devstack deployment so you can try it out

= Architecture =
[[File:Ceilosca-Architecture-Train.png|Ceilosca Architecture Component Diagram, Train release]]

In the Rocky release, Ceilosca consists of a '''Monasca Publisher''' component for Ceilometer to send samples to the Monasca API.

Historically, Ceilosca up through the Pike release had two major components:
* A '''publisher''' component that takes samples from the Ceilometer Agent and publishes them to the Monasca API.
* A '''storage driver''' component that works with the Ceilometer V2 API to use the Monasca API to retrieve stored samples.
[[File:Ceilosca-Architecture-recreatedslide.png|Monasca Architecture Component Diagram]]

Another [https://alln-extcloud-storage.cisco.com/ciscoblogs/ceilostica_arch-550x265.png Ceilosca Archiecture diagram]

Additionally, the Ceilosca project includes '''testing resources''' and a '''Devstack plugin''' to allow deployment.

= Deployment and Troubleshooting =

== Deploying ==
For information on how to deploy Ceilosca, see the [https://git.openstack.org/cgit/openstack/monasca-ceilometer/tree/README.md monasca-ceilometer/README.md] and [https://git.openstack.org/cgit/openstack/monasca-ceilometer/tree/devstack/README.md monasca-ceilometer/devstack/README.md].

In brief, Ceilosca may be overlaid on a devstack deployment which has Ceilometer and Monasca services, or you may use the provided Vagrant file to deploy a complete Ceilosca devstack, or you may use the examples and directions provided to include Ceilosca in your own deployment mechanism.

Note that the README.md has information about how to configure which meters are collected (TL;DR whatever is configured in Ceilometer's polling.yml) and available through the Monasca API.

== Troubleshooting - Common deployment ==
* Check the Ceilometer services are running
* Check /etc/ceilometer/pipeline.yaml has a proper publishers: -monasca:// entry
* Check /etc/ceilometer/pipeline.yaml has meters defined that match what you want to gather
* Check that /etc/ceilometer/ceilometer.conf has an entry in [database] for metering_connection that specifies a monasca://http: address that points to your monasca instance
* Look in /var/log/ceilometer.log or /opt/stack/logs/ceilometer.log for errors
* Look in /var/log/monasca/api/monasca-api.log for connection issues from ceilometer

== Troubleshooting - Devstack deployment ==
* Look in /opt/stack/logs/ceilometer.log for errors
* The Ceilometer command line client needs to have the correct environment variables set. If you have trouble connecting to Monasca, try using the default password from Monasca.
* export OS_PROJECT_NAME=service
* export OS_IDENTITY_API_VERSION=3
* export OS_PASSWORD=<password set in local.conf as SERVICE_PASSWORD>
* export OS_AUTH_URL=http://<fill in your Keystone ip>/identity/v3/
* export OS_USERNAME=ceilometer
* export OS_TENANT_NAME=service
* Check items in the Common deployment section below

= Repositories =
* OpenStack Git:
https://git.openstack.org/cgit/openstack/monasca-ceilometer/
* Github:
https://github.com/openstack/monasca-ceilometer

= Coding Standards =
All Python code conforms to the OpenStack standards at http://docs.openstack.org/developer/hacking/.

= License =

Copyright (c) 2015-2016 Hewlett-Packard Development Company, L.P.

Copyright (c) 2017 Hewlett Packard Enterprise Development LP

Copyright (c) 2015 Cisco.

Copyright (c) 2018 SUSE LLC

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied.
See the License for the specific language governing permissions and
limitations under the License.

Heat-Monasca-Auto-Scaling

2019-10-29T20:53:22Z

Joseph Davis: train workshop

== Introduction ==
In Liberty, heat has introduced the support for the monasca based auto-scaling, which is similar to the ceilometer based auto-scaling in heat. To use this feature, please follow the details given below:

== Setup the cloud environment ==
* Install python-monascaclient [1] in all of the heat-engines, deployed in your cloud environment and restart those heat-engines.
* Install and configure the monasca-agent [2] to monitor the VMs performance metric

== Create the auto-scale template ==
Use the monasca template [3] as reference and update the template with required alarm-definition expression. In the expression, its mandatory to mention the 'scale_group' measurement as :
<code>
avg(vm.cpu.utilization_perc{scale_group=XXX})
</code>

And the same 'scale_group' should be configured as metadata scale group elements (nova instances) as:
<code>
metadata: {"scale_group": XXX}
</code>

This will help monasca to aggregate the metrics from all elements in the given scale group and find out whether alarm threshold is reached as defined in the alarm expression.
In usual practice, scale_group XXX is filled by stack ID or scale group ID, which gives uniqueness.

== Train Workshop ==
A [https://www.openstack.org/summit/denver-2019/summit-schedule/events/23291/scaling-should-be-easy-and-automatic-heat-and-monasca workshop] demonstrating using auto-scaling with Monasca was presented at the Denver 2019 Summit. [5]

== Mitaka User session ==
In mitaka summit, monasca based auto-scaling is demonstrated and recorded video is available here [4]

== Reference ==
# https://pypi.python.org/pypi/python-monascaclient
# https://github.com/openstack/monasca-agent
# https://github.com/openstack/heat-templates/blob/master/hot/monasca/autoscaling.yaml
# https://www.openstack.org/summit/tokyo-2015/videos/presentation/auto-scaling-cloud-infrastructure-and-applications
# https://www.openstack.org/summit/denver-2019/summit-schedule/events/23291/scaling-should-be-easy-and-automatic-heat-and-monasca

Ceilosca

2019-10-29T19:48:04Z

Joseph Davis: /* Overview */

= Overview =
Ceilosca is a combination of [[Ceilometer]] and [[Monasca]]. In Ceilosca data collected using the Ceilometer Agents (Notification Agent, Central/Compute Agent) is sent to the Monasca API and retrieved back using the Ceilometer V2 API for backwards compatibility.

Note that the Telemetry project has deprecated the Ceilometer v2 API starting in Newton and the API has been removed in the Queens release of Ceilometer. Ceilosca continues to support the Ceilometer v2 API through Pike, though users are encouraged to switch to using the Monasca API going forward. Data gathered through the Ceilometer agents will still continue to be published and stored in Monasca.

=== News ===
(Oct 2019) With the Ussuri OpenStack Release, Ceilosca will be integrated in the Ceilometer codebase as an official publisher. This will make it even easier to deploy and use. Watch commit https://review.opendev.org/#/c/562400/ for the latest status.

= Presentations =
:'''Tokyo Summit 2015'''
::Ceilometer+Monasca=Ceilosca
::Video: https://www.openstack.org/summit/tokyo-2015/videos/presentation/ceilometer-monascaceilosca
::Slides: https://www.slideshare.net/FabioGiannetti/ceilosca

= Features =
* Makes use of existing Ceilometer sample gathering functionality through OpenStack services
* Leverages the Monasca monitoring setup, thus reducing overall complexity in cloud deployment
* Integrated with devstack deployment so you can try it out

= Architecture =
[[File:Ceilosca-Architecture-Train.png|Ceilosca Architecture Component Diagram, Train release]]

In the Rocky release, Ceilosca consists of a '''Monasca Publisher''' component for Ceilometer to send samples to the Monasca API.

Historically, Ceilosca up through the Pike release had two major components:
* A '''publisher''' component that takes samples from the Ceilometer Agent and publishes them to the Monasca API.
* A '''storage driver''' component that works with the Ceilometer V2 API to use the Monasca API to retrieve stored samples.
[[File:Ceilosca-Architecture-recreatedslide.png|Monasca Architecture Component Diagram]]

Another [https://alln-extcloud-storage.cisco.com/ciscoblogs/ceilostica_arch-550x265.png Ceilosca Archiecture diagram]

Additionally, the Ceilosca project includes '''testing resources''' and a '''Devstack plugin''' to allow deployment.

= Deployment and Troubleshooting =

== Deploying ==
For information on how to deploy Ceilosca, see the [https://git.openstack.org/cgit/openstack/monasca-ceilometer/tree/README.md monasca-ceilometer/README.md] and [https://git.openstack.org/cgit/openstack/monasca-ceilometer/tree/devstack/README.md monasca-ceilometer/devstack/README.md].

In brief, Ceilosca may be overlaid on a devstack deployment which has Ceilometer and Monasca services, or you may use the provided Vagrant file to deploy a complete Ceilosca devstack, or you may use the examples and directions provided to include Ceilosca in your own deployment mechanism.

Note that the README.md has information about how to configure which meters are collected (TL;DR whatever is configured in Ceilometer's polling.yml) and available through the Monasca API.

== Troubleshooting - Common deployment ==
* Check the Ceilometer services are running
* Check /etc/ceilometer/pipeline.yaml has a proper publishers: -monasca:// entry
* Check /etc/ceilometer/pipeline.yaml has meters defined that match what you want to gather
* Check that /etc/ceilometer/ceilometer.conf has an entry in [database] for metering_connection that specifies a monasca://http: address that points to your monasca instance
* Look in /var/log/ceilometer.log or /opt/stack/logs/ceilometer.log for errors
* Look in /var/log/monasca/api/monasca-api.log for connection issues from ceilometer

== Troubleshooting - Devstack deployment ==
* Look in /opt/stack/logs/ceilometer.log for errors
* The Ceilometer command line client needs to have the correct environment variables set. If you have trouble connecting to Monasca, try using the default password from Monasca.
* export OS_PROJECT_NAME=service
* export OS_IDENTITY_API_VERSION=3
* export OS_PASSWORD=<password set in local.conf as SERVICE_PASSWORD>
* export OS_AUTH_URL=http://<fill in your Keystone ip>/identity/v3/
* export OS_USERNAME=ceilometer
* export OS_TENANT_NAME=service
* Check items in the Common deployment section below

= Repositories =
* OpenStack Git:
https://git.openstack.org/cgit/openstack/monasca-ceilometer/
* Github:
https://github.com/openstack/monasca-ceilometer

= Coding Standards =
All Python code conforms to the OpenStack standards at http://docs.openstack.org/developer/hacking/.

= License =

Copyright (c) 2015-2016 Hewlett-Packard Development Company, L.P.

Copyright (c) 2017 Hewlett Packard Enterprise Development LP

Copyright (c) 2015 Cisco.

Copyright (c) 2018 SUSE LLC

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied.
See the License for the specific language governing permissions and
limitations under the License.

Ceilosca

2019-10-29T19:44:39Z

Joseph Davis: Update diagram for Train

= Overview =
Ceilosca is a combination of [[Ceilometer]] and [[Monasca]]. In Ceilosca data collected using the Ceilometer Agents (Notification Agent, Central/Compute Agent) is sent to the Monasca API and retrieved back using the Ceilometer V2 API for backwards compatibility.

Note that the Telemetry project has deprecated the Ceilometer v2 API starting in Newton and the API has been removed in the Queens release of Ceilometer. Ceilosca continues to support the Ceilometer v2 API through Pike, though users are encouraged to switch to using the Monasca API going forward. Data gathered through the Ceilometer agents will still continue to be published and stored in Monasca.

= Presentations =
:'''Tokyo Summit 2015'''
::Ceilometer+Monasca=Ceilosca
::Video: https://www.openstack.org/summit/tokyo-2015/videos/presentation/ceilometer-monascaceilosca
::Slides: https://www.slideshare.net/FabioGiannetti/ceilosca

= Features =
* Makes use of existing Ceilometer sample gathering functionality through OpenStack services
* Leverages the Monasca monitoring setup, thus reducing overall complexity in cloud deployment
* Integrated with devstack deployment so you can try it out

= Architecture =
[[File:Ceilosca-Architecture-Train.png|Ceilosca Architecture Component Diagram, Train release]]

In the Rocky release, Ceilosca consists of a '''Monasca Publisher''' component for Ceilometer to send samples to the Monasca API.

Historically, Ceilosca up through the Pike release had two major components:
* A '''publisher''' component that takes samples from the Ceilometer Agent and publishes them to the Monasca API.
* A '''storage driver''' component that works with the Ceilometer V2 API to use the Monasca API to retrieve stored samples.
[[File:Ceilosca-Architecture-recreatedslide.png|Monasca Architecture Component Diagram]]

Another [https://alln-extcloud-storage.cisco.com/ciscoblogs/ceilostica_arch-550x265.png Ceilosca Archiecture diagram]

Additionally, the Ceilosca project includes '''testing resources''' and a '''Devstack plugin''' to allow deployment.

= Deployment and Troubleshooting =

== Deploying ==
For information on how to deploy Ceilosca, see the [https://git.openstack.org/cgit/openstack/monasca-ceilometer/tree/README.md monasca-ceilometer/README.md] and [https://git.openstack.org/cgit/openstack/monasca-ceilometer/tree/devstack/README.md monasca-ceilometer/devstack/README.md].

In brief, Ceilosca may be overlaid on a devstack deployment which has Ceilometer and Monasca services, or you may use the provided Vagrant file to deploy a complete Ceilosca devstack, or you may use the examples and directions provided to include Ceilosca in your own deployment mechanism.

Note that the README.md has information about how to configure which meters are collected (TL;DR whatever is configured in Ceilometer's polling.yml) and available through the Monasca API.

== Troubleshooting - Common deployment ==
* Check the Ceilometer services are running
* Check /etc/ceilometer/pipeline.yaml has a proper publishers: -monasca:// entry
* Check /etc/ceilometer/pipeline.yaml has meters defined that match what you want to gather
* Check that /etc/ceilometer/ceilometer.conf has an entry in [database] for metering_connection that specifies a monasca://http: address that points to your monasca instance
* Look in /var/log/ceilometer.log or /opt/stack/logs/ceilometer.log for errors
* Look in /var/log/monasca/api/monasca-api.log for connection issues from ceilometer

== Troubleshooting - Devstack deployment ==
* Look in /opt/stack/logs/ceilometer.log for errors
* The Ceilometer command line client needs to have the correct environment variables set. If you have trouble connecting to Monasca, try using the default password from Monasca.
* export OS_PROJECT_NAME=service
* export OS_IDENTITY_API_VERSION=3
* export OS_PASSWORD=<password set in local.conf as SERVICE_PASSWORD>
* export OS_AUTH_URL=http://<fill in your Keystone ip>/identity/v3/
* export OS_USERNAME=ceilometer
* export OS_TENANT_NAME=service
* Check items in the Common deployment section below

= Repositories =
* OpenStack Git:
https://git.openstack.org/cgit/openstack/monasca-ceilometer/
* Github:
https://github.com/openstack/monasca-ceilometer

= Coding Standards =
All Python code conforms to the OpenStack standards at http://docs.openstack.org/developer/hacking/.

= License =

Copyright (c) 2015-2016 Hewlett-Packard Development Company, L.P.

Copyright (c) 2017 Hewlett Packard Enterprise Development LP

Copyright (c) 2015 Cisco.

Copyright (c) 2018 SUSE LLC

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied.
See the License for the specific language governing permissions and
limitations under the License.

File:Ceilosca-Architecture-Train.png

2019-10-29T19:42:31Z

Joseph Davis: Architecture diagram for Ceilosca as of the Train release.

Architecture diagram for Ceilosca as of the Train release.

Auto-scaling SIG/Theory of Auto-Scaling

2019-05-15T23:37:37Z

Joseph Davis: /* Considerations and Guidelines */

= Theory of Auto-Scaling =

== General Description ==

<fill in>
<what is the scope of auto-scaling, how does it differ from self-healing, what does it have in common with self-healing>

== Conceptual Diagram ==

[[File:OpenStack-Auto-Scaling.svg|Auto-Scaling Architecture Component Diagram]]

== Components of Auto-Scaling ==

OpenStack offers a rich set of services to build, manage, orchestrate, and provision a cloud. This gives administrators some choices in how to best serve their customer's needs.

* Scaling units - There are a number of components that can be controlled with Auto-Scaling.
** Compute Host
** VM running on a Compute Host
** Container running on a Compute Host
** Network Attached Storage
** Virtual Network Functions
* Monitoring Service - either using an agent installed on the Scaling unit, or using a polling method to retrieve metrics
** [https://wiki.openstack.org/wiki/Monasca Monasca]
** [https://wiki.openstack.org/wiki/Telemetry Ceilometer from the Telemetry project]
** Prometheus
* Alarming Service
** Monasca has a built in alarm thresholding service and notification service
** [https://wiki.openstack.org/wiki/Telemetry Aodh from the Telemetry project]
* Decision Services - There are a number of services in OpenStack that can interpret metrics and alarms based on configured logic and produce commands to Orchestration Engines
** Congress
** Heat
** Vitrage
** Watcher
* Orchestration Engines
** Heat
** Senlin is a clustering engine for OpenStack, and can orchestrate auto-scaling
** [https://wiki.openstack.org/wiki/Tacker Tacker]

== Considerations and Guidelines ==
* Monitoring takes resources, plan accordingly
* Avoid scaling too quickly or too often
* Don't expect instantaneous scaling (see above)
** Define thresholds to be predictive of scale needs, not reactive to a bad state
* Be aware of where the logic for scaling is (alarm thresholds, decision services)

=== Anecdotes and Stories ===

Auto-scaling SIG/Theory of Auto-Scaling

2019-05-15T23:29:46Z

Joseph Davis: /* Components of Auto-Scaling */

= Theory of Auto-Scaling =

== General Description ==

<fill in>
<what is the scope of auto-scaling, how does it differ from self-healing, what does it have in common with self-healing>

== Conceptual Diagram ==

[[File:OpenStack-Auto-Scaling.svg|Auto-Scaling Architecture Component Diagram]]

== Components of Auto-Scaling ==

OpenStack offers a rich set of services to build, manage, orchestrate, and provision a cloud. This gives administrators some choices in how to best serve their customer's needs.

* Scaling units - There are a number of components that can be controlled with Auto-Scaling.
** Compute Host
** VM running on a Compute Host
** Container running on a Compute Host
** Network Attached Storage
** Virtual Network Functions
* Monitoring Service - either using an agent installed on the Scaling unit, or using a polling method to retrieve metrics
** [https://wiki.openstack.org/wiki/Monasca Monasca]
** [https://wiki.openstack.org/wiki/Telemetry Ceilometer from the Telemetry project]
** Prometheus
* Alarming Service
** Monasca has a built in alarm thresholding service and notification service
** [https://wiki.openstack.org/wiki/Telemetry Aodh from the Telemetry project]
* Decision Services - There are a number of services in OpenStack that can interpret metrics and alarms based on configured logic and produce commands to Orchestration Engines
** Congress
** Heat
** Vitrage
** Watcher
* Orchestration Engines
** Heat
** Senlin is a clustering engine for OpenStack, and can orchestrate auto-scaling
** [https://wiki.openstack.org/wiki/Tacker Tacker]

== Considerations and Guidelines ==
* Monitoring takes resources, plan accordingly
* Avoid scaling too quickly or too often
* Don't expect instantaneous scaling (see above)
* Be aware of where the logic for scaling is (alarm thresholds, decision services)

=== Anecdotes and Stories ===

Auto-scaling SIG/Theory of Auto-Scaling

2019-05-15T23:28:22Z

Joseph Davis: /* Components of Auto-Scaling */

= Theory of Auto-Scaling =

== General Description ==

<fill in>
<what is the scope of auto-scaling, how does it differ from self-healing, what does it have in common with self-healing>

== Conceptual Diagram ==

[[File:OpenStack-Auto-Scaling.svg|Auto-Scaling Architecture Component Diagram]]

== Components of Auto-Scaling ==

OpenStack offers a rich set of services to build, manage, orchestrate, and provision a cloud. This gives administrators some choices in how to best serve their customer's needs.

* Scaling units - There are a number of components that can be controlled with Auto-Scaling.
** Compute Host
** VM running on a Compute Host
** Container running on a Compute Host
** Network Attached Storage
** Virtual Network Functions
* Monitoring Service
** [https://wiki.openstack.org/wiki/Monasca Monasca]
** [https://wiki.openstack.org/wiki/Telemetry Ceilometer from the Telemetry project]
** Prometheus
* Alarming Service
** Monasca has a built in alarm thresholding service and notification service
** [https://wiki.openstack.org/wiki/Telemetry Aodh from the Telemetry project]
* Decision Services - There are a number of services in OpenStack that can interpret metrics and alarms based on configured logic and produce commands to Orchestration Engines
** Congress
** Heat
** Vitrage
** Watcher
* Orchestration Engines
** Heat
** Senlin is a clustering engine for OpenStack, and can orchestrate auto-scaling
** [https://wiki.openstack.org/wiki/Tacker Tacker]

== Considerations and Guidelines ==
* Monitoring takes resources, plan accordingly
* Avoid scaling too quickly or too often
* Don't expect instantaneous scaling (see above)
* Be aware of where the logic for scaling is (alarm thresholds, decision services)

=== Anecdotes and Stories ===

Auto-scaling SIG/Theory of Auto-Scaling

2019-05-15T23:23:07Z

Joseph Davis: /* General Description */

= Theory of Auto-Scaling =

== General Description ==

<fill in>
<what is the scope of auto-scaling, how does it differ from self-healing, what does it have in common with self-healing>

== Conceptual Diagram ==

[[File:OpenStack-Auto-Scaling.svg|Auto-Scaling Architecture Component Diagram]]

== Components of Auto-Scaling ==

OpenStack offers a rich set of services to build, manage, orchestrate, and provision a cloud. This gives administrators some choices in how to best serve their customer's needs.

* Scaling units - There are a number of components that can be controlled with Auto-Scaling.
** Compute Host
** VM running on a Compute Host
** Container running on a Compute Host
** Network Attached Storage
** Virtual Network Functions
* Monitoring Service
** [https://wiki.openstack.org/wiki/Monasca Monasca]
** [https://wiki.openstack.org/wiki/Telemetry Ceilometer from the Telemetry project]
* Alarming Service
** Monasca has a built in alarm thresholding service and notification service
** [https://wiki.openstack.org/wiki/Telemetry Aodh from the Telemetry project]
* Decision Services - There are a number of services in OpenStack that can interpret metrics and alarms based on configured logic and produce commands to Orchestration Engines
** Congress
** Heat
** Vitrage
** Watcher
* Orchestration Engines
** Heat
** Senlin is a clustering engine for OpenStack, and can orchestrate auto-scaling
** [https://wiki.openstack.org/wiki/Tacker Tacker]

== Considerations and Guidelines ==
* Monitoring takes resources, plan accordingly
* Avoid scaling too quickly or too often
* Don't expect instantaneous scaling (see above)
* Be aware of where the logic for scaling is (alarm thresholds, decision services)

=== Anecdotes and Stories ===

Auto-scaling SIG/Theory of Auto-Scaling

2019-05-15T23:19:36Z

Joseph Davis: /* Conceptual Diagram */

= Theory of Auto-Scaling =

== General Description ==

<fill in>

== Conceptual Diagram ==

[[File:OpenStack-Auto-Scaling.svg|Auto-Scaling Architecture Component Diagram]]

== Components of Auto-Scaling ==

OpenStack offers a rich set of services to build, manage, orchestrate, and provision a cloud. This gives administrators some choices in how to best serve their customer's needs.

* Scaling units - There are a number of components that can be controlled with Auto-Scaling.
** Compute Host
** VM running on a Compute Host
** Container running on a Compute Host
** Network Attached Storage
** Virtual Network Functions
* Monitoring Service
** [https://wiki.openstack.org/wiki/Monasca Monasca]
** [https://wiki.openstack.org/wiki/Telemetry Ceilometer from the Telemetry project]
* Alarming Service
** Monasca has a built in alarm thresholding service and notification service
** [https://wiki.openstack.org/wiki/Telemetry Aodh from the Telemetry project]
* Decision Services - There are a number of services in OpenStack that can interpret metrics and alarms based on configured logic and produce commands to Orchestration Engines
** Congress
** Heat
** Vitrage
** Watcher
* Orchestration Engines
** Heat
** Senlin is a clustering engine for OpenStack, and can orchestrate auto-scaling
** [https://wiki.openstack.org/wiki/Tacker Tacker]

== Considerations and Guidelines ==
* Monitoring takes resources, plan accordingly
* Avoid scaling too quickly or too often
* Don't expect instantaneous scaling (see above)
* Be aware of where the logic for scaling is (alarm thresholds, decision services)

=== Anecdotes and Stories ===

Auto-scaling SIG/Theory of Auto-Scaling

2019-05-15T23:19:14Z

Joseph Davis: /* Components of Auto-Scaling */

= Theory of Auto-Scaling =

== General Description ==

<fill in>

== Conceptual Diagram ==

[[File:OpenStack-Auto-Scaling.svg|Auto-Scaling Architecture Component Diagram]]

If you prefer PlantUML

@startuml

cloud Cloud\n {
rectangle host as "Host" {
}
rectangle host2 as "Host" {
agent VM
agent VM2 as "VM"
agent Container
agent Container2 as "Container"
}
}

agent MS as "Monitoring Service"
agent DS as "Decision Services\n(Clustering,\nOptimization,\nRoot Cause)"
agent Heat as "Orchestration \nEngine"

host -down-> MS
VM -down-> MS
Container -down-> MS : "Metric \nSamples"

MS -down-> DS : "Alarms"
MS -down-> Heat : "Alarms"

DS -right-> Heat : "Scaling Commands"

Heat -up-> host : "Orchestration"
Heat -up-> VM2 : "Orchestration"
Heat -up-> Container2 : "Orchestration"

@enduml

== Components of Auto-Scaling ==

OpenStack offers a rich set of services to build, manage, orchestrate, and provision a cloud. This gives administrators some choices in how to best serve their customer's needs.

* Scaling units - There are a number of components that can be controlled with Auto-Scaling.
** Compute Host
** VM running on a Compute Host
** Container running on a Compute Host
** Network Attached Storage
** Virtual Network Functions
* Monitoring Service
** [https://wiki.openstack.org/wiki/Monasca Monasca]
** [https://wiki.openstack.org/wiki/Telemetry Ceilometer from the Telemetry project]
* Alarming Service
** Monasca has a built in alarm thresholding service and notification service
** [https://wiki.openstack.org/wiki/Telemetry Aodh from the Telemetry project]
* Decision Services - There are a number of services in OpenStack that can interpret metrics and alarms based on configured logic and produce commands to Orchestration Engines
** Congress
** Heat
** Vitrage
** Watcher
* Orchestration Engines
** Heat
** Senlin is a clustering engine for OpenStack, and can orchestrate auto-scaling
** [https://wiki.openstack.org/wiki/Tacker Tacker]

== Considerations and Guidelines ==
* Monitoring takes resources, plan accordingly
* Avoid scaling too quickly or too often
* Don't expect instantaneous scaling (see above)
* Be aware of where the logic for scaling is (alarm thresholds, decision services)

=== Anecdotes and Stories ===

Auto-scaling SIG/Theory of Auto-Scaling

2019-05-15T23:07:00Z

Joseph Davis: /* Theory of Auto-Scaling */ components

= Theory of Auto-Scaling =

== General Description ==

<fill in>

== Conceptual Diagram ==

[[File:OpenStack-Auto-Scaling.svg|Auto-Scaling Architecture Component Diagram]]

If you prefer PlantUML

@startuml

cloud Cloud\n {
rectangle host as "Host" {
}
rectangle host2 as "Host" {
agent VM
agent VM2 as "VM"
agent Container
agent Container2 as "Container"
}
}

agent MS as "Monitoring Service"
agent DS as "Decision Services\n(Clustering,\nOptimization,\nRoot Cause)"
agent Heat as "Orchestration \nEngine"

host -down-> MS
VM -down-> MS
Container -down-> MS : "Metric \nSamples"

MS -down-> DS : "Alarms"
MS -down-> Heat : "Alarms"

DS -right-> Heat : "Scaling Commands"

Heat -up-> host : "Orchestration"
Heat -up-> VM2 : "Orchestration"
Heat -up-> Container2 : "Orchestration"

@enduml

== Components of Auto-Scaling ==

OpenStack offers a rich set of services to build, manage, orchestrate, and provision a cloud. This gives administrators some choices in how to best serve their customer's needs.

* Scaling units - There are a number of components that can be controlled with Auto-Scaling.
** Compute Host
** VM running on a Compute Host
** Container running on a Compute Host
** Network Attached Storage
** Virtual Network Functions
* Monitoring Service
** [https://wiki.openstack.org/wiki/Monasca Monasca]
** [https://wiki.openstack.org/wiki/Telemetry Ceilometer from the Telemetry project]
* Alarming Service
** Monasca has a built in alarm thresholding service and notification service
** [https://wiki.openstack.org/wiki/Telemetry Aodh from the Telemetry project]
* Decision Services - There are a number of services in OpenStack that can interpret metrics and alarms based on configured logic and produce commands to Orchestration Engines
** Congress
** Heat
** Tacker
** Vitrage
** Watcher
* Orchestration Engines
** Heat
** Senlin is a clustering engine for OpenStack, and can orchestrate auto-scaling
** [https://wiki.openstack.org/wiki/Tacker Tacker]

== Considerations and Guidelines ==
* Monitoring takes resources, plan accordingly
* Avoid scaling too quickly or too often
* Don't expect instantaneous scaling (see above)
* Be aware of where the logic for scaling is (alarm thresholds, decision services)

=== Anecdotes and Stories ===

File:OpenStack-Auto-Scaling.svg

2019-05-15T22:52:51Z

Joseph Davis: Overall component diagram

Overall component diagram

Auto-scaling SIG/Theory of Auto-Scaling

2019-05-15T22:52:05Z

Joseph Davis: Initial framework

Monasca

2019-05-15T22:01:23Z

Joseph Davis: /* Features */

Monasca

2019-05-15T21:38:56Z

Joseph Davis: /* Comparisons to alternatives */

Monasca

2019-05-15T21:37:51Z

Joseph Davis: /* Requirements */ push down to heading 4

Monasca

2019-05-15T21:36:52Z

Joseph Davis: /* Requirements */

Monasca

2019-05-15T21:31:58Z

Joseph Davis: /* Features */

Monasca

2019-05-15T21:31:45Z

Joseph Davis: /* Features */

Auto-scaling SIG

2019-05-15T16:39:10Z

Joseph Davis: /* Goals */ (summarized 3 goals from PTG)

== Auto-scaling SIG ==

'''Status''': In review

'''Original proposal''': http://lists.openstack.org/pipermail/openstack-discuss/2019-January/001656.html

=== Mission ===

This [[OpenStack_SIGs|SIG]] aims to improve the experience of developing, operating, and using auto-scaling and its related features (like metering, cluster schedule, life cycle management), and to coordinate efforts across projects and communities (like k8s cluster auto-scaling on OpenStack). The SIG also provides a central place to put tests, documentations, and even common libraries for auto-scaling features.

The SIG is expected to focus more on auto-scaling user workloads; however work on auto-scaling infrastructure is also welcome, especially considering that user workloads in an undercloud are actually infrastructure in the corresponding overcloud.

=== Background ===
OpenStack provides multiple methods to auto-scale your cluster (Like using Heat AutoScalingGroup, Senlin Cluster, etc.). However without general coordination across projects, it may not be easy for users and ops to achieve auto-scaling on OpenStack. Developers tend to be focused on individual projects rather than cross-project integration. Most of the components required by auto-scaling already exist within OpenStack, but we need to provide a more simple way for users and ops to adopt auto-scaling. And allowing developers to coordinate together instead of implement something all over again.

=== Goals ===
* Set up SIG, init doc, meeting, ml
* Create Forum and Auto-scaling SIG PTG sessions - done, https://etherpad.openstack.org/p/DEN-auto-scaling-SIG
* Collect use cases for Auto-Scaling in OpenStack
* Improve documentation for Auto-Scaling features in various OpenStack services
* Foster cross-project communication and development of Auto-Scaling features

=== Getting Involved ===

We're currently working on a structure to make everyone more easier to getting involved. Our initial git repository, documentation, and other information will be up soon.

=== Community Infrastructure / Resources ===

* Wiki: this page
* [https://storyboard.openstack.org/#!/project/openstack/auto-scaling-sig SIG StoryBoard] (for an authoritative list of all ongoing work within the SIG)
* [http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-discuss openstack-discuss mailing list]; use the <code>[auto-scaling]</code> tag
* IRC channel: #openstack-auto-scaling on [http://freenode.net/ Freenode] IRC
* Regular IRC meetings 0900 and 2300 UTC on even weeks on IRC #openstack-auto-scaling [https://review.opendev.org/#/c/656810/]
* [https://review.openstack.org/#/q/project:openstack/auto-scaling-sig patch reviews] (gerrit)
* [https://opendev.org/openstack/auto-scaling-sig Git repository for SIG] (more content coming soon)

=== SIG Chairs ===

* Rico Lin
* Duc Truong
* Zane Bitter

We welcome anyone who willing to join and help. And we surely are willing to re-elect chairs as soon as we got our init setup done.

=== Upcoming events ===

TBD

=== Past events ===

* Superuser article, Sept 2018
** [https://superuser.openstack.org/articles/how-to-auto-scale-a-self-healing-cluster-with-heat/ How to auto scale a self healing cluster with Heat]
* Events at Berlin Forum, November 2018
** [https://etherpad.openstack.org/p/autoscaling-integration-and-feedback Autoscaling Integration, improvement, and feedback]
* Events at Denver OpenInfra Summit and PTG, April 2019
** [https://www.openstack.org/summit/denver-2019/summit-schedule/events/23379/how-blizzard-entertainment-uses-autoscaling-with-overwatch (presentation) How Blizzard uses Auto-scaling with Overwatch]
** [https://www.openstack.org/summit/denver-2019/summit-schedule/events/23291/scaling-should-be-easy-and-automatic-heat-and-monasca (workshop) Scaling should be Easy and Automatic]
** [https://etherpad.openstack.org/p/DEN-auto-scaling-SIG Auto-Scaling SIG PTG etherpad]

=== Project liasons ===
This is a good idea, but we need some more discussion on who and how we can bridge this system up.

Monasca/Migration from Ceilometer

2019-05-13T23:27:55Z

Joseph Davis: /* Considerations for Migration */

=Migration to Monasca from Ceilometer=
'''This page is a Work in Progress.'''

With many changes in the Telemetry project, there has been a lot of interest in using Monasca as the monitoring and metrics service in OpenStack. (ref Train PTG notes)

The Monasca and Telemetry projects have both been official OpenStack projects for many years, with some overlap and some complementary features. With Monasca, it is possible to have the functionality to monitor, meter, alarm, gather logs, and soon to track Events in your OpenStack installation.

== Comparison of Features ==
(to be filled in. level of detail a question, but might be nice to have a table listing the options across the top, Ceilometer/Gnocchi, Ceilometer publishing to Monasca, Monasca only)

== Considerations for Migration ==
(to be filled in. Likely need to point out that Events are not fully supported yet)

- Data currently stored in Ceilometer will not be migrated to a new Monasca monitoring system. So plan for a final backup and export of data from Ceilometer, and choose a cut-over date.

=== Supported Use Cases ===

* User wishes to gather monitoring information from their compute nodes (and VMs or Containers)
* User wishes to gather metering data from their compute nodes for Billing purposes
* User wishes to visualize their monitoring data (in Grafana)
* User wishes to export their metering data to CloudKitty for billing
* User wishes to export their metering data to another billing system (up to them to check on compatibility, but many support Monasca)
* User wishes to use monitoring data for auto-scaling or self-healing features (Vitrage, Heat, Watcher, etc)

== Recommended Migration Sequence ==
# Identify which Ceilometer metrics are being actively used and their value to the user. List the comparable metrics in Monasca.
# Do a proof of concept install with Monasca. Configure metrics in Monasca. Check that metrics in Monasca meet expectations.
# Verify any billing or external systems that use Ceilometer data work with Monasca data.
# Notify users of the cut-over date. This is likely easiest to accomplish coordinated with an OpenStack upgrade.
# Install Monasca. Shut down Ceilometer.
# Profit!

Monasca/Migration from Ceilometer

2019-05-13T23:26:31Z

Joseph Davis: /* Migration to Monasca from Ceilometer */

=Migration to Monasca from Ceilometer=
'''This page is a Work in Progress.'''

With many changes in the Telemetry project, there has been a lot of interest in using Monasca as the monitoring and metrics service in OpenStack. (ref Train PTG notes)

The Monasca and Telemetry projects have both been official OpenStack projects for many years, with some overlap and some complementary features. With Monasca, it is possible to have the functionality to monitor, meter, alarm, gather logs, and soon to track Events in your OpenStack installation.

== Comparison of Features ==
(to be filled in. level of detail a question, but might be nice to have a table listing the options across the top, Ceilometer/Gnocchi, Ceilometer publishing to Monasca, Monasca only)

== Considerations for Migration ==
(to be filled in. Likely need to point out that Events are not fully supported yet)
- Data currently stored in Ceilometer will not be migrated to a new Monasca monitoring system. So plan for a final backup and export of data from Ceilometer, and choose an cut-over date.

=== Supported Use Cases ===

* User wishes to gather monitoring information from their compute nodes (and VMs or Containers)
* User wishes to gather metering data from their compute nodes for Billing purposes
* User wishes to visualize their monitoring data (in Grafana)
* User wishes to export their metering data to CloudKitty for billing
* User wishes to export their metering data to another billing system (up to them to check on compatibility, but many support Monasca)
* User wishes to use monitoring data for auto-scaling or self-healing features (Vitrage, Heat, Watcher, etc)

== Recommended Migration Sequence ==
# Identify which Ceilometer metrics are being actively used and their value to the user. List the comparable metrics in Monasca.
# Do a proof of concept install with Monasca. Configure metrics in Monasca. Check that metrics in Monasca meet expectations.
# Verify any billing or external systems that use Ceilometer data work with Monasca data.
# Notify users of the cut-over date. This is likely easiest to accomplish coordinated with an OpenStack upgrade.
# Install Monasca. Shut down Ceilometer.
# Profit!

Monasca/Migration from Ceilometer

2019-05-08T17:43:06Z

Joseph Davis: start an outline

=Migration to Monasca from Ceilometer=
This page is a Work in Progress.

With many changes in the Telemetry project, there has been a lot of interest in using Monasca as the monitoring and metrics service in OpenStack. (ref Train PTG notes)

The Monasca and Telemetry projects have both been official OpenStack projects for many years, with some overlap and some complementary features. With Monasca, it is possible to have the functionality to monitor, meter, alarm, gather logs, and soon to track Events in your OpenStack installation.

== Comparison of Features ==
(to be filled in. level of detail a question, but might be nice to have a table listing the options across the top, Ceilometer/Gnocchi, Ceilometer publishing to Monasca, Monasca only)

== Considerations for Migration ==
(to be filled in. Likely need to point out that Events are not fully supported yet)

== Supported Use Cases ==

* User wishes to gather monitoring information from their compute nodes (and VMs or Containers)
* User wishes to gather metering data from their compute nodes for Billing purposes
* User wishes to visualize their monitoring data (in Grafana)
* User wishes to export their metering data to CloudKitty for billing
* User wishes to export their metering data to another billing system (up to them to check on compatibility, but many support Monasca)
* User wishes to use monitoring data for auto-scaling or self-healing features (Vitrage, Heat, Watcher, etc)

Monasca

2019-05-08T17:31:30Z

Joseph Davis: /* Architecture */

Monasca

2019-05-08T17:29:19Z

Joseph Davis:

Monasca

2019-05-08T17:27:42Z

Joseph Davis: trimming

Monasca

2019-05-08T17:19:00Z

Joseph Davis: /* Documentation */

== Overview ==

Monasca is a open-source multi-tenant, highly scalable, performant, fault-tolerant monitoring-as-a-service solution that integrates with OpenStack. It uses a REST API for high-speed metrics processing and querying and has a streaming alarm engine and notification engine.

[[File:OpenStack_Project_Monasca_vertical.png|thumb|right]]

==== Contribution ====
* [https://storyboard.openstack.org/#!/project_group/59 Project Group on StoryBoard]
* [https://storyboard.openstack.org/#!/board/111 Important activities in Stein release]
* [https://storyboard.openstack.org/#!/board/141 Important activities in Train release]
* [https://docs.openstack.org/monasca-api/latest/contributor/index.html Contribution Guidelines]

==== Communication and Meetings ====
* IRC: #openstack-monasca on freenode.net
* Weekly Meetings:
** http://eavesdrop.openstack.org/#Monasca_Team_Meeting
** [[Meetings/Monasca]]
* Use [Monasca] tag on [http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-discuss openstack-discuss mailing list].

===== PTG meetings =====
* [[MonascaSteinPTG|Stein PTG Summary (Denver)]]
* (coming soon - Train summary)

==== Documentation ====
: Monasca API (and links to other documents): https://docs.openstack.org/monasca-api/latest/
: Monasca command line interface: https://docs.openstack.org/python-monascaclient/latest/
: Monasca API Specification: https://github.com/openstack/monasca-api/blob/master/docs/monasca-api-spec.md
: Agent Documentation: https://github.com/openstack/monasca-agent

==== Presentations ====
There have been many interesting and excellent presentations on Monasca, many given at the OpenStack or OpenInfra Summits.
: [[Monasca/Presentations]] (updated with Denver 2019 links)

==== Repositories ====

: '''Code'''
:: Core: https://git.openstack.org/cgit/?q=monasca (old git link)
:: API: https://opendev.org/openstack/monasca-api

: '''Deployment'''
: The following repositories are available for deploying Monasca:
:: Docker: https://github.com/monasca/monasca-docker
:: Kubernetes: https://github.com/monasca/monasca-helm
:: Ansible: https://github.com/search?utf8=%E2%9C%93&q=ansible-monasca
:: Puppet: https://git.openstack.org/openstack/puppet-monasca

==== Bugs ====

===== All open bugs =====
: https://storyboard.openstack.org/#!/worklist/213

===== In Stein triaged bugs =====
: https://storyboard.openstack.org/#!/worklist/467

===== Bug-fixing progress =====
: https://storyboard.openstack.org/#!/board/114

== Requirements ==
[[Monasca/Requirements]].

See also https://opendev.org/openstack/monasca-specs/

== Features ==

This section describes the overall features.

* A highly performant, scalable, reliable and fault-tolerant Monitoring as a Service (MONaaS) solution that scales to service provider metrics levels of metrics throughput. Performance, scalability and high-availability have been designed in from the start. Can process 100s of thousands of metrics/sec as well as offer data retention periods of greater than a year with no data loss while still processing interactive queries.

* Rest API for storing and querying metrics and historical information. Most monitoring solution use special transports and protocols, such as CollectD or NSCA (Nagios). In our solution, http is the only protocol used. This simplifies the overall design and also allows for a much richer way of describing the data via dimensions.

* Multi-tenant and authenticated. Metrics are submitted and authenticated using Keystone and stored associated with a tenant ID.

* Metrics defined using a set of (key, value) pairs called dimensions.

* Real-time thresholding and alarming on metrics.

* Compound alarms described using a simple expressive grammar composed of alarm sub-expressions and logical operators.

* Monitoring agent that supports a number of built-in system and service checks and also supports Nagios checks and statsd.

* Open-source monitoring solution built on open-source technologies.

=== Comparisons to alternatives ===
[[Monasca/Comparison]]

== Architecture ==
[[File:monasca-arch-component-diagram.png|Monasca Architecture Component Diagram]]
* Monitoring Agent (monasca-agent): A modern Python based monitoring agent that consists of several sub-components and supports system metrics, such as cpu utilization and available memory, Nagios plugins, statsd and many built-in checks for services such as MySQL, RabbitMQ, and many others.
* Monitoring API (monasca-api): A well-defined and documented RESTful API for monitoring that is primarily focused on the following concepts and areas:
** Metrics: Store and query massive amounts of metrics in real-time.
** Statistics: Query statistics for metrics.
** Alarm Definitions: Create, update, query and delete alarm definitions.
** Alarms: Query and delete the alarm history.
***Simple expressive grammar for creating compound alarms composed of alarm subexpressions and logical operators.
***Alarm severities can be associated with alarms.
***The complete alarm state transition history is stored and queryable which allows for subsequent root cause analysis (RCA) or advanced analytics.
**Notification Methods: Create and delete notification methods and associate them with alarms, such as email. Supports the ability to notify users directly via email when an alarm state transitions occur.
** The Monasca API has both Java and Python implementations avaialble.
* Persister (monasca-persister): Consumes metrics and alarm state transitions from the MessageQ and stores them in the Metrics and Alarms database.
** The Persister has both Java and Python implementations.
* Transform and Aggregation Engine (monasca-transform): Transform metric names and values, such as delta or time-based derivative calculations, and creates new metrics that are published to the Message Queue. The Transform Engine is not available yet.
* Anomaly and Prediction Engine: Evaluates prediction and anomalies and generates predicted metrics as well as anomaly likelihood and anomaly scores. The Anomaly and Prediction Engine is currently in a prototype status.
* Threshold Engine (monasca-thresh): Computes thresholds on metrics and publishes alarms to the MessageQ when exceeded. Based on Apache Storm a free and open distributed real-time computation system.
* Notification Engine (monasca-notification): Consumes alarm state transition messages from the MessageQ and sends notifications, such as emails for alarms. The Notification Engine is Python based.
* Analytics Engine (monasca-analytics): Consumes alarm state transisitions and metrics from the MessageQ and does anomaly detection and alarm clustering/correlation.
* Message Queue: A third-party component that primarily receives published metrics from the Monitoring API and alarm state transition messages from the Threshold Engine that are consumed by other components, such as the Persister and Notification Engine. The Message Queue is also used to publish and consume other events in the system. Currently, a Kafka based MessageQ is supported. Kafka is a high performance, distributed, fault-tolerant, and scalable message queue with durability built-in. We will look at other alternatives, such as RabbitMQ and in-fact in our previous implementation RabbitMQ was supported, but due to performance, scale, durability and high-availability limitiations with RabbitMQ we have moved to Kafka.
* Metrics and Alarms Database: A third-party component that primarily stores metrics and the alarm state history. Currently, Vertica and InfluxDB are supported. Support for Cassandra is in progress.
* Config Database: A third-party component that stores a lot of the configuration and other information in the system. Currently, MySQL is supported. Support for PostgreSQL is in progress.
* Monitoring Client (python-monascaclient): A Python command line client and library that communicates and controls the Monitoring API. The Monitoring Client was written using the OpenStack Heat Python client as a framework. The Monitoring Client also has a Python library, "monascaclient" similar to the other OpenStack clients, that can be used to quickly build additional capabilities. The Monitoring Client library is used by the Monitoring UI, Ceilometer publisher, and other components.
* Monitoring UI: A Horizon dashboard for visualizing the overall health and status of an OpenStack cloud.
* Ceilometer publisher: A multi-publisher plugin for Ceilometer, not shown, that converts and publishes samples to the Monitoring API.
Most of the components are described in their respective repositories. However, there aren't any repositories for the third-party components used, so we describe some of the relevant details here.

Further historical context for the architecture can be found at [[Monasca/Architecture Details]]

=== Message Schema ===
[[Monasca/Message Schema]]

=== Message Queue ===
A distributed, performant, scalable, HA message queue for distributing metrics, alarms and events in the monitoring system. Currently, based on Kafka.

==== Messages ====
There are several messages that are published and consumed by various components in Monasca via the MessageQ. See [[Monasca/Message Schema|Message Schema]].

=== Metrics and Alarms Database ===

A high-performance analytics database that can store massive amounts of metrics and alarms in real-time and also support interactive queries. Currently Vertica and InfluxDB are supported.

The SQL schema that is used by Vertica is as follows:

* MonMetrics.Measurements: Stores the actual measurements that are sent.
** id: An integer ID for the measurement.
** definition_dimensions_id: A reference to DefinitionDimensions.
** time_stamp
** value
* MonMetrics.DefinitionDimensions
** id: A sha1 hash of (defintion_id, dimension_set_id)
** definition_id: A reference to the Definitions.id
** dimension_set_id: A reference to the Dimensions.dimension_set_id
* MonMetrics.Definitions
** id: A sha1 hash of the (name, tenant_id, region)
** name: Name of the metric.
** tenant_id: The tenant_id that submitted the metric.
** region: The region the metric was submitted under.
* MonMetric.Dimensions
** dimension_set_id: A sha1 hash of the set of dimenions for a metric.
** name: Name of dimension.
** value: Value of dimension.

=== Config Database ===
The config database store all the configuration information. Currently based on MySQL.

The SQL schema is as follows:

* alarm
** id
** tenant_id
** name
** description
** expression
** state
** actions_enabled
** created_at
** updated_at
** deleted_at
* alarm_action
** alarm_id
** alarm_state
** action_id
* notification_method
** id
** tenant_id
** name
** type
** address
** created_at
** updated_at
* sub_alarm
** id
** alarm_id
** function
** metric_name
** operator
** threshold
** period
** periods
** state
** created_at
** updated_at
* sub_alarm_dimension
** sub_alarm_id
** dimension_name
** value

== Events ==
Support for real-time event stream processing in Monasca is in progress. For more details see the link at, [[Monasca/Events]].

== Logging ==
Support for logging in Monasca is under discussion. For more details see the link at, [[Monasca/Logging]].

== Transform and Aggregation Engine ==
For more details see the link at, [[Monasca/Transform]].

== Analytics ==
Support for anomaly detection and alarm clustering/correlation is in progress. For more details see the link at, [[Monasca/Analytics]].

== Monitoring ==
Enablement and usage for monitoring the status of Monasca is under discussion. For more details see the link at, [[Monasca/Monitoring_Of_Monasca]]

== UI/UX Support ==
Adding more support for common UI/UX queries is under discussion. For more details see the link at, [[Monasca/UI_UX_Support]]

== Keystone Requirements ==
Monasca relies on keystone for running and there are requirements about which keystone configuration must exist.

* The endpoint for the api must be registered in keystone as the 'monasca' service.
* The api must have an admin token to use in verifying the keystone tokens it receives.
* For each project which uses Monasca two users must exist, one will be in the 'monasca-agent' role and be used by the monasca-agent's running on machines. The other should not be in that role and can be used logging into the UI, using the CLI or for direct queries against the API.

== Post Metric Sequence ==

This section describes the sequence of operations involved in posting a metric to the Monasca API.

[[File:monasca-arch-post-metric-diagram.png|Monasca Architecture Post Metric Diagram]]

# A metric is posted to the Monasca API.
# The Monasca API authenticates and validates the request and publishes the metric to the the Message Queue.
# The Persister consumes the metric from the Message Queue and stores in the Metrics Store.
# The Transform Engine consumes the metrics from the Message Queue, performs transform and aggregation operations on metrics, and publishes metrics that it creates back to Message Queue.
# The Threshold Engine consumes metrics from the Message Queue and evaluates alarms. If a state change occurs in an alarm, an "alarm-state-transitioned-event" is published to the Message Queue.
# The Notification Engine consumes "alarm-state-transitioned-events" from the Message Queue, evaluates whether they have a Notification Method associated with it, and sends the appropriate notification, such as email.
# The Persister consumes the "alarm-state-transitioned-event" from the Message Queue and stores it in the Alarm State History Store.

== Alarm Managers ==
This section describes the new features including alarm grouping, inhibition and silencing.

Examples:

* Silence
SilenceRule1 = '{"alarm-silencing-definition-created": {"name": "silence_rule_1", "matchers": {"severity": "LOW"}, "start_time": "2017-02-21 20:00:00", "end_time": "2017-02-21 22:00:00"}}'
Two alarm transitions: AT1 and AT2
AT1_severity = HIGH
AT2_severity = LOW

Output: AT2 get silenced and AT1 send a notification.
SilenceRule1 expires after "end_time"="2017-02-21 22:00:00".

* Inhibit
InhibitionRule1 = '{"alarm-inhibition-definition-created": {"name": "inhibit_rule_1","source":{"severity":"HIGH"},"target":{"severity":"LOW"}, "equals":["tenantId"], "exclusions": {"alarm_name": "vm is dead"}}}'
Three alarm transitions: AT1, AT2 and AT3
AT1_tenantId = "d42bc"
AT2_tenantId = "d42bc"
AT3_tenantId = "d42bc"
AT1_severity = HIGH
AT2_severity = LOW
AT3_severity = LOW
AT1_alarm_name = "cpu high"
AT2_alarm_name = "memory high"
AT3_alarm_name = "vm is dead"
AT1_state = ALARM
AT2_state = ALARM
AT3_state = ALARM

Output: AT1 is the source alarm which will send a notification. AT2 is the target alarm and will get inhibited. AT3 matches the exclusions and will send a notification immediately.

* Grouping Scenario 1

GroupingRule1 = '{"alarm-grouping-definition-created":
{"name": "group_rule_1", "matchers": ["hostname"], "id": "b7163","repeat_interval": "2h", "group_wait": "30s", "exclusions": {"alarm_name": "cpu_percent_high"}, "tenantId": " d42bc", "alarm_actions": ["cd892"], "ok_actions": ["ad892"], "undetermined_actions": ["cf892"]}}'

Three alarm transitions: AT1, AT2 and AT3

AT1_hostname = host1
AT2_hostname = host1
AT3_hostname = host2
AT1_alarm_name = cpu_percent_high
AT2_alarm_name = cpu_system_perc_high
AT3_alarm_name = cpu_percent_high
AT1_state = ALARM
AT2_state = ALARM
AT3_state = ALARM

Output: AT1 and AT3 match exclusions and send notifications immediately. Generate a grouped notification “group_notification_rule_1_host1_alarm[1]” and send out using alarm_actions ["cd892"]. There are no alarm_actions, ok_actions or undermined_actions associated with the AT1, AT2, AT3 alarm definitions.

* Grouping Scenario 2

GroupingRule2 = '{"alarm-grouping-definition-created":
{"name": "group_rule_2", "matchers": ["hostname"], "id": "b7163","repeat_interval": "2h", "group_wait": "30s", "tenantId": " d42bc", "alarm_actions": ["cd892"], "ok_actions": ["ad892"], "undetermined_actions": ["cf892"]}}'

Three alarm transitions: AT1 with "alarm_actions": ["123ab"], AT2 with "alarm_actions": ["cd839"] and AT3

AT1_hostname = host1
AT2_hostname = host1
AT3_hostname = host2
AT1_state = ALARM
AT2_state = ALARM
AT3_state = ALARM

Output: Generate two grouped notifications “group_notification_rule_2_host1_alarm[2]” and “group_notification_rule_2_host2_alarm[1]”. Both using alarm_actions ["cd892"]. Also since AT1 and AT2 has their own alarm actions associated with them, there will be two more notifications sent out.

* Silenced and Grouped

SilenceRule1 = '{"alarm-silencing-definition-created": {"name": "silence_rule_1", "matchers": {"severity": "LOW"}, "start_time": "1487269470498", "end_time": "1587269470498"}}'

GroupingRule2 = '{"alarm-grouping-definition-created":
{"name": "group_rule_2", "matchers": ["hostname"], "id": "b7163","repeat_interval": "2h", "group_wait": "30s", "tenantId": " d42bc", "alarm_actions": ["cd892"], "ok_actions": ["ad892"], "undetermined_actions": ["cf892"]}}'

Four alarm transitions: AT1, AT2, AT3 and AT4

Silencing rule:
AT1_severity = HIGH
AT2_severity = LOW
AT3_severity = HIGH
AT4_severity = HIGH

Grouping rule:
AT1_hostname = host1
AT2_hostname = host1
AT3_hostname = host2
AT4_hostname = host1
AT1_state = ALARM
AT2_state = ALARM
AT3_state = OK
AT4_state = ALARM

Output: Generate two grouped notifications “group_notification_rule_2_host1_alarm[2]” using alarm action "cd892" and “group_notification_rule_2_host2_ok[1]” using ok action "ad892". AT2 is silenced so it isn't include in group_notification_rule_2_host1 count. There are no alarm_actions, ok_actions or undermined_actions associated with the AT1, AT2, AT3, AT4 alarm definitions.

* Silenced and inhibited (source alarm get silenced)

SilenceRule2 = '{"alarm-silencing-definition-created": {"name": "silence_rule_2", "matchers": {"severity": "HIGH, "hostname": "host1"}, "start_time": "2017-02-21 15:00:00", "end_time": "2017-02-21 21:00:00"}}'

InhibitionRule1 = '{"alarm-inhibition-definition-created": {"name": "inhibit_rule_1","source":{"severity":"HIGH"},"target":{"severity":"LOW"}, "equals":["tenantId"]}}'

Alarm transitions: AT1, AT2

Inhibition rule:
AT1_state = ALARM
AT2_state = ALARM
AT1_tenantId = "d42bc"
AT2_tenantId = "d42bc"
AT1_severity = HIGH
AT2_severity = LOW

Silencing rule:
AT1_hostname = host1
AT2_hostname = host2
AT1_severity = HIGH
AT2_severity = LOW

Output: no notification sent out. For inhibition, AT1 is the source alarm, AT2 is the target alarm. But at the same time, AT1 get silenced because it matches the silence rule.

* Inhibited and grouped

InhibitionRule1 = '{"alarm-inhibition-definition-created": {"name": "inhibit_rule_1","source":{"severity":"HIGH"},"target":{"severity":"LOW"}, "equals":["tenantId"]}}'

GroupingRule1 = '{"alarm-grouping-definition-created": {"name": "group_rule_1", "matchers": ["hostname"], "id": "b7163", "repeat_interval": "2h", "group_wait": "30s", "tenantId": " d42bc", "alarm_actions": ["cd892"], "ok_actions": ["ad892"], "undetermined_actions": ["cf892"]}}'

Alarm transitions: AT1, AT2, AT3

Inhibition rule:
AT1_tenantId = "d42bc"
AT2_tenantId = "d42bc"
AT3_tenantId = "d42bc"
AT1_severity = HIGH
AT2_severity = LOW
AT3_severity = HIGH
AT1_state = OK
AT2_state = ALARM
AT3_state = ALARM

Grouping rule:
AT1_hostname = host1
AT2_hostname = host2
AT3_hostname = host1
AT1_state = OK
AT2_state = ALARM
AT3_state = ALARM

Output: AT2 gets inhibited because its severity is low. AT3 is the source alarm. Since AT1 is in OK state, it is not a source alarm. For grouping, AT1 and AT3 has the same host name but different state. So there will be two grouped notifications sent out: “group_notification_rule_1_host1_ok[1]” and “group_notification_rule_1_host1_alarm[1]”.

* Silenced, inhibited and grouped

SilenceRule2 = '{"alarm-silencing-definition-created": {"name": "silence_rule_2", "matchers": {"severity": "HIGH, "hostname": "host1"}, "start_time": "2017-02-21 15:00:00", "end_time": "2017-02-21 21:00:00"}}'

InhibitionRule1 = '{"alarm-inhibition-definition-created": {"name": "inhibit_rule_1","source":{"severity":"HIGH"},"target":{"severity":"LOW"}, "equals":["tenantId"]}}'

GroupingRule1 = '{"alarm-grouping-definition-created": {"name": "group_rule_1", "matchers": ["hostname"], "id": "b7163", "repeat_interval": "2h", "group_wait": "30s", "tenantId": " d42bc", "alarm_actions": ["cd892"], "ok_actions": ["ad892"], "undetermined_actions": ["cf892"]}}'

Alarm transitions: AT1, AT2, AT3 and AT5

Silence rule:
AT1_severity = HIGH AT1_hostname = host1 (silenced)
AT2_severity = LOW AT2_hostname = host2
AT3_severity = HIGH AT3_hostname = host1(silenced)
AT5_severity = HIGH AT5_hostname = host3

Inhibition rule:
AT1_tenantId = "d42bc"
AT2_tenantId = "d42bc"
AT3_tenantId = "d42bc"
AT5_tenantId = "d42bc"
AT1_state = ALARM
AT2_state = ALARM
AT3_state = OK
AT5_state = UNDETERMINED
AT1_severity = HIGH (source)
AT2_severity = LOW (target)
AT3_severity = HIGH
AT5_severity = HIGH

Grouping rule:
AT1_hostname = host1
AT2_hostname = host2
AT3_hostname = host1
AT5_hostname = host3

Output: AT1 is in "group_notification_rule_1_host1_alarm" group and silenced. AT3 is in "group_notification_rule_1_host1_ok" group and silenced. AT2 is in "group_notification_rule_1_host2_alarm" group and inhibited. AT5 is in "group_notification_rule_1_host3_undetermined" group and will send notification “group_notification_rule_1_host3_undetermined[1]” using undetermined action "cf892".

= Development Environment =

* Monasca DevStack Plugin:
** DevStack is the primary developmement environment for OpenStack.
***See http://docs.openstack.org/developer/devstack/
** The Monasca DevStack plugin installs the Monasca Service, Agent, Horizon Monitoring Panel, and Grafana
** README at, https://github.com/openstack/monasca-api/tree/master/devstack
** Best way to get started is to use Vagrant with the Vagrantfile at, https://github.com/openstack/monasca-api/blob/master/devstack/Vagrantfile.

* Project and Bug tracking
** Monasca on Storyboard https://storyboard.openstack.org/#!/project/list?q=monasca
** Monasca on LaunchPad. https://launchpad.net/monasca (history only, Storyboard is the official tracking since 2017)

* Monasca projects source code. https://git.openstack.org/cgit/?q=monasca

= Coding Standards =

* Python: All Python code conforms to the OpenStack standards at, http://docs.openstack.org/developer/hacking/.
** Note, all components in Monasca, except for the Threshold Engine, have been ported to Python.

* Java: Several of the Monasca components are available as Java. OpenStack does not have any Java coding standards. We've adopted the Google Java Style at, https://google.github.io/styleguide/javaguide.html.
** The standard says either 80 or 100 length lines. We've adopted 100.

= Technologies =

Monasca uses a number of third-party technologies:

* Internal Processing and Middleware
** Apache Kafka (http://kafka.apache.org): Apache Kafka is publish-subscribe messaging rethought as a distributed commit log. Kafka is a highly performant, distributed, fault-tolerant, and scalable message queue with durability built-in.
** Apache Storm (http://storm.incubator.apache.org/): Apache Storm is a free and open source distributed realtime computation system. Storm makes it easy to reliably process unbounded streams of data, doing for realtime processing what Hadoop did for batch processing.
** ZooKeeper (http://zookeeper.apache.org/): Used by Kafka and Storm.
** Apache Spark: Used by Monasca Transform as an aggregation engine.

* Configuration database:
** MySQL: MySQL is supported as a Config Database.
** PostgreSQL: Support for POSTgres, via Hibernate and SQLAlchemy, for the Config Database.

* Vagrant (http://www.vagrantup.com/): Vagrant provides easy to configure, reproducible, and portable work environments built on top of industry-standard technology and controlled by a single consistent workflow to help maximize the productivity and flexibility of you and your team.

* Dropwizard (https://dropwizard.github.io/dropwizard/): Dropwizard pulls together stable, mature libraries from the Java ecosystem into a simple, light-weight package that lets you focus on getting things done. Dropwizard has out-of-the-box support for sophisticated configuration, application metrics, logging, operational tools, and much more, allowing you and your team to ship a production-quality web service in the shortest time possible.

* Time Series Database:
** InfluxDB (http://influxdb.com/): An open-source distributed time series database with no external dependencies. InfluxDB is supported for the Metrics Database.
** Vertica (http://www.vertica.com): A commercial Enterprise class SQL analytics database that is highly scalable. It offers built-in automatic high-availability and excels at in-database analytics and compressing and storing massive amounts of data. A free community version of Vertica is available that can store up to 1 TB of data with no time-limit is available at, https://my.vertica.com/community/. Vertica is supported for the Metrics Database, though no longer commonly used.
** Cassandra(https://cassandra.apache.org): Cassandra is supported for the Metrics Database.

= License =

Copyright (c) 2014, 2015 Hewlett-Packard Development Company, L.P.

(C) Copyright 2019 SUSE LLC

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied.
See the License for the specific language governing permissions and
limitations under the License.

[[File:yklogo.png|200px|thumb|left|Monasca uses YourKit Profiler for Java development]]

 
 
 
 

[https://www.yourkit.com/ Visit YourKit website for more information]

Monasca

2019-05-08T17:15:24Z

Joseph Davis: /* Development Environment */

== Overview ==

Monasca is a open-source multi-tenant, highly scalable, performant, fault-tolerant monitoring-as-a-service solution that integrates with OpenStack. It uses a REST API for high-speed metrics processing and querying and has a streaming alarm engine and notification engine.

[[File:OpenStack_Project_Monasca_vertical.png|thumb|right]]

==== Contribution ====
* [https://storyboard.openstack.org/#!/project_group/59 Project Group on StoryBoard]
* [https://storyboard.openstack.org/#!/board/111 Important activities in Stein release]
* [https://storyboard.openstack.org/#!/board/141 Important activities in Train release]
* [https://docs.openstack.org/monasca-api/latest/contributor/index.html Contribution Guidelines]

==== Communication and Meetings ====
* IRC: #openstack-monasca on freenode.net
* Weekly Meetings:
** http://eavesdrop.openstack.org/#Monasca_Team_Meeting
** [[Meetings/Monasca]]
* Use [Monasca] tag on [http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-discuss openstack-discuss mailing list].

===== PTG meetings =====
* [[MonascaSteinPTG|Stein PTG Summary (Denver)]]
* (coming soon - Train summary)

==== Documentation ====
: Monasca API Specification: https://github.com/openstack/monasca-api/blob/master/docs/monasca-api-spec.md
: Agent Documentation: https://github.com/openstack/monasca-agent

==== Presentations ====
There have been many interesting and excellent presentations on Monasca, many given at the OpenStack or OpenInfra Summits.
: [[Monasca/Presentations]] (updated with Denver 2019 links)

==== Repositories ====

: '''Code'''
:: Core: https://git.openstack.org/cgit/?q=monasca (old git link)
:: API: https://opendev.org/openstack/monasca-api

: '''Deployment'''
: The following repositories are available for deploying Monasca:
:: Docker: https://github.com/monasca/monasca-docker
:: Kubernetes: https://github.com/monasca/monasca-helm
:: Ansible: https://github.com/search?utf8=%E2%9C%93&q=ansible-monasca
:: Puppet: https://git.openstack.org/openstack/puppet-monasca

==== Bugs ====

===== All open bugs =====
: https://storyboard.openstack.org/#!/worklist/213

===== In Stein triaged bugs =====
: https://storyboard.openstack.org/#!/worklist/467

===== Bug-fixing progress =====
: https://storyboard.openstack.org/#!/board/114

== Requirements ==
[[Monasca/Requirements]].

See also https://opendev.org/openstack/monasca-specs/

== Features ==

This section describes the overall features.

* A highly performant, scalable, reliable and fault-tolerant Monitoring as a Service (MONaaS) solution that scales to service provider metrics levels of metrics throughput. Performance, scalability and high-availability have been designed in from the start. Can process 100s of thousands of metrics/sec as well as offer data retention periods of greater than a year with no data loss while still processing interactive queries.

* Rest API for storing and querying metrics and historical information. Most monitoring solution use special transports and protocols, such as CollectD or NSCA (Nagios). In our solution, http is the only protocol used. This simplifies the overall design and also allows for a much richer way of describing the data via dimensions.

* Multi-tenant and authenticated. Metrics are submitted and authenticated using Keystone and stored associated with a tenant ID.

* Metrics defined using a set of (key, value) pairs called dimensions.

* Real-time thresholding and alarming on metrics.

* Compound alarms described using a simple expressive grammar composed of alarm sub-expressions and logical operators.

* Monitoring agent that supports a number of built-in system and service checks and also supports Nagios checks and statsd.

* Open-source monitoring solution built on open-source technologies.

=== Comparisons to alternatives ===
[[Monasca/Comparison]]

== Architecture ==
[[File:monasca-arch-component-diagram.png|Monasca Architecture Component Diagram]]
* Monitoring Agent (monasca-agent): A modern Python based monitoring agent that consists of several sub-components and supports system metrics, such as cpu utilization and available memory, Nagios plugins, statsd and many built-in checks for services such as MySQL, RabbitMQ, and many others.
* Monitoring API (monasca-api): A well-defined and documented RESTful API for monitoring that is primarily focused on the following concepts and areas:
** Metrics: Store and query massive amounts of metrics in real-time.
** Statistics: Query statistics for metrics.
** Alarm Definitions: Create, update, query and delete alarm definitions.
** Alarms: Query and delete the alarm history.
***Simple expressive grammar for creating compound alarms composed of alarm subexpressions and logical operators.
***Alarm severities can be associated with alarms.
***The complete alarm state transition history is stored and queryable which allows for subsequent root cause analysis (RCA) or advanced analytics.
**Notification Methods: Create and delete notification methods and associate them with alarms, such as email. Supports the ability to notify users directly via email when an alarm state transitions occur.
** The Monasca API has both Java and Python implementations avaialble.
* Persister (monasca-persister): Consumes metrics and alarm state transitions from the MessageQ and stores them in the Metrics and Alarms database.
** The Persister has both Java and Python implementations.
* Transform and Aggregation Engine (monasca-transform): Transform metric names and values, such as delta or time-based derivative calculations, and creates new metrics that are published to the Message Queue. The Transform Engine is not available yet.
* Anomaly and Prediction Engine: Evaluates prediction and anomalies and generates predicted metrics as well as anomaly likelihood and anomaly scores. The Anomaly and Prediction Engine is currently in a prototype status.
* Threshold Engine (monasca-thresh): Computes thresholds on metrics and publishes alarms to the MessageQ when exceeded. Based on Apache Storm a free and open distributed real-time computation system.
* Notification Engine (monasca-notification): Consumes alarm state transition messages from the MessageQ and sends notifications, such as emails for alarms. The Notification Engine is Python based.
* Analytics Engine (monasca-analytics): Consumes alarm state transisitions and metrics from the MessageQ and does anomaly detection and alarm clustering/correlation.
* Message Queue: A third-party component that primarily receives published metrics from the Monitoring API and alarm state transition messages from the Threshold Engine that are consumed by other components, such as the Persister and Notification Engine. The Message Queue is also used to publish and consume other events in the system. Currently, a Kafka based MessageQ is supported. Kafka is a high performance, distributed, fault-tolerant, and scalable message queue with durability built-in. We will look at other alternatives, such as RabbitMQ and in-fact in our previous implementation RabbitMQ was supported, but due to performance, scale, durability and high-availability limitiations with RabbitMQ we have moved to Kafka.
* Metrics and Alarms Database: A third-party component that primarily stores metrics and the alarm state history. Currently, Vertica and InfluxDB are supported. Support for Cassandra is in progress.
* Config Database: A third-party component that stores a lot of the configuration and other information in the system. Currently, MySQL is supported. Support for PostgreSQL is in progress.
* Monitoring Client (python-monascaclient): A Python command line client and library that communicates and controls the Monitoring API. The Monitoring Client was written using the OpenStack Heat Python client as a framework. The Monitoring Client also has a Python library, "monascaclient" similar to the other OpenStack clients, that can be used to quickly build additional capabilities. The Monitoring Client library is used by the Monitoring UI, Ceilometer publisher, and other components.
* Monitoring UI: A Horizon dashboard for visualizing the overall health and status of an OpenStack cloud.
* Ceilometer publisher: A multi-publisher plugin for Ceilometer, not shown, that converts and publishes samples to the Monitoring API.
Most of the components are described in their respective repositories. However, there aren't any repositories for the third-party components used, so we describe some of the relevant details here.

Further historical context for the architecture can be found at [[Monasca/Architecture Details]]

=== Message Schema ===
[[Monasca/Message Schema]]

=== Message Queue ===
A distributed, performant, scalable, HA message queue for distributing metrics, alarms and events in the monitoring system. Currently, based on Kafka.

==== Messages ====
There are several messages that are published and consumed by various components in Monasca via the MessageQ. See [[Monasca/Message Schema|Message Schema]].

=== Metrics and Alarms Database ===

A high-performance analytics database that can store massive amounts of metrics and alarms in real-time and also support interactive queries. Currently Vertica and InfluxDB are supported.

The SQL schema that is used by Vertica is as follows:

* MonMetrics.Measurements: Stores the actual measurements that are sent.
** id: An integer ID for the measurement.
** definition_dimensions_id: A reference to DefinitionDimensions.
** time_stamp
** value
* MonMetrics.DefinitionDimensions
** id: A sha1 hash of (defintion_id, dimension_set_id)
** definition_id: A reference to the Definitions.id
** dimension_set_id: A reference to the Dimensions.dimension_set_id
* MonMetrics.Definitions
** id: A sha1 hash of the (name, tenant_id, region)
** name: Name of the metric.
** tenant_id: The tenant_id that submitted the metric.
** region: The region the metric was submitted under.
* MonMetric.Dimensions
** dimension_set_id: A sha1 hash of the set of dimenions for a metric.
** name: Name of dimension.
** value: Value of dimension.

=== Config Database ===
The config database store all the configuration information. Currently based on MySQL.

The SQL schema is as follows:

* alarm
** id
** tenant_id
** name
** description
** expression
** state
** actions_enabled
** created_at
** updated_at
** deleted_at
* alarm_action
** alarm_id
** alarm_state
** action_id
* notification_method
** id
** tenant_id
** name
** type
** address
** created_at
** updated_at
* sub_alarm
** id
** alarm_id
** function
** metric_name
** operator
** threshold
** period
** periods
** state
** created_at
** updated_at
* sub_alarm_dimension
** sub_alarm_id
** dimension_name
** value

== Events ==
Support for real-time event stream processing in Monasca is in progress. For more details see the link at, [[Monasca/Events]].

== Logging ==
Support for logging in Monasca is under discussion. For more details see the link at, [[Monasca/Logging]].

== Transform and Aggregation Engine ==
For more details see the link at, [[Monasca/Transform]].

== Analytics ==
Support for anomaly detection and alarm clustering/correlation is in progress. For more details see the link at, [[Monasca/Analytics]].

== Monitoring ==
Enablement and usage for monitoring the status of Monasca is under discussion. For more details see the link at, [[Monasca/Monitoring_Of_Monasca]]

== UI/UX Support ==
Adding more support for common UI/UX queries is under discussion. For more details see the link at, [[Monasca/UI_UX_Support]]

== Keystone Requirements ==
Monasca relies on keystone for running and there are requirements about which keystone configuration must exist.

* The endpoint for the api must be registered in keystone as the 'monasca' service.
* The api must have an admin token to use in verifying the keystone tokens it receives.
* For each project which uses Monasca two users must exist, one will be in the 'monasca-agent' role and be used by the monasca-agent's running on machines. The other should not be in that role and can be used logging into the UI, using the CLI or for direct queries against the API.

== Post Metric Sequence ==

This section describes the sequence of operations involved in posting a metric to the Monasca API.

[[File:monasca-arch-post-metric-diagram.png|Monasca Architecture Post Metric Diagram]]

# A metric is posted to the Monasca API.
# The Monasca API authenticates and validates the request and publishes the metric to the the Message Queue.
# The Persister consumes the metric from the Message Queue and stores in the Metrics Store.
# The Transform Engine consumes the metrics from the Message Queue, performs transform and aggregation operations on metrics, and publishes metrics that it creates back to Message Queue.
# The Threshold Engine consumes metrics from the Message Queue and evaluates alarms. If a state change occurs in an alarm, an "alarm-state-transitioned-event" is published to the Message Queue.
# The Notification Engine consumes "alarm-state-transitioned-events" from the Message Queue, evaluates whether they have a Notification Method associated with it, and sends the appropriate notification, such as email.
# The Persister consumes the "alarm-state-transitioned-event" from the Message Queue and stores it in the Alarm State History Store.

== Alarm Managers ==
This section describes the new features including alarm grouping, inhibition and silencing.

Examples:

* Silence
SilenceRule1 = '{"alarm-silencing-definition-created": {"name": "silence_rule_1", "matchers": {"severity": "LOW"}, "start_time": "2017-02-21 20:00:00", "end_time": "2017-02-21 22:00:00"}}'
Two alarm transitions: AT1 and AT2
AT1_severity = HIGH
AT2_severity = LOW

Output: AT2 get silenced and AT1 send a notification.
SilenceRule1 expires after "end_time"="2017-02-21 22:00:00".

* Inhibit
InhibitionRule1 = '{"alarm-inhibition-definition-created": {"name": "inhibit_rule_1","source":{"severity":"HIGH"},"target":{"severity":"LOW"}, "equals":["tenantId"], "exclusions": {"alarm_name": "vm is dead"}}}'
Three alarm transitions: AT1, AT2 and AT3
AT1_tenantId = "d42bc"
AT2_tenantId = "d42bc"
AT3_tenantId = "d42bc"
AT1_severity = HIGH
AT2_severity = LOW
AT3_severity = LOW
AT1_alarm_name = "cpu high"
AT2_alarm_name = "memory high"
AT3_alarm_name = "vm is dead"
AT1_state = ALARM
AT2_state = ALARM
AT3_state = ALARM

Output: AT1 is the source alarm which will send a notification. AT2 is the target alarm and will get inhibited. AT3 matches the exclusions and will send a notification immediately.

* Grouping Scenario 1

GroupingRule1 = '{"alarm-grouping-definition-created":
{"name": "group_rule_1", "matchers": ["hostname"], "id": "b7163","repeat_interval": "2h", "group_wait": "30s", "exclusions": {"alarm_name": "cpu_percent_high"}, "tenantId": " d42bc", "alarm_actions": ["cd892"], "ok_actions": ["ad892"], "undetermined_actions": ["cf892"]}}'

Three alarm transitions: AT1, AT2 and AT3

AT1_hostname = host1
AT2_hostname = host1
AT3_hostname = host2
AT1_alarm_name = cpu_percent_high
AT2_alarm_name = cpu_system_perc_high
AT3_alarm_name = cpu_percent_high
AT1_state = ALARM
AT2_state = ALARM
AT3_state = ALARM

Output: AT1 and AT3 match exclusions and send notifications immediately. Generate a grouped notification “group_notification_rule_1_host1_alarm[1]” and send out using alarm_actions ["cd892"]. There are no alarm_actions, ok_actions or undermined_actions associated with the AT1, AT2, AT3 alarm definitions.

* Grouping Scenario 2

GroupingRule2 = '{"alarm-grouping-definition-created":
{"name": "group_rule_2", "matchers": ["hostname"], "id": "b7163","repeat_interval": "2h", "group_wait": "30s", "tenantId": " d42bc", "alarm_actions": ["cd892"], "ok_actions": ["ad892"], "undetermined_actions": ["cf892"]}}'

Three alarm transitions: AT1 with "alarm_actions": ["123ab"], AT2 with "alarm_actions": ["cd839"] and AT3

AT1_hostname = host1
AT2_hostname = host1
AT3_hostname = host2
AT1_state = ALARM
AT2_state = ALARM
AT3_state = ALARM

Output: Generate two grouped notifications “group_notification_rule_2_host1_alarm[2]” and “group_notification_rule_2_host2_alarm[1]”. Both using alarm_actions ["cd892"]. Also since AT1 and AT2 has their own alarm actions associated with them, there will be two more notifications sent out.

* Silenced and Grouped

SilenceRule1 = '{"alarm-silencing-definition-created": {"name": "silence_rule_1", "matchers": {"severity": "LOW"}, "start_time": "1487269470498", "end_time": "1587269470498"}}'

GroupingRule2 = '{"alarm-grouping-definition-created":
{"name": "group_rule_2", "matchers": ["hostname"], "id": "b7163","repeat_interval": "2h", "group_wait": "30s", "tenantId": " d42bc", "alarm_actions": ["cd892"], "ok_actions": ["ad892"], "undetermined_actions": ["cf892"]}}'

Four alarm transitions: AT1, AT2, AT3 and AT4

Silencing rule:
AT1_severity = HIGH
AT2_severity = LOW
AT3_severity = HIGH
AT4_severity = HIGH

Grouping rule:
AT1_hostname = host1
AT2_hostname = host1
AT3_hostname = host2
AT4_hostname = host1
AT1_state = ALARM
AT2_state = ALARM
AT3_state = OK
AT4_state = ALARM

Output: Generate two grouped notifications “group_notification_rule_2_host1_alarm[2]” using alarm action "cd892" and “group_notification_rule_2_host2_ok[1]” using ok action "ad892". AT2 is silenced so it isn't include in group_notification_rule_2_host1 count. There are no alarm_actions, ok_actions or undermined_actions associated with the AT1, AT2, AT3, AT4 alarm definitions.

* Silenced and inhibited (source alarm get silenced)

SilenceRule2 = '{"alarm-silencing-definition-created": {"name": "silence_rule_2", "matchers": {"severity": "HIGH, "hostname": "host1"}, "start_time": "2017-02-21 15:00:00", "end_time": "2017-02-21 21:00:00"}}'

InhibitionRule1 = '{"alarm-inhibition-definition-created": {"name": "inhibit_rule_1","source":{"severity":"HIGH"},"target":{"severity":"LOW"}, "equals":["tenantId"]}}'

Alarm transitions: AT1, AT2

Inhibition rule:
AT1_state = ALARM
AT2_state = ALARM
AT1_tenantId = "d42bc"
AT2_tenantId = "d42bc"
AT1_severity = HIGH
AT2_severity = LOW

Silencing rule:
AT1_hostname = host1
AT2_hostname = host2
AT1_severity = HIGH
AT2_severity = LOW

Output: no notification sent out. For inhibition, AT1 is the source alarm, AT2 is the target alarm. But at the same time, AT1 get silenced because it matches the silence rule.

* Inhibited and grouped

InhibitionRule1 = '{"alarm-inhibition-definition-created": {"name": "inhibit_rule_1","source":{"severity":"HIGH"},"target":{"severity":"LOW"}, "equals":["tenantId"]}}'

GroupingRule1 = '{"alarm-grouping-definition-created": {"name": "group_rule_1", "matchers": ["hostname"], "id": "b7163", "repeat_interval": "2h", "group_wait": "30s", "tenantId": " d42bc", "alarm_actions": ["cd892"], "ok_actions": ["ad892"], "undetermined_actions": ["cf892"]}}'

Alarm transitions: AT1, AT2, AT3

Inhibition rule:
AT1_tenantId = "d42bc"
AT2_tenantId = "d42bc"
AT3_tenantId = "d42bc"
AT1_severity = HIGH
AT2_severity = LOW
AT3_severity = HIGH
AT1_state = OK
AT2_state = ALARM
AT3_state = ALARM

Grouping rule:
AT1_hostname = host1
AT2_hostname = host2
AT3_hostname = host1
AT1_state = OK
AT2_state = ALARM
AT3_state = ALARM

Output: AT2 gets inhibited because its severity is low. AT3 is the source alarm. Since AT1 is in OK state, it is not a source alarm. For grouping, AT1 and AT3 has the same host name but different state. So there will be two grouped notifications sent out: “group_notification_rule_1_host1_ok[1]” and “group_notification_rule_1_host1_alarm[1]”.

* Silenced, inhibited and grouped

SilenceRule2 = '{"alarm-silencing-definition-created": {"name": "silence_rule_2", "matchers": {"severity": "HIGH, "hostname": "host1"}, "start_time": "2017-02-21 15:00:00", "end_time": "2017-02-21 21:00:00"}}'

InhibitionRule1 = '{"alarm-inhibition-definition-created": {"name": "inhibit_rule_1","source":{"severity":"HIGH"},"target":{"severity":"LOW"}, "equals":["tenantId"]}}'

GroupingRule1 = '{"alarm-grouping-definition-created": {"name": "group_rule_1", "matchers": ["hostname"], "id": "b7163", "repeat_interval": "2h", "group_wait": "30s", "tenantId": " d42bc", "alarm_actions": ["cd892"], "ok_actions": ["ad892"], "undetermined_actions": ["cf892"]}}'

Alarm transitions: AT1, AT2, AT3 and AT5

Silence rule:
AT1_severity = HIGH AT1_hostname = host1 (silenced)
AT2_severity = LOW AT2_hostname = host2
AT3_severity = HIGH AT3_hostname = host1(silenced)
AT5_severity = HIGH AT5_hostname = host3

Inhibition rule:
AT1_tenantId = "d42bc"
AT2_tenantId = "d42bc"
AT3_tenantId = "d42bc"
AT5_tenantId = "d42bc"
AT1_state = ALARM
AT2_state = ALARM
AT3_state = OK
AT5_state = UNDETERMINED
AT1_severity = HIGH (source)
AT2_severity = LOW (target)
AT3_severity = HIGH
AT5_severity = HIGH

Grouping rule:
AT1_hostname = host1
AT2_hostname = host2
AT3_hostname = host1
AT5_hostname = host3

Output: AT1 is in "group_notification_rule_1_host1_alarm" group and silenced. AT3 is in "group_notification_rule_1_host1_ok" group and silenced. AT2 is in "group_notification_rule_1_host2_alarm" group and inhibited. AT5 is in "group_notification_rule_1_host3_undetermined" group and will send notification “group_notification_rule_1_host3_undetermined[1]” using undetermined action "cf892".

= Development Environment =

* Monasca DevStack Plugin:
** DevStack is the primary developmement environment for OpenStack.
***See http://docs.openstack.org/developer/devstack/
** The Monasca DevStack plugin installs the Monasca Service, Agent, Horizon Monitoring Panel, and Grafana
** README at, https://github.com/openstack/monasca-api/tree/master/devstack
** Best way to get started is to use Vagrant with the Vagrantfile at, https://github.com/openstack/monasca-api/blob/master/devstack/Vagrantfile.

* Project and Bug tracking
** Monasca on Storyboard https://storyboard.openstack.org/#!/project/list?q=monasca
** Monasca on LaunchPad. https://launchpad.net/monasca (history only, Storyboard is the official tracking since 2017)

* Monasca projects source code. https://git.openstack.org/cgit/?q=monasca

= Coding Standards =

* Python: All Python code conforms to the OpenStack standards at, http://docs.openstack.org/developer/hacking/.
** Note, all components in Monasca, except for the Threshold Engine, have been ported to Python.

* Java: Several of the Monasca components are available as Java. OpenStack does not have any Java coding standards. We've adopted the Google Java Style at, https://google.github.io/styleguide/javaguide.html.
** The standard says either 80 or 100 length lines. We've adopted 100.

= Technologies =

Monasca uses a number of third-party technologies:

* Internal Processing and Middleware
** Apache Kafka (http://kafka.apache.org): Apache Kafka is publish-subscribe messaging rethought as a distributed commit log. Kafka is a highly performant, distributed, fault-tolerant, and scalable message queue with durability built-in.
** Apache Storm (http://storm.incubator.apache.org/): Apache Storm is a free and open source distributed realtime computation system. Storm makes it easy to reliably process unbounded streams of data, doing for realtime processing what Hadoop did for batch processing.
** ZooKeeper (http://zookeeper.apache.org/): Used by Kafka and Storm.
** Apache Spark: Used by Monasca Transform as an aggregation engine.

* Configuration database:
** MySQL: MySQL is supported as a Config Database.
** PostgreSQL: Support for POSTgres, via Hibernate and SQLAlchemy, for the Config Database.

* Vagrant (http://www.vagrantup.com/): Vagrant provides easy to configure, reproducible, and portable work environments built on top of industry-standard technology and controlled by a single consistent workflow to help maximize the productivity and flexibility of you and your team.

* Dropwizard (https://dropwizard.github.io/dropwizard/): Dropwizard pulls together stable, mature libraries from the Java ecosystem into a simple, light-weight package that lets you focus on getting things done. Dropwizard has out-of-the-box support for sophisticated configuration, application metrics, logging, operational tools, and much more, allowing you and your team to ship a production-quality web service in the shortest time possible.

* Time Series Database:
** InfluxDB (http://influxdb.com/): An open-source distributed time series database with no external dependencies. InfluxDB is supported for the Metrics Database.
** Vertica (http://www.vertica.com): A commercial Enterprise class SQL analytics database that is highly scalable. It offers built-in automatic high-availability and excels at in-database analytics and compressing and storing massive amounts of data. A free community version of Vertica is available that can store up to 1 TB of data with no time-limit is available at, https://my.vertica.com/community/. Vertica is supported for the Metrics Database, though no longer commonly used.
** Cassandra(https://cassandra.apache.org): Cassandra is supported for the Metrics Database.

= License =

Copyright (c) 2014, 2015 Hewlett-Packard Development Company, L.P.

(C) Copyright 2019 SUSE LLC

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied.
See the License for the specific language governing permissions and
limitations under the License.

[[File:yklogo.png|200px|thumb|left|Monasca uses YourKit Profiler for Java development]]

 
 
 
 

[https://www.yourkit.com/ Visit YourKit website for more information]

Monasca

2019-05-08T17:08:56Z

Joseph Davis: /* License */

== Overview ==

Monasca is a open-source multi-tenant, highly scalable, performant, fault-tolerant monitoring-as-a-service solution that integrates with OpenStack. It uses a REST API for high-speed metrics processing and querying and has a streaming alarm engine and notification engine.

[[File:OpenStack_Project_Monasca_vertical.png|thumb|right]]

==== Contribution ====
* [https://storyboard.openstack.org/#!/project_group/59 Project Group on StoryBoard]
* [https://storyboard.openstack.org/#!/board/111 Important activities in Stein release]
* [https://storyboard.openstack.org/#!/board/141 Important activities in Train release]
* [https://docs.openstack.org/monasca-api/latest/contributor/index.html Contribution Guidelines]

==== Communication and Meetings ====
* IRC: #openstack-monasca on freenode.net
* Weekly Meetings:
** http://eavesdrop.openstack.org/#Monasca_Team_Meeting
** [[Meetings/Monasca]]
* Use [Monasca] tag on [http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-discuss openstack-discuss mailing list].

===== PTG meetings =====
* [[MonascaSteinPTG|Stein PTG Summary (Denver)]]
* (coming soon - Train summary)

==== Documentation ====
: Monasca API Specification: https://github.com/openstack/monasca-api/blob/master/docs/monasca-api-spec.md
: Agent Documentation: https://github.com/openstack/monasca-agent

==== Presentations ====
There have been many interesting and excellent presentations on Monasca, many given at the OpenStack or OpenInfra Summits.
: [[Monasca/Presentations]] (updated with Denver 2019 links)

==== Repositories ====

: '''Code'''
:: Core: https://git.openstack.org/cgit/?q=monasca (old git link)
:: API: https://opendev.org/openstack/monasca-api

: '''Deployment'''
: The following repositories are available for deploying Monasca:
:: Docker: https://github.com/monasca/monasca-docker
:: Kubernetes: https://github.com/monasca/monasca-helm
:: Ansible: https://github.com/search?utf8=%E2%9C%93&q=ansible-monasca
:: Puppet: https://git.openstack.org/openstack/puppet-monasca

==== Bugs ====

===== All open bugs =====
: https://storyboard.openstack.org/#!/worklist/213

===== In Stein triaged bugs =====
: https://storyboard.openstack.org/#!/worklist/467

===== Bug-fixing progress =====
: https://storyboard.openstack.org/#!/board/114

== Requirements ==
[[Monasca/Requirements]].

See also https://opendev.org/openstack/monasca-specs/

== Features ==

This section describes the overall features.

* A highly performant, scalable, reliable and fault-tolerant Monitoring as a Service (MONaaS) solution that scales to service provider metrics levels of metrics throughput. Performance, scalability and high-availability have been designed in from the start. Can process 100s of thousands of metrics/sec as well as offer data retention periods of greater than a year with no data loss while still processing interactive queries.

* Rest API for storing and querying metrics and historical information. Most monitoring solution use special transports and protocols, such as CollectD or NSCA (Nagios). In our solution, http is the only protocol used. This simplifies the overall design and also allows for a much richer way of describing the data via dimensions.

* Multi-tenant and authenticated. Metrics are submitted and authenticated using Keystone and stored associated with a tenant ID.

* Metrics defined using a set of (key, value) pairs called dimensions.

* Real-time thresholding and alarming on metrics.

* Compound alarms described using a simple expressive grammar composed of alarm sub-expressions and logical operators.

* Monitoring agent that supports a number of built-in system and service checks and also supports Nagios checks and statsd.

* Open-source monitoring solution built on open-source technologies.

=== Comparisons to alternatives ===
[[Monasca/Comparison]]

== Architecture ==
[[File:monasca-arch-component-diagram.png|Monasca Architecture Component Diagram]]
* Monitoring Agent (monasca-agent): A modern Python based monitoring agent that consists of several sub-components and supports system metrics, such as cpu utilization and available memory, Nagios plugins, statsd and many built-in checks for services such as MySQL, RabbitMQ, and many others.
* Monitoring API (monasca-api): A well-defined and documented RESTful API for monitoring that is primarily focused on the following concepts and areas:
** Metrics: Store and query massive amounts of metrics in real-time.
** Statistics: Query statistics for metrics.
** Alarm Definitions: Create, update, query and delete alarm definitions.
** Alarms: Query and delete the alarm history.
***Simple expressive grammar for creating compound alarms composed of alarm subexpressions and logical operators.
***Alarm severities can be associated with alarms.
***The complete alarm state transition history is stored and queryable which allows for subsequent root cause analysis (RCA) or advanced analytics.
**Notification Methods: Create and delete notification methods and associate them with alarms, such as email. Supports the ability to notify users directly via email when an alarm state transitions occur.
** The Monasca API has both Java and Python implementations avaialble.
* Persister (monasca-persister): Consumes metrics and alarm state transitions from the MessageQ and stores them in the Metrics and Alarms database.
** The Persister has both Java and Python implementations.
* Transform and Aggregation Engine (monasca-transform): Transform metric names and values, such as delta or time-based derivative calculations, and creates new metrics that are published to the Message Queue. The Transform Engine is not available yet.
* Anomaly and Prediction Engine: Evaluates prediction and anomalies and generates predicted metrics as well as anomaly likelihood and anomaly scores. The Anomaly and Prediction Engine is currently in a prototype status.
* Threshold Engine (monasca-thresh): Computes thresholds on metrics and publishes alarms to the MessageQ when exceeded. Based on Apache Storm a free and open distributed real-time computation system.
* Notification Engine (monasca-notification): Consumes alarm state transition messages from the MessageQ and sends notifications, such as emails for alarms. The Notification Engine is Python based.
* Analytics Engine (monasca-analytics): Consumes alarm state transisitions and metrics from the MessageQ and does anomaly detection and alarm clustering/correlation.
* Message Queue: A third-party component that primarily receives published metrics from the Monitoring API and alarm state transition messages from the Threshold Engine that are consumed by other components, such as the Persister and Notification Engine. The Message Queue is also used to publish and consume other events in the system. Currently, a Kafka based MessageQ is supported. Kafka is a high performance, distributed, fault-tolerant, and scalable message queue with durability built-in. We will look at other alternatives, such as RabbitMQ and in-fact in our previous implementation RabbitMQ was supported, but due to performance, scale, durability and high-availability limitiations with RabbitMQ we have moved to Kafka.
* Metrics and Alarms Database: A third-party component that primarily stores metrics and the alarm state history. Currently, Vertica and InfluxDB are supported. Support for Cassandra is in progress.
* Config Database: A third-party component that stores a lot of the configuration and other information in the system. Currently, MySQL is supported. Support for PostgreSQL is in progress.
* Monitoring Client (python-monascaclient): A Python command line client and library that communicates and controls the Monitoring API. The Monitoring Client was written using the OpenStack Heat Python client as a framework. The Monitoring Client also has a Python library, "monascaclient" similar to the other OpenStack clients, that can be used to quickly build additional capabilities. The Monitoring Client library is used by the Monitoring UI, Ceilometer publisher, and other components.
* Monitoring UI: A Horizon dashboard for visualizing the overall health and status of an OpenStack cloud.
* Ceilometer publisher: A multi-publisher plugin for Ceilometer, not shown, that converts and publishes samples to the Monitoring API.
Most of the components are described in their respective repositories. However, there aren't any repositories for the third-party components used, so we describe some of the relevant details here.

Further historical context for the architecture can be found at [[Monasca/Architecture Details]]

=== Message Schema ===
[[Monasca/Message Schema]]

=== Message Queue ===
A distributed, performant, scalable, HA message queue for distributing metrics, alarms and events in the monitoring system. Currently, based on Kafka.

==== Messages ====
There are several messages that are published and consumed by various components in Monasca via the MessageQ. See [[Monasca/Message Schema|Message Schema]].

=== Metrics and Alarms Database ===

A high-performance analytics database that can store massive amounts of metrics and alarms in real-time and also support interactive queries. Currently Vertica and InfluxDB are supported.

The SQL schema that is used by Vertica is as follows:

* MonMetrics.Measurements: Stores the actual measurements that are sent.
** id: An integer ID for the measurement.
** definition_dimensions_id: A reference to DefinitionDimensions.
** time_stamp
** value
* MonMetrics.DefinitionDimensions
** id: A sha1 hash of (defintion_id, dimension_set_id)
** definition_id: A reference to the Definitions.id
** dimension_set_id: A reference to the Dimensions.dimension_set_id
* MonMetrics.Definitions
** id: A sha1 hash of the (name, tenant_id, region)
** name: Name of the metric.
** tenant_id: The tenant_id that submitted the metric.
** region: The region the metric was submitted under.
* MonMetric.Dimensions
** dimension_set_id: A sha1 hash of the set of dimenions for a metric.
** name: Name of dimension.
** value: Value of dimension.

=== Config Database ===
The config database store all the configuration information. Currently based on MySQL.

The SQL schema is as follows:

* alarm
** id
** tenant_id
** name
** description
** expression
** state
** actions_enabled
** created_at
** updated_at
** deleted_at
* alarm_action
** alarm_id
** alarm_state
** action_id
* notification_method
** id
** tenant_id
** name
** type
** address
** created_at
** updated_at
* sub_alarm
** id
** alarm_id
** function
** metric_name
** operator
** threshold
** period
** periods
** state
** created_at
** updated_at
* sub_alarm_dimension
** sub_alarm_id
** dimension_name
** value

== Events ==
Support for real-time event stream processing in Monasca is in progress. For more details see the link at, [[Monasca/Events]].

== Logging ==
Support for logging in Monasca is under discussion. For more details see the link at, [[Monasca/Logging]].

== Transform and Aggregation Engine ==
For more details see the link at, [[Monasca/Transform]].

== Analytics ==
Support for anomaly detection and alarm clustering/correlation is in progress. For more details see the link at, [[Monasca/Analytics]].

== Monitoring ==
Enablement and usage for monitoring the status of Monasca is under discussion. For more details see the link at, [[Monasca/Monitoring_Of_Monasca]]

== UI/UX Support ==
Adding more support for common UI/UX queries is under discussion. For more details see the link at, [[Monasca/UI_UX_Support]]

== Keystone Requirements ==
Monasca relies on keystone for running and there are requirements about which keystone configuration must exist.

* The endpoint for the api must be registered in keystone as the 'monasca' service.
* The api must have an admin token to use in verifying the keystone tokens it receives.
* For each project which uses Monasca two users must exist, one will be in the 'monasca-agent' role and be used by the monasca-agent's running on machines. The other should not be in that role and can be used logging into the UI, using the CLI or for direct queries against the API.

== Post Metric Sequence ==

This section describes the sequence of operations involved in posting a metric to the Monasca API.

[[File:monasca-arch-post-metric-diagram.png|Monasca Architecture Post Metric Diagram]]

# A metric is posted to the Monasca API.
# The Monasca API authenticates and validates the request and publishes the metric to the the Message Queue.
# The Persister consumes the metric from the Message Queue and stores in the Metrics Store.
# The Transform Engine consumes the metrics from the Message Queue, performs transform and aggregation operations on metrics, and publishes metrics that it creates back to Message Queue.
# The Threshold Engine consumes metrics from the Message Queue and evaluates alarms. If a state change occurs in an alarm, an "alarm-state-transitioned-event" is published to the Message Queue.
# The Notification Engine consumes "alarm-state-transitioned-events" from the Message Queue, evaluates whether they have a Notification Method associated with it, and sends the appropriate notification, such as email.
# The Persister consumes the "alarm-state-transitioned-event" from the Message Queue and stores it in the Alarm State History Store.

== Alarm Managers ==
This section describes the new features including alarm grouping, inhibition and silencing.

Examples:

* Silence
SilenceRule1 = '{"alarm-silencing-definition-created": {"name": "silence_rule_1", "matchers": {"severity": "LOW"}, "start_time": "2017-02-21 20:00:00", "end_time": "2017-02-21 22:00:00"}}'
Two alarm transitions: AT1 and AT2
AT1_severity = HIGH
AT2_severity = LOW

Output: AT2 get silenced and AT1 send a notification.
SilenceRule1 expires after "end_time"="2017-02-21 22:00:00".

* Inhibit
InhibitionRule1 = '{"alarm-inhibition-definition-created": {"name": "inhibit_rule_1","source":{"severity":"HIGH"},"target":{"severity":"LOW"}, "equals":["tenantId"], "exclusions": {"alarm_name": "vm is dead"}}}'
Three alarm transitions: AT1, AT2 and AT3
AT1_tenantId = "d42bc"
AT2_tenantId = "d42bc"
AT3_tenantId = "d42bc"
AT1_severity = HIGH
AT2_severity = LOW
AT3_severity = LOW
AT1_alarm_name = "cpu high"
AT2_alarm_name = "memory high"
AT3_alarm_name = "vm is dead"
AT1_state = ALARM
AT2_state = ALARM
AT3_state = ALARM

Output: AT1 is the source alarm which will send a notification. AT2 is the target alarm and will get inhibited. AT3 matches the exclusions and will send a notification immediately.

* Grouping Scenario 1

GroupingRule1 = '{"alarm-grouping-definition-created":
{"name": "group_rule_1", "matchers": ["hostname"], "id": "b7163","repeat_interval": "2h", "group_wait": "30s", "exclusions": {"alarm_name": "cpu_percent_high"}, "tenantId": " d42bc", "alarm_actions": ["cd892"], "ok_actions": ["ad892"], "undetermined_actions": ["cf892"]}}'

Three alarm transitions: AT1, AT2 and AT3

AT1_hostname = host1
AT2_hostname = host1
AT3_hostname = host2
AT1_alarm_name = cpu_percent_high
AT2_alarm_name = cpu_system_perc_high
AT3_alarm_name = cpu_percent_high
AT1_state = ALARM
AT2_state = ALARM
AT3_state = ALARM

Output: AT1 and AT3 match exclusions and send notifications immediately. Generate a grouped notification “group_notification_rule_1_host1_alarm[1]” and send out using alarm_actions ["cd892"]. There are no alarm_actions, ok_actions or undermined_actions associated with the AT1, AT2, AT3 alarm definitions.

* Grouping Scenario 2

GroupingRule2 = '{"alarm-grouping-definition-created":
{"name": "group_rule_2", "matchers": ["hostname"], "id": "b7163","repeat_interval": "2h", "group_wait": "30s", "tenantId": " d42bc", "alarm_actions": ["cd892"], "ok_actions": ["ad892"], "undetermined_actions": ["cf892"]}}'

Three alarm transitions: AT1 with "alarm_actions": ["123ab"], AT2 with "alarm_actions": ["cd839"] and AT3

AT1_hostname = host1
AT2_hostname = host1
AT3_hostname = host2
AT1_state = ALARM
AT2_state = ALARM
AT3_state = ALARM

Output: Generate two grouped notifications “group_notification_rule_2_host1_alarm[2]” and “group_notification_rule_2_host2_alarm[1]”. Both using alarm_actions ["cd892"]. Also since AT1 and AT2 has their own alarm actions associated with them, there will be two more notifications sent out.

* Silenced and Grouped

SilenceRule1 = '{"alarm-silencing-definition-created": {"name": "silence_rule_1", "matchers": {"severity": "LOW"}, "start_time": "1487269470498", "end_time": "1587269470498"}}'

GroupingRule2 = '{"alarm-grouping-definition-created":
{"name": "group_rule_2", "matchers": ["hostname"], "id": "b7163","repeat_interval": "2h", "group_wait": "30s", "tenantId": " d42bc", "alarm_actions": ["cd892"], "ok_actions": ["ad892"], "undetermined_actions": ["cf892"]}}'

Four alarm transitions: AT1, AT2, AT3 and AT4

Silencing rule:
AT1_severity = HIGH
AT2_severity = LOW
AT3_severity = HIGH
AT4_severity = HIGH

Grouping rule:
AT1_hostname = host1
AT2_hostname = host1
AT3_hostname = host2
AT4_hostname = host1
AT1_state = ALARM
AT2_state = ALARM
AT3_state = OK
AT4_state = ALARM

Output: Generate two grouped notifications “group_notification_rule_2_host1_alarm[2]” using alarm action "cd892" and “group_notification_rule_2_host2_ok[1]” using ok action "ad892". AT2 is silenced so it isn't include in group_notification_rule_2_host1 count. There are no alarm_actions, ok_actions or undermined_actions associated with the AT1, AT2, AT3, AT4 alarm definitions.

* Silenced and inhibited (source alarm get silenced)

SilenceRule2 = '{"alarm-silencing-definition-created": {"name": "silence_rule_2", "matchers": {"severity": "HIGH, "hostname": "host1"}, "start_time": "2017-02-21 15:00:00", "end_time": "2017-02-21 21:00:00"}}'

InhibitionRule1 = '{"alarm-inhibition-definition-created": {"name": "inhibit_rule_1","source":{"severity":"HIGH"},"target":{"severity":"LOW"}, "equals":["tenantId"]}}'

Alarm transitions: AT1, AT2

Inhibition rule:
AT1_state = ALARM
AT2_state = ALARM
AT1_tenantId = "d42bc"
AT2_tenantId = "d42bc"
AT1_severity = HIGH
AT2_severity = LOW

Silencing rule:
AT1_hostname = host1
AT2_hostname = host2
AT1_severity = HIGH
AT2_severity = LOW

Output: no notification sent out. For inhibition, AT1 is the source alarm, AT2 is the target alarm. But at the same time, AT1 get silenced because it matches the silence rule.

* Inhibited and grouped

InhibitionRule1 = '{"alarm-inhibition-definition-created": {"name": "inhibit_rule_1","source":{"severity":"HIGH"},"target":{"severity":"LOW"}, "equals":["tenantId"]}}'

GroupingRule1 = '{"alarm-grouping-definition-created": {"name": "group_rule_1", "matchers": ["hostname"], "id": "b7163", "repeat_interval": "2h", "group_wait": "30s", "tenantId": " d42bc", "alarm_actions": ["cd892"], "ok_actions": ["ad892"], "undetermined_actions": ["cf892"]}}'

Alarm transitions: AT1, AT2, AT3

Inhibition rule:
AT1_tenantId = "d42bc"
AT2_tenantId = "d42bc"
AT3_tenantId = "d42bc"
AT1_severity = HIGH
AT2_severity = LOW
AT3_severity = HIGH
AT1_state = OK
AT2_state = ALARM
AT3_state = ALARM

Grouping rule:
AT1_hostname = host1
AT2_hostname = host2
AT3_hostname = host1
AT1_state = OK
AT2_state = ALARM
AT3_state = ALARM

Output: AT2 gets inhibited because its severity is low. AT3 is the source alarm. Since AT1 is in OK state, it is not a source alarm. For grouping, AT1 and AT3 has the same host name but different state. So there will be two grouped notifications sent out: “group_notification_rule_1_host1_ok[1]” and “group_notification_rule_1_host1_alarm[1]”.

* Silenced, inhibited and grouped

SilenceRule2 = '{"alarm-silencing-definition-created": {"name": "silence_rule_2", "matchers": {"severity": "HIGH, "hostname": "host1"}, "start_time": "2017-02-21 15:00:00", "end_time": "2017-02-21 21:00:00"}}'

InhibitionRule1 = '{"alarm-inhibition-definition-created": {"name": "inhibit_rule_1","source":{"severity":"HIGH"},"target":{"severity":"LOW"}, "equals":["tenantId"]}}'

GroupingRule1 = '{"alarm-grouping-definition-created": {"name": "group_rule_1", "matchers": ["hostname"], "id": "b7163", "repeat_interval": "2h", "group_wait": "30s", "tenantId": " d42bc", "alarm_actions": ["cd892"], "ok_actions": ["ad892"], "undetermined_actions": ["cf892"]}}'

Alarm transitions: AT1, AT2, AT3 and AT5

Silence rule:
AT1_severity = HIGH AT1_hostname = host1 (silenced)
AT2_severity = LOW AT2_hostname = host2
AT3_severity = HIGH AT3_hostname = host1(silenced)
AT5_severity = HIGH AT5_hostname = host3

Inhibition rule:
AT1_tenantId = "d42bc"
AT2_tenantId = "d42bc"
AT3_tenantId = "d42bc"
AT5_tenantId = "d42bc"
AT1_state = ALARM
AT2_state = ALARM
AT3_state = OK
AT5_state = UNDETERMINED
AT1_severity = HIGH (source)
AT2_severity = LOW (target)
AT3_severity = HIGH
AT5_severity = HIGH

Grouping rule:
AT1_hostname = host1
AT2_hostname = host2
AT3_hostname = host1
AT5_hostname = host3

Output: AT1 is in "group_notification_rule_1_host1_alarm" group and silenced. AT3 is in "group_notification_rule_1_host1_ok" group and silenced. AT2 is in "group_notification_rule_1_host2_alarm" group and inhibited. AT5 is in "group_notification_rule_1_host3_undetermined" group and will send notification “group_notification_rule_1_host3_undetermined[1]” using undetermined action "cf892".

= Development Environment =

* Monasca DevStack Plugin:
** DevStack is the primary developmement environment for OpenStack.
***See http://docs.openstack.org/developer/devstack/
** The Monasca DevStack plugin installs the Monasca Service, Agent, Horizon Monitoring Panel, and Grafana
** README at, https://github.com/openstack/monasca-api/tree/master/devstack
** Best way to get started is to use Vagrant with the Vagrantfile at, https://github.com/openstack/monasca-api/blob/master/devstack/Vagrantfile.

* Monasca on LaunchPad. https://launchpad.net/monasca

* Monasca projects source code. https://git.openstack.org/cgit/?q=monasca

= Coding Standards =

* Python: All Python code conforms to the OpenStack standards at, http://docs.openstack.org/developer/hacking/.
** Note, all components in Monasca, except for the Threshold Engine, have been ported to Python.

* Java: Several of the Monasca components are available as Java. OpenStack does not have any Java coding standards. We've adopted the Google Java Style at, https://google.github.io/styleguide/javaguide.html.
** The standard says either 80 or 100 length lines. We've adopted 100.

= Technologies =

Monasca uses a number of third-party technologies:

* Internal Processing and Middleware
** Apache Kafka (http://kafka.apache.org): Apache Kafka is publish-subscribe messaging rethought as a distributed commit log. Kafka is a highly performant, distributed, fault-tolerant, and scalable message queue with durability built-in.
** Apache Storm (http://storm.incubator.apache.org/): Apache Storm is a free and open source distributed realtime computation system. Storm makes it easy to reliably process unbounded streams of data, doing for realtime processing what Hadoop did for batch processing.
** ZooKeeper (http://zookeeper.apache.org/): Used by Kafka and Storm.
** Apache Spark: Used by Monasca Transform as an aggregation engine.

* Configuration database:
** MySQL: MySQL is supported as a Config Database.
** PostgreSQL: Support for POSTgres, via Hibernate and SQLAlchemy, for the Config Database.

* Vagrant (http://www.vagrantup.com/): Vagrant provides easy to configure, reproducible, and portable work environments built on top of industry-standard technology and controlled by a single consistent workflow to help maximize the productivity and flexibility of you and your team.

* Dropwizard (https://dropwizard.github.io/dropwizard/): Dropwizard pulls together stable, mature libraries from the Java ecosystem into a simple, light-weight package that lets you focus on getting things done. Dropwizard has out-of-the-box support for sophisticated configuration, application metrics, logging, operational tools, and much more, allowing you and your team to ship a production-quality web service in the shortest time possible.

* Time Series Database:
** InfluxDB (http://influxdb.com/): An open-source distributed time series database with no external dependencies. InfluxDB is supported for the Metrics Database.
** Vertica (http://www.vertica.com): A commercial Enterprise class SQL analytics database that is highly scalable. It offers built-in automatic high-availability and excels at in-database analytics and compressing and storing massive amounts of data. A free community version of Vertica is available that can store up to 1 TB of data with no time-limit is available at, https://my.vertica.com/community/. Vertica is supported for the Metrics Database, though no longer commonly used.
** Cassandra(https://cassandra.apache.org): Cassandra is supported for the Metrics Database.

= License =

Copyright (c) 2014, 2015 Hewlett-Packard Development Company, L.P.

(C) Copyright 2019 SUSE LLC

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied.
See the License for the specific language governing permissions and
limitations under the License.

[[File:yklogo.png|200px|thumb|left|Monasca uses YourKit Profiler for Java development]]

 
 
 
 

[https://www.yourkit.com/ Visit YourKit website for more information]

Monasca

2019-05-08T17:08:40Z

Joseph Davis: /* License */ add SUSE (need to add Fujitsu, others)

== Overview ==

Monasca is a open-source multi-tenant, highly scalable, performant, fault-tolerant monitoring-as-a-service solution that integrates with OpenStack. It uses a REST API for high-speed metrics processing and querying and has a streaming alarm engine and notification engine.

[[File:OpenStack_Project_Monasca_vertical.png|thumb|right]]

==== Contribution ====
* [https://storyboard.openstack.org/#!/project_group/59 Project Group on StoryBoard]
* [https://storyboard.openstack.org/#!/board/111 Important activities in Stein release]
* [https://storyboard.openstack.org/#!/board/141 Important activities in Train release]
* [https://docs.openstack.org/monasca-api/latest/contributor/index.html Contribution Guidelines]

==== Communication and Meetings ====
* IRC: #openstack-monasca on freenode.net
* Weekly Meetings:
** http://eavesdrop.openstack.org/#Monasca_Team_Meeting
** [[Meetings/Monasca]]
* Use [Monasca] tag on [http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-discuss openstack-discuss mailing list].

===== PTG meetings =====
* [[MonascaSteinPTG|Stein PTG Summary (Denver)]]
* (coming soon - Train summary)

==== Documentation ====
: Monasca API Specification: https://github.com/openstack/monasca-api/blob/master/docs/monasca-api-spec.md
: Agent Documentation: https://github.com/openstack/monasca-agent

==== Presentations ====
There have been many interesting and excellent presentations on Monasca, many given at the OpenStack or OpenInfra Summits.
: [[Monasca/Presentations]] (updated with Denver 2019 links)

==== Repositories ====

: '''Code'''
:: Core: https://git.openstack.org/cgit/?q=monasca (old git link)
:: API: https://opendev.org/openstack/monasca-api

: '''Deployment'''
: The following repositories are available for deploying Monasca:
:: Docker: https://github.com/monasca/monasca-docker
:: Kubernetes: https://github.com/monasca/monasca-helm
:: Ansible: https://github.com/search?utf8=%E2%9C%93&q=ansible-monasca
:: Puppet: https://git.openstack.org/openstack/puppet-monasca

==== Bugs ====

===== All open bugs =====
: https://storyboard.openstack.org/#!/worklist/213

===== In Stein triaged bugs =====
: https://storyboard.openstack.org/#!/worklist/467

===== Bug-fixing progress =====
: https://storyboard.openstack.org/#!/board/114

== Requirements ==
[[Monasca/Requirements]].

See also https://opendev.org/openstack/monasca-specs/

== Features ==

This section describes the overall features.

* A highly performant, scalable, reliable and fault-tolerant Monitoring as a Service (MONaaS) solution that scales to service provider metrics levels of metrics throughput. Performance, scalability and high-availability have been designed in from the start. Can process 100s of thousands of metrics/sec as well as offer data retention periods of greater than a year with no data loss while still processing interactive queries.

* Rest API for storing and querying metrics and historical information. Most monitoring solution use special transports and protocols, such as CollectD or NSCA (Nagios). In our solution, http is the only protocol used. This simplifies the overall design and also allows for a much richer way of describing the data via dimensions.

* Multi-tenant and authenticated. Metrics are submitted and authenticated using Keystone and stored associated with a tenant ID.

* Metrics defined using a set of (key, value) pairs called dimensions.

* Real-time thresholding and alarming on metrics.

* Compound alarms described using a simple expressive grammar composed of alarm sub-expressions and logical operators.

* Monitoring agent that supports a number of built-in system and service checks and also supports Nagios checks and statsd.

* Open-source monitoring solution built on open-source technologies.

=== Comparisons to alternatives ===
[[Monasca/Comparison]]

== Architecture ==
[[File:monasca-arch-component-diagram.png|Monasca Architecture Component Diagram]]
* Monitoring Agent (monasca-agent): A modern Python based monitoring agent that consists of several sub-components and supports system metrics, such as cpu utilization and available memory, Nagios plugins, statsd and many built-in checks for services such as MySQL, RabbitMQ, and many others.
* Monitoring API (monasca-api): A well-defined and documented RESTful API for monitoring that is primarily focused on the following concepts and areas:
** Metrics: Store and query massive amounts of metrics in real-time.
** Statistics: Query statistics for metrics.
** Alarm Definitions: Create, update, query and delete alarm definitions.
** Alarms: Query and delete the alarm history.
***Simple expressive grammar for creating compound alarms composed of alarm subexpressions and logical operators.
***Alarm severities can be associated with alarms.
***The complete alarm state transition history is stored and queryable which allows for subsequent root cause analysis (RCA) or advanced analytics.
**Notification Methods: Create and delete notification methods and associate them with alarms, such as email. Supports the ability to notify users directly via email when an alarm state transitions occur.
** The Monasca API has both Java and Python implementations avaialble.
* Persister (monasca-persister): Consumes metrics and alarm state transitions from the MessageQ and stores them in the Metrics and Alarms database.
** The Persister has both Java and Python implementations.
* Transform and Aggregation Engine (monasca-transform): Transform metric names and values, such as delta or time-based derivative calculations, and creates new metrics that are published to the Message Queue. The Transform Engine is not available yet.
* Anomaly and Prediction Engine: Evaluates prediction and anomalies and generates predicted metrics as well as anomaly likelihood and anomaly scores. The Anomaly and Prediction Engine is currently in a prototype status.
* Threshold Engine (monasca-thresh): Computes thresholds on metrics and publishes alarms to the MessageQ when exceeded. Based on Apache Storm a free and open distributed real-time computation system.
* Notification Engine (monasca-notification): Consumes alarm state transition messages from the MessageQ and sends notifications, such as emails for alarms. The Notification Engine is Python based.
* Analytics Engine (monasca-analytics): Consumes alarm state transisitions and metrics from the MessageQ and does anomaly detection and alarm clustering/correlation.
* Message Queue: A third-party component that primarily receives published metrics from the Monitoring API and alarm state transition messages from the Threshold Engine that are consumed by other components, such as the Persister and Notification Engine. The Message Queue is also used to publish and consume other events in the system. Currently, a Kafka based MessageQ is supported. Kafka is a high performance, distributed, fault-tolerant, and scalable message queue with durability built-in. We will look at other alternatives, such as RabbitMQ and in-fact in our previous implementation RabbitMQ was supported, but due to performance, scale, durability and high-availability limitiations with RabbitMQ we have moved to Kafka.
* Metrics and Alarms Database: A third-party component that primarily stores metrics and the alarm state history. Currently, Vertica and InfluxDB are supported. Support for Cassandra is in progress.
* Config Database: A third-party component that stores a lot of the configuration and other information in the system. Currently, MySQL is supported. Support for PostgreSQL is in progress.
* Monitoring Client (python-monascaclient): A Python command line client and library that communicates and controls the Monitoring API. The Monitoring Client was written using the OpenStack Heat Python client as a framework. The Monitoring Client also has a Python library, "monascaclient" similar to the other OpenStack clients, that can be used to quickly build additional capabilities. The Monitoring Client library is used by the Monitoring UI, Ceilometer publisher, and other components.
* Monitoring UI: A Horizon dashboard for visualizing the overall health and status of an OpenStack cloud.
* Ceilometer publisher: A multi-publisher plugin for Ceilometer, not shown, that converts and publishes samples to the Monitoring API.
Most of the components are described in their respective repositories. However, there aren't any repositories for the third-party components used, so we describe some of the relevant details here.

Further historical context for the architecture can be found at [[Monasca/Architecture Details]]

=== Message Schema ===
[[Monasca/Message Schema]]

=== Message Queue ===
A distributed, performant, scalable, HA message queue for distributing metrics, alarms and events in the monitoring system. Currently, based on Kafka.

==== Messages ====
There are several messages that are published and consumed by various components in Monasca via the MessageQ. See [[Monasca/Message Schema|Message Schema]].

=== Metrics and Alarms Database ===

A high-performance analytics database that can store massive amounts of metrics and alarms in real-time and also support interactive queries. Currently Vertica and InfluxDB are supported.

The SQL schema that is used by Vertica is as follows:

* MonMetrics.Measurements: Stores the actual measurements that are sent.
** id: An integer ID for the measurement.
** definition_dimensions_id: A reference to DefinitionDimensions.
** time_stamp
** value
* MonMetrics.DefinitionDimensions
** id: A sha1 hash of (defintion_id, dimension_set_id)
** definition_id: A reference to the Definitions.id
** dimension_set_id: A reference to the Dimensions.dimension_set_id
* MonMetrics.Definitions
** id: A sha1 hash of the (name, tenant_id, region)
** name: Name of the metric.
** tenant_id: The tenant_id that submitted the metric.
** region: The region the metric was submitted under.
* MonMetric.Dimensions
** dimension_set_id: A sha1 hash of the set of dimenions for a metric.
** name: Name of dimension.
** value: Value of dimension.

=== Config Database ===
The config database store all the configuration information. Currently based on MySQL.

The SQL schema is as follows:

* alarm
** id
** tenant_id
** name
** description
** expression
** state
** actions_enabled
** created_at
** updated_at
** deleted_at
* alarm_action
** alarm_id
** alarm_state
** action_id
* notification_method
** id
** tenant_id
** name
** type
** address
** created_at
** updated_at
* sub_alarm
** id
** alarm_id
** function
** metric_name
** operator
** threshold
** period
** periods
** state
** created_at
** updated_at
* sub_alarm_dimension
** sub_alarm_id
** dimension_name
** value

== Events ==
Support for real-time event stream processing in Monasca is in progress. For more details see the link at, [[Monasca/Events]].

== Logging ==
Support for logging in Monasca is under discussion. For more details see the link at, [[Monasca/Logging]].

== Transform and Aggregation Engine ==
For more details see the link at, [[Monasca/Transform]].

== Analytics ==
Support for anomaly detection and alarm clustering/correlation is in progress. For more details see the link at, [[Monasca/Analytics]].

== Monitoring ==
Enablement and usage for monitoring the status of Monasca is under discussion. For more details see the link at, [[Monasca/Monitoring_Of_Monasca]]

== UI/UX Support ==
Adding more support for common UI/UX queries is under discussion. For more details see the link at, [[Monasca/UI_UX_Support]]

== Keystone Requirements ==
Monasca relies on keystone for running and there are requirements about which keystone configuration must exist.

* The endpoint for the api must be registered in keystone as the 'monasca' service.
* The api must have an admin token to use in verifying the keystone tokens it receives.
* For each project which uses Monasca two users must exist, one will be in the 'monasca-agent' role and be used by the monasca-agent's running on machines. The other should not be in that role and can be used logging into the UI, using the CLI or for direct queries against the API.

== Post Metric Sequence ==

This section describes the sequence of operations involved in posting a metric to the Monasca API.

[[File:monasca-arch-post-metric-diagram.png|Monasca Architecture Post Metric Diagram]]

# A metric is posted to the Monasca API.
# The Monasca API authenticates and validates the request and publishes the metric to the the Message Queue.
# The Persister consumes the metric from the Message Queue and stores in the Metrics Store.
# The Transform Engine consumes the metrics from the Message Queue, performs transform and aggregation operations on metrics, and publishes metrics that it creates back to Message Queue.
# The Threshold Engine consumes metrics from the Message Queue and evaluates alarms. If a state change occurs in an alarm, an "alarm-state-transitioned-event" is published to the Message Queue.
# The Notification Engine consumes "alarm-state-transitioned-events" from the Message Queue, evaluates whether they have a Notification Method associated with it, and sends the appropriate notification, such as email.
# The Persister consumes the "alarm-state-transitioned-event" from the Message Queue and stores it in the Alarm State History Store.

== Alarm Managers ==
This section describes the new features including alarm grouping, inhibition and silencing.

Examples:

* Silence
SilenceRule1 = '{"alarm-silencing-definition-created": {"name": "silence_rule_1", "matchers": {"severity": "LOW"}, "start_time": "2017-02-21 20:00:00", "end_time": "2017-02-21 22:00:00"}}'
Two alarm transitions: AT1 and AT2
AT1_severity = HIGH
AT2_severity = LOW

Output: AT2 get silenced and AT1 send a notification.
SilenceRule1 expires after "end_time"="2017-02-21 22:00:00".

* Inhibit
InhibitionRule1 = '{"alarm-inhibition-definition-created": {"name": "inhibit_rule_1","source":{"severity":"HIGH"},"target":{"severity":"LOW"}, "equals":["tenantId"], "exclusions": {"alarm_name": "vm is dead"}}}'
Three alarm transitions: AT1, AT2 and AT3
AT1_tenantId = "d42bc"
AT2_tenantId = "d42bc"
AT3_tenantId = "d42bc"
AT1_severity = HIGH
AT2_severity = LOW
AT3_severity = LOW
AT1_alarm_name = "cpu high"
AT2_alarm_name = "memory high"
AT3_alarm_name = "vm is dead"
AT1_state = ALARM
AT2_state = ALARM
AT3_state = ALARM

Output: AT1 is the source alarm which will send a notification. AT2 is the target alarm and will get inhibited. AT3 matches the exclusions and will send a notification immediately.

* Grouping Scenario 1

GroupingRule1 = '{"alarm-grouping-definition-created":
{"name": "group_rule_1", "matchers": ["hostname"], "id": "b7163","repeat_interval": "2h", "group_wait": "30s", "exclusions": {"alarm_name": "cpu_percent_high"}, "tenantId": " d42bc", "alarm_actions": ["cd892"], "ok_actions": ["ad892"], "undetermined_actions": ["cf892"]}}'

Three alarm transitions: AT1, AT2 and AT3

AT1_hostname = host1
AT2_hostname = host1
AT3_hostname = host2
AT1_alarm_name = cpu_percent_high
AT2_alarm_name = cpu_system_perc_high
AT3_alarm_name = cpu_percent_high
AT1_state = ALARM
AT2_state = ALARM
AT3_state = ALARM

Output: AT1 and AT3 match exclusions and send notifications immediately. Generate a grouped notification “group_notification_rule_1_host1_alarm[1]” and send out using alarm_actions ["cd892"]. There are no alarm_actions, ok_actions or undermined_actions associated with the AT1, AT2, AT3 alarm definitions.

* Grouping Scenario 2

GroupingRule2 = '{"alarm-grouping-definition-created":
{"name": "group_rule_2", "matchers": ["hostname"], "id": "b7163","repeat_interval": "2h", "group_wait": "30s", "tenantId": " d42bc", "alarm_actions": ["cd892"], "ok_actions": ["ad892"], "undetermined_actions": ["cf892"]}}'

Three alarm transitions: AT1 with "alarm_actions": ["123ab"], AT2 with "alarm_actions": ["cd839"] and AT3

AT1_hostname = host1
AT2_hostname = host1
AT3_hostname = host2
AT1_state = ALARM
AT2_state = ALARM
AT3_state = ALARM

Output: Generate two grouped notifications “group_notification_rule_2_host1_alarm[2]” and “group_notification_rule_2_host2_alarm[1]”. Both using alarm_actions ["cd892"]. Also since AT1 and AT2 has their own alarm actions associated with them, there will be two more notifications sent out.

* Silenced and Grouped

SilenceRule1 = '{"alarm-silencing-definition-created": {"name": "silence_rule_1", "matchers": {"severity": "LOW"}, "start_time": "1487269470498", "end_time": "1587269470498"}}'

GroupingRule2 = '{"alarm-grouping-definition-created":
{"name": "group_rule_2", "matchers": ["hostname"], "id": "b7163","repeat_interval": "2h", "group_wait": "30s", "tenantId": " d42bc", "alarm_actions": ["cd892"], "ok_actions": ["ad892"], "undetermined_actions": ["cf892"]}}'

Four alarm transitions: AT1, AT2, AT3 and AT4

Silencing rule:
AT1_severity = HIGH
AT2_severity = LOW
AT3_severity = HIGH
AT4_severity = HIGH

Grouping rule:
AT1_hostname = host1
AT2_hostname = host1
AT3_hostname = host2
AT4_hostname = host1
AT1_state = ALARM
AT2_state = ALARM
AT3_state = OK
AT4_state = ALARM

Output: Generate two grouped notifications “group_notification_rule_2_host1_alarm[2]” using alarm action "cd892" and “group_notification_rule_2_host2_ok[1]” using ok action "ad892". AT2 is silenced so it isn't include in group_notification_rule_2_host1 count. There are no alarm_actions, ok_actions or undermined_actions associated with the AT1, AT2, AT3, AT4 alarm definitions.

* Silenced and inhibited (source alarm get silenced)

SilenceRule2 = '{"alarm-silencing-definition-created": {"name": "silence_rule_2", "matchers": {"severity": "HIGH, "hostname": "host1"}, "start_time": "2017-02-21 15:00:00", "end_time": "2017-02-21 21:00:00"}}'

InhibitionRule1 = '{"alarm-inhibition-definition-created": {"name": "inhibit_rule_1","source":{"severity":"HIGH"},"target":{"severity":"LOW"}, "equals":["tenantId"]}}'

Alarm transitions: AT1, AT2

Inhibition rule:
AT1_state = ALARM
AT2_state = ALARM
AT1_tenantId = "d42bc"
AT2_tenantId = "d42bc"
AT1_severity = HIGH
AT2_severity = LOW

Silencing rule:
AT1_hostname = host1
AT2_hostname = host2
AT1_severity = HIGH
AT2_severity = LOW

Output: no notification sent out. For inhibition, AT1 is the source alarm, AT2 is the target alarm. But at the same time, AT1 get silenced because it matches the silence rule.

* Inhibited and grouped

InhibitionRule1 = '{"alarm-inhibition-definition-created": {"name": "inhibit_rule_1","source":{"severity":"HIGH"},"target":{"severity":"LOW"}, "equals":["tenantId"]}}'

GroupingRule1 = '{"alarm-grouping-definition-created": {"name": "group_rule_1", "matchers": ["hostname"], "id": "b7163", "repeat_interval": "2h", "group_wait": "30s", "tenantId": " d42bc", "alarm_actions": ["cd892"], "ok_actions": ["ad892"], "undetermined_actions": ["cf892"]}}'

Alarm transitions: AT1, AT2, AT3

Inhibition rule:
AT1_tenantId = "d42bc"
AT2_tenantId = "d42bc"
AT3_tenantId = "d42bc"
AT1_severity = HIGH
AT2_severity = LOW
AT3_severity = HIGH
AT1_state = OK
AT2_state = ALARM
AT3_state = ALARM

Grouping rule:
AT1_hostname = host1
AT2_hostname = host2
AT3_hostname = host1
AT1_state = OK
AT2_state = ALARM
AT3_state = ALARM

Output: AT2 gets inhibited because its severity is low. AT3 is the source alarm. Since AT1 is in OK state, it is not a source alarm. For grouping, AT1 and AT3 has the same host name but different state. So there will be two grouped notifications sent out: “group_notification_rule_1_host1_ok[1]” and “group_notification_rule_1_host1_alarm[1]”.

* Silenced, inhibited and grouped

SilenceRule2 = '{"alarm-silencing-definition-created": {"name": "silence_rule_2", "matchers": {"severity": "HIGH, "hostname": "host1"}, "start_time": "2017-02-21 15:00:00", "end_time": "2017-02-21 21:00:00"}}'

InhibitionRule1 = '{"alarm-inhibition-definition-created": {"name": "inhibit_rule_1","source":{"severity":"HIGH"},"target":{"severity":"LOW"}, "equals":["tenantId"]}}'

GroupingRule1 = '{"alarm-grouping-definition-created": {"name": "group_rule_1", "matchers": ["hostname"], "id": "b7163", "repeat_interval": "2h", "group_wait": "30s", "tenantId": " d42bc", "alarm_actions": ["cd892"], "ok_actions": ["ad892"], "undetermined_actions": ["cf892"]}}'

Alarm transitions: AT1, AT2, AT3 and AT5

Silence rule:
AT1_severity = HIGH AT1_hostname = host1 (silenced)
AT2_severity = LOW AT2_hostname = host2
AT3_severity = HIGH AT3_hostname = host1(silenced)
AT5_severity = HIGH AT5_hostname = host3

Inhibition rule:
AT1_tenantId = "d42bc"
AT2_tenantId = "d42bc"
AT3_tenantId = "d42bc"
AT5_tenantId = "d42bc"
AT1_state = ALARM
AT2_state = ALARM
AT3_state = OK
AT5_state = UNDETERMINED
AT1_severity = HIGH (source)
AT2_severity = LOW (target)
AT3_severity = HIGH
AT5_severity = HIGH

Grouping rule:
AT1_hostname = host1
AT2_hostname = host2
AT3_hostname = host1
AT5_hostname = host3

Output: AT1 is in "group_notification_rule_1_host1_alarm" group and silenced. AT3 is in "group_notification_rule_1_host1_ok" group and silenced. AT2 is in "group_notification_rule_1_host2_alarm" group and inhibited. AT5 is in "group_notification_rule_1_host3_undetermined" group and will send notification “group_notification_rule_1_host3_undetermined[1]” using undetermined action "cf892".

= Development Environment =

* Monasca DevStack Plugin:
** DevStack is the primary developmement environment for OpenStack.
***See http://docs.openstack.org/developer/devstack/
** The Monasca DevStack plugin installs the Monasca Service, Agent, Horizon Monitoring Panel, and Grafana
** README at, https://github.com/openstack/monasca-api/tree/master/devstack
** Best way to get started is to use Vagrant with the Vagrantfile at, https://github.com/openstack/monasca-api/blob/master/devstack/Vagrantfile.

* Monasca on LaunchPad. https://launchpad.net/monasca

* Monasca projects source code. https://git.openstack.org/cgit/?q=monasca

= Coding Standards =

* Python: All Python code conforms to the OpenStack standards at, http://docs.openstack.org/developer/hacking/.
** Note, all components in Monasca, except for the Threshold Engine, have been ported to Python.

* Java: Several of the Monasca components are available as Java. OpenStack does not have any Java coding standards. We've adopted the Google Java Style at, https://google.github.io/styleguide/javaguide.html.
** The standard says either 80 or 100 length lines. We've adopted 100.

= Technologies =

Monasca uses a number of third-party technologies:

* Internal Processing and Middleware
** Apache Kafka (http://kafka.apache.org): Apache Kafka is publish-subscribe messaging rethought as a distributed commit log. Kafka is a highly performant, distributed, fault-tolerant, and scalable message queue with durability built-in.
** Apache Storm (http://storm.incubator.apache.org/): Apache Storm is a free and open source distributed realtime computation system. Storm makes it easy to reliably process unbounded streams of data, doing for realtime processing what Hadoop did for batch processing.
** ZooKeeper (http://zookeeper.apache.org/): Used by Kafka and Storm.
** Apache Spark: Used by Monasca Transform as an aggregation engine.

* Configuration database:
** MySQL: MySQL is supported as a Config Database.
** PostgreSQL: Support for POSTgres, via Hibernate and SQLAlchemy, for the Config Database.

* Vagrant (http://www.vagrantup.com/): Vagrant provides easy to configure, reproducible, and portable work environments built on top of industry-standard technology and controlled by a single consistent workflow to help maximize the productivity and flexibility of you and your team.

* Dropwizard (https://dropwizard.github.io/dropwizard/): Dropwizard pulls together stable, mature libraries from the Java ecosystem into a simple, light-weight package that lets you focus on getting things done. Dropwizard has out-of-the-box support for sophisticated configuration, application metrics, logging, operational tools, and much more, allowing you and your team to ship a production-quality web service in the shortest time possible.

* Time Series Database:
** InfluxDB (http://influxdb.com/): An open-source distributed time series database with no external dependencies. InfluxDB is supported for the Metrics Database.
** Vertica (http://www.vertica.com): A commercial Enterprise class SQL analytics database that is highly scalable. It offers built-in automatic high-availability and excels at in-database analytics and compressing and storing massive amounts of data. A free community version of Vertica is available that can store up to 1 TB of data with no time-limit is available at, https://my.vertica.com/community/. Vertica is supported for the Metrics Database, though no longer commonly used.
** Cassandra(https://cassandra.apache.org): Cassandra is supported for the Metrics Database.

= License =

Copyright (c) 2014, 2015 Hewlett-Packard Development Company, L.P.
(C) Copyright 2019 SUSE LLC

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied.
See the License for the specific language governing permissions and
limitations under the License.

[[File:yklogo.png|200px|thumb|left|Monasca uses YourKit Profiler for Java development]]

 
 
 
 

[https://www.yourkit.com/ Visit YourKit website for more information]

Monasca

2019-05-08T17:07:25Z

Joseph Davis: /* Technologies */ cass

== Overview ==

Monasca is a open-source multi-tenant, highly scalable, performant, fault-tolerant monitoring-as-a-service solution that integrates with OpenStack. It uses a REST API for high-speed metrics processing and querying and has a streaming alarm engine and notification engine.

[[File:OpenStack_Project_Monasca_vertical.png|thumb|right]]

==== Contribution ====
* [https://storyboard.openstack.org/#!/project_group/59 Project Group on StoryBoard]
* [https://storyboard.openstack.org/#!/board/111 Important activities in Stein release]
* [https://storyboard.openstack.org/#!/board/141 Important activities in Train release]
* [https://docs.openstack.org/monasca-api/latest/contributor/index.html Contribution Guidelines]

==== Communication and Meetings ====
* IRC: #openstack-monasca on freenode.net
* Weekly Meetings:
** http://eavesdrop.openstack.org/#Monasca_Team_Meeting
** [[Meetings/Monasca]]
* Use [Monasca] tag on [http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-discuss openstack-discuss mailing list].

===== PTG meetings =====
* [[MonascaSteinPTG|Stein PTG Summary (Denver)]]
* (coming soon - Train summary)

==== Documentation ====
: Monasca API Specification: https://github.com/openstack/monasca-api/blob/master/docs/monasca-api-spec.md
: Agent Documentation: https://github.com/openstack/monasca-agent

==== Presentations ====
There have been many interesting and excellent presentations on Monasca, many given at the OpenStack or OpenInfra Summits.
: [[Monasca/Presentations]] (updated with Denver 2019 links)

==== Repositories ====

: '''Code'''
:: Core: https://git.openstack.org/cgit/?q=monasca (old git link)
:: API: https://opendev.org/openstack/monasca-api

: '''Deployment'''
: The following repositories are available for deploying Monasca:
:: Docker: https://github.com/monasca/monasca-docker
:: Kubernetes: https://github.com/monasca/monasca-helm
:: Ansible: https://github.com/search?utf8=%E2%9C%93&q=ansible-monasca
:: Puppet: https://git.openstack.org/openstack/puppet-monasca

==== Bugs ====

===== All open bugs =====
: https://storyboard.openstack.org/#!/worklist/213

===== In Stein triaged bugs =====
: https://storyboard.openstack.org/#!/worklist/467

===== Bug-fixing progress =====
: https://storyboard.openstack.org/#!/board/114

== Requirements ==
[[Monasca/Requirements]].

See also https://opendev.org/openstack/monasca-specs/

== Features ==

This section describes the overall features.

* A highly performant, scalable, reliable and fault-tolerant Monitoring as a Service (MONaaS) solution that scales to service provider metrics levels of metrics throughput. Performance, scalability and high-availability have been designed in from the start. Can process 100s of thousands of metrics/sec as well as offer data retention periods of greater than a year with no data loss while still processing interactive queries.

* Rest API for storing and querying metrics and historical information. Most monitoring solution use special transports and protocols, such as CollectD or NSCA (Nagios). In our solution, http is the only protocol used. This simplifies the overall design and also allows for a much richer way of describing the data via dimensions.

* Multi-tenant and authenticated. Metrics are submitted and authenticated using Keystone and stored associated with a tenant ID.

* Metrics defined using a set of (key, value) pairs called dimensions.

* Real-time thresholding and alarming on metrics.

* Compound alarms described using a simple expressive grammar composed of alarm sub-expressions and logical operators.

* Monitoring agent that supports a number of built-in system and service checks and also supports Nagios checks and statsd.

* Open-source monitoring solution built on open-source technologies.

=== Comparisons to alternatives ===
[[Monasca/Comparison]]

== Architecture ==
[[File:monasca-arch-component-diagram.png|Monasca Architecture Component Diagram]]
* Monitoring Agent (monasca-agent): A modern Python based monitoring agent that consists of several sub-components and supports system metrics, such as cpu utilization and available memory, Nagios plugins, statsd and many built-in checks for services such as MySQL, RabbitMQ, and many others.
* Monitoring API (monasca-api): A well-defined and documented RESTful API for monitoring that is primarily focused on the following concepts and areas:
** Metrics: Store and query massive amounts of metrics in real-time.
** Statistics: Query statistics for metrics.
** Alarm Definitions: Create, update, query and delete alarm definitions.
** Alarms: Query and delete the alarm history.
***Simple expressive grammar for creating compound alarms composed of alarm subexpressions and logical operators.
***Alarm severities can be associated with alarms.
***The complete alarm state transition history is stored and queryable which allows for subsequent root cause analysis (RCA) or advanced analytics.
**Notification Methods: Create and delete notification methods and associate them with alarms, such as email. Supports the ability to notify users directly via email when an alarm state transitions occur.
** The Monasca API has both Java and Python implementations avaialble.
* Persister (monasca-persister): Consumes metrics and alarm state transitions from the MessageQ and stores them in the Metrics and Alarms database.
** The Persister has both Java and Python implementations.
* Transform and Aggregation Engine (monasca-transform): Transform metric names and values, such as delta or time-based derivative calculations, and creates new metrics that are published to the Message Queue. The Transform Engine is not available yet.
* Anomaly and Prediction Engine: Evaluates prediction and anomalies and generates predicted metrics as well as anomaly likelihood and anomaly scores. The Anomaly and Prediction Engine is currently in a prototype status.
* Threshold Engine (monasca-thresh): Computes thresholds on metrics and publishes alarms to the MessageQ when exceeded. Based on Apache Storm a free and open distributed real-time computation system.
* Notification Engine (monasca-notification): Consumes alarm state transition messages from the MessageQ and sends notifications, such as emails for alarms. The Notification Engine is Python based.
* Analytics Engine (monasca-analytics): Consumes alarm state transisitions and metrics from the MessageQ and does anomaly detection and alarm clustering/correlation.
* Message Queue: A third-party component that primarily receives published metrics from the Monitoring API and alarm state transition messages from the Threshold Engine that are consumed by other components, such as the Persister and Notification Engine. The Message Queue is also used to publish and consume other events in the system. Currently, a Kafka based MessageQ is supported. Kafka is a high performance, distributed, fault-tolerant, and scalable message queue with durability built-in. We will look at other alternatives, such as RabbitMQ and in-fact in our previous implementation RabbitMQ was supported, but due to performance, scale, durability and high-availability limitiations with RabbitMQ we have moved to Kafka.
* Metrics and Alarms Database: A third-party component that primarily stores metrics and the alarm state history. Currently, Vertica and InfluxDB are supported. Support for Cassandra is in progress.
* Config Database: A third-party component that stores a lot of the configuration and other information in the system. Currently, MySQL is supported. Support for PostgreSQL is in progress.
* Monitoring Client (python-monascaclient): A Python command line client and library that communicates and controls the Monitoring API. The Monitoring Client was written using the OpenStack Heat Python client as a framework. The Monitoring Client also has a Python library, "monascaclient" similar to the other OpenStack clients, that can be used to quickly build additional capabilities. The Monitoring Client library is used by the Monitoring UI, Ceilometer publisher, and other components.
* Monitoring UI: A Horizon dashboard for visualizing the overall health and status of an OpenStack cloud.
* Ceilometer publisher: A multi-publisher plugin for Ceilometer, not shown, that converts and publishes samples to the Monitoring API.
Most of the components are described in their respective repositories. However, there aren't any repositories for the third-party components used, so we describe some of the relevant details here.

Further historical context for the architecture can be found at [[Monasca/Architecture Details]]

=== Message Schema ===
[[Monasca/Message Schema]]

=== Message Queue ===
A distributed, performant, scalable, HA message queue for distributing metrics, alarms and events in the monitoring system. Currently, based on Kafka.

==== Messages ====
There are several messages that are published and consumed by various components in Monasca via the MessageQ. See [[Monasca/Message Schema|Message Schema]].

=== Metrics and Alarms Database ===

A high-performance analytics database that can store massive amounts of metrics and alarms in real-time and also support interactive queries. Currently Vertica and InfluxDB are supported.

The SQL schema that is used by Vertica is as follows:

* MonMetrics.Measurements: Stores the actual measurements that are sent.
** id: An integer ID for the measurement.
** definition_dimensions_id: A reference to DefinitionDimensions.
** time_stamp
** value
* MonMetrics.DefinitionDimensions
** id: A sha1 hash of (defintion_id, dimension_set_id)
** definition_id: A reference to the Definitions.id
** dimension_set_id: A reference to the Dimensions.dimension_set_id
* MonMetrics.Definitions
** id: A sha1 hash of the (name, tenant_id, region)
** name: Name of the metric.
** tenant_id: The tenant_id that submitted the metric.
** region: The region the metric was submitted under.
* MonMetric.Dimensions
** dimension_set_id: A sha1 hash of the set of dimenions for a metric.
** name: Name of dimension.
** value: Value of dimension.

=== Config Database ===
The config database store all the configuration information. Currently based on MySQL.

The SQL schema is as follows:

* alarm
** id
** tenant_id
** name
** description
** expression
** state
** actions_enabled
** created_at
** updated_at
** deleted_at
* alarm_action
** alarm_id
** alarm_state
** action_id
* notification_method
** id
** tenant_id
** name
** type
** address
** created_at
** updated_at
* sub_alarm
** id
** alarm_id
** function
** metric_name
** operator
** threshold
** period
** periods
** state
** created_at
** updated_at
* sub_alarm_dimension
** sub_alarm_id
** dimension_name
** value

== Events ==
Support for real-time event stream processing in Monasca is in progress. For more details see the link at, [[Monasca/Events]].

== Logging ==
Support for logging in Monasca is under discussion. For more details see the link at, [[Monasca/Logging]].

== Transform and Aggregation Engine ==
For more details see the link at, [[Monasca/Transform]].

== Analytics ==
Support for anomaly detection and alarm clustering/correlation is in progress. For more details see the link at, [[Monasca/Analytics]].

== Monitoring ==
Enablement and usage for monitoring the status of Monasca is under discussion. For more details see the link at, [[Monasca/Monitoring_Of_Monasca]]

== UI/UX Support ==
Adding more support for common UI/UX queries is under discussion. For more details see the link at, [[Monasca/UI_UX_Support]]

== Keystone Requirements ==
Monasca relies on keystone for running and there are requirements about which keystone configuration must exist.

* The endpoint for the api must be registered in keystone as the 'monasca' service.
* The api must have an admin token to use in verifying the keystone tokens it receives.
* For each project which uses Monasca two users must exist, one will be in the 'monasca-agent' role and be used by the monasca-agent's running on machines. The other should not be in that role and can be used logging into the UI, using the CLI or for direct queries against the API.

== Post Metric Sequence ==

This section describes the sequence of operations involved in posting a metric to the Monasca API.

[[File:monasca-arch-post-metric-diagram.png|Monasca Architecture Post Metric Diagram]]

# A metric is posted to the Monasca API.
# The Monasca API authenticates and validates the request and publishes the metric to the the Message Queue.
# The Persister consumes the metric from the Message Queue and stores in the Metrics Store.
# The Transform Engine consumes the metrics from the Message Queue, performs transform and aggregation operations on metrics, and publishes metrics that it creates back to Message Queue.
# The Threshold Engine consumes metrics from the Message Queue and evaluates alarms. If a state change occurs in an alarm, an "alarm-state-transitioned-event" is published to the Message Queue.
# The Notification Engine consumes "alarm-state-transitioned-events" from the Message Queue, evaluates whether they have a Notification Method associated with it, and sends the appropriate notification, such as email.
# The Persister consumes the "alarm-state-transitioned-event" from the Message Queue and stores it in the Alarm State History Store.

== Alarm Managers ==
This section describes the new features including alarm grouping, inhibition and silencing.

Examples:

* Silence
SilenceRule1 = '{"alarm-silencing-definition-created": {"name": "silence_rule_1", "matchers": {"severity": "LOW"}, "start_time": "2017-02-21 20:00:00", "end_time": "2017-02-21 22:00:00"}}'
Two alarm transitions: AT1 and AT2
AT1_severity = HIGH
AT2_severity = LOW

Output: AT2 get silenced and AT1 send a notification.
SilenceRule1 expires after "end_time"="2017-02-21 22:00:00".

* Inhibit
InhibitionRule1 = '{"alarm-inhibition-definition-created": {"name": "inhibit_rule_1","source":{"severity":"HIGH"},"target":{"severity":"LOW"}, "equals":["tenantId"], "exclusions": {"alarm_name": "vm is dead"}}}'
Three alarm transitions: AT1, AT2 and AT3
AT1_tenantId = "d42bc"
AT2_tenantId = "d42bc"
AT3_tenantId = "d42bc"
AT1_severity = HIGH
AT2_severity = LOW
AT3_severity = LOW
AT1_alarm_name = "cpu high"
AT2_alarm_name = "memory high"
AT3_alarm_name = "vm is dead"
AT1_state = ALARM
AT2_state = ALARM
AT3_state = ALARM

Output: AT1 is the source alarm which will send a notification. AT2 is the target alarm and will get inhibited. AT3 matches the exclusions and will send a notification immediately.

* Grouping Scenario 1

GroupingRule1 = '{"alarm-grouping-definition-created":
{"name": "group_rule_1", "matchers": ["hostname"], "id": "b7163","repeat_interval": "2h", "group_wait": "30s", "exclusions": {"alarm_name": "cpu_percent_high"}, "tenantId": " d42bc", "alarm_actions": ["cd892"], "ok_actions": ["ad892"], "undetermined_actions": ["cf892"]}}'

Three alarm transitions: AT1, AT2 and AT3

AT1_hostname = host1
AT2_hostname = host1
AT3_hostname = host2
AT1_alarm_name = cpu_percent_high
AT2_alarm_name = cpu_system_perc_high
AT3_alarm_name = cpu_percent_high
AT1_state = ALARM
AT2_state = ALARM
AT3_state = ALARM

Output: AT1 and AT3 match exclusions and send notifications immediately. Generate a grouped notification “group_notification_rule_1_host1_alarm[1]” and send out using alarm_actions ["cd892"]. There are no alarm_actions, ok_actions or undermined_actions associated with the AT1, AT2, AT3 alarm definitions.

* Grouping Scenario 2

GroupingRule2 = '{"alarm-grouping-definition-created":
{"name": "group_rule_2", "matchers": ["hostname"], "id": "b7163","repeat_interval": "2h", "group_wait": "30s", "tenantId": " d42bc", "alarm_actions": ["cd892"], "ok_actions": ["ad892"], "undetermined_actions": ["cf892"]}}'

Three alarm transitions: AT1 with "alarm_actions": ["123ab"], AT2 with "alarm_actions": ["cd839"] and AT3

AT1_hostname = host1
AT2_hostname = host1
AT3_hostname = host2
AT1_state = ALARM
AT2_state = ALARM
AT3_state = ALARM

Output: Generate two grouped notifications “group_notification_rule_2_host1_alarm[2]” and “group_notification_rule_2_host2_alarm[1]”. Both using alarm_actions ["cd892"]. Also since AT1 and AT2 has their own alarm actions associated with them, there will be two more notifications sent out.

* Silenced and Grouped

SilenceRule1 = '{"alarm-silencing-definition-created": {"name": "silence_rule_1", "matchers": {"severity": "LOW"}, "start_time": "1487269470498", "end_time": "1587269470498"}}'

GroupingRule2 = '{"alarm-grouping-definition-created":
{"name": "group_rule_2", "matchers": ["hostname"], "id": "b7163","repeat_interval": "2h", "group_wait": "30s", "tenantId": " d42bc", "alarm_actions": ["cd892"], "ok_actions": ["ad892"], "undetermined_actions": ["cf892"]}}'

Four alarm transitions: AT1, AT2, AT3 and AT4

Silencing rule:
AT1_severity = HIGH
AT2_severity = LOW
AT3_severity = HIGH
AT4_severity = HIGH

Grouping rule:
AT1_hostname = host1
AT2_hostname = host1
AT3_hostname = host2
AT4_hostname = host1
AT1_state = ALARM
AT2_state = ALARM
AT3_state = OK
AT4_state = ALARM

Output: Generate two grouped notifications “group_notification_rule_2_host1_alarm[2]” using alarm action "cd892" and “group_notification_rule_2_host2_ok[1]” using ok action "ad892". AT2 is silenced so it isn't include in group_notification_rule_2_host1 count. There are no alarm_actions, ok_actions or undermined_actions associated with the AT1, AT2, AT3, AT4 alarm definitions.

* Silenced and inhibited (source alarm get silenced)

SilenceRule2 = '{"alarm-silencing-definition-created": {"name": "silence_rule_2", "matchers": {"severity": "HIGH, "hostname": "host1"}, "start_time": "2017-02-21 15:00:00", "end_time": "2017-02-21 21:00:00"}}'

InhibitionRule1 = '{"alarm-inhibition-definition-created": {"name": "inhibit_rule_1","source":{"severity":"HIGH"},"target":{"severity":"LOW"}, "equals":["tenantId"]}}'

Alarm transitions: AT1, AT2

Inhibition rule:
AT1_state = ALARM
AT2_state = ALARM
AT1_tenantId = "d42bc"
AT2_tenantId = "d42bc"
AT1_severity = HIGH
AT2_severity = LOW

Silencing rule:
AT1_hostname = host1
AT2_hostname = host2
AT1_severity = HIGH
AT2_severity = LOW

Output: no notification sent out. For inhibition, AT1 is the source alarm, AT2 is the target alarm. But at the same time, AT1 get silenced because it matches the silence rule.

* Inhibited and grouped

InhibitionRule1 = '{"alarm-inhibition-definition-created": {"name": "inhibit_rule_1","source":{"severity":"HIGH"},"target":{"severity":"LOW"}, "equals":["tenantId"]}}'

GroupingRule1 = '{"alarm-grouping-definition-created": {"name": "group_rule_1", "matchers": ["hostname"], "id": "b7163", "repeat_interval": "2h", "group_wait": "30s", "tenantId": " d42bc", "alarm_actions": ["cd892"], "ok_actions": ["ad892"], "undetermined_actions": ["cf892"]}}'

Alarm transitions: AT1, AT2, AT3

Inhibition rule:
AT1_tenantId = "d42bc"
AT2_tenantId = "d42bc"
AT3_tenantId = "d42bc"
AT1_severity = HIGH
AT2_severity = LOW
AT3_severity = HIGH
AT1_state = OK
AT2_state = ALARM
AT3_state = ALARM

Grouping rule:
AT1_hostname = host1
AT2_hostname = host2
AT3_hostname = host1
AT1_state = OK
AT2_state = ALARM
AT3_state = ALARM

Output: AT2 gets inhibited because its severity is low. AT3 is the source alarm. Since AT1 is in OK state, it is not a source alarm. For grouping, AT1 and AT3 has the same host name but different state. So there will be two grouped notifications sent out: “group_notification_rule_1_host1_ok[1]” and “group_notification_rule_1_host1_alarm[1]”.

* Silenced, inhibited and grouped

SilenceRule2 = '{"alarm-silencing-definition-created": {"name": "silence_rule_2", "matchers": {"severity": "HIGH, "hostname": "host1"}, "start_time": "2017-02-21 15:00:00", "end_time": "2017-02-21 21:00:00"}}'

InhibitionRule1 = '{"alarm-inhibition-definition-created": {"name": "inhibit_rule_1","source":{"severity":"HIGH"},"target":{"severity":"LOW"}, "equals":["tenantId"]}}'

GroupingRule1 = '{"alarm-grouping-definition-created": {"name": "group_rule_1", "matchers": ["hostname"], "id": "b7163", "repeat_interval": "2h", "group_wait": "30s", "tenantId": " d42bc", "alarm_actions": ["cd892"], "ok_actions": ["ad892"], "undetermined_actions": ["cf892"]}}'

Alarm transitions: AT1, AT2, AT3 and AT5

Silence rule:
AT1_severity = HIGH AT1_hostname = host1 (silenced)
AT2_severity = LOW AT2_hostname = host2
AT3_severity = HIGH AT3_hostname = host1(silenced)
AT5_severity = HIGH AT5_hostname = host3

Inhibition rule:
AT1_tenantId = "d42bc"
AT2_tenantId = "d42bc"
AT3_tenantId = "d42bc"
AT5_tenantId = "d42bc"
AT1_state = ALARM
AT2_state = ALARM
AT3_state = OK
AT5_state = UNDETERMINED
AT1_severity = HIGH (source)
AT2_severity = LOW (target)
AT3_severity = HIGH
AT5_severity = HIGH

Grouping rule:
AT1_hostname = host1
AT2_hostname = host2
AT3_hostname = host1
AT5_hostname = host3

Output: AT1 is in "group_notification_rule_1_host1_alarm" group and silenced. AT3 is in "group_notification_rule_1_host1_ok" group and silenced. AT2 is in "group_notification_rule_1_host2_alarm" group and inhibited. AT5 is in "group_notification_rule_1_host3_undetermined" group and will send notification “group_notification_rule_1_host3_undetermined[1]” using undetermined action "cf892".

= Development Environment =

* Monasca DevStack Plugin:
** DevStack is the primary developmement environment for OpenStack.
***See http://docs.openstack.org/developer/devstack/
** The Monasca DevStack plugin installs the Monasca Service, Agent, Horizon Monitoring Panel, and Grafana
** README at, https://github.com/openstack/monasca-api/tree/master/devstack
** Best way to get started is to use Vagrant with the Vagrantfile at, https://github.com/openstack/monasca-api/blob/master/devstack/Vagrantfile.

* Monasca on LaunchPad. https://launchpad.net/monasca

* Monasca projects source code. https://git.openstack.org/cgit/?q=monasca

= Coding Standards =

* Python: All Python code conforms to the OpenStack standards at, http://docs.openstack.org/developer/hacking/.
** Note, all components in Monasca, except for the Threshold Engine, have been ported to Python.

* Java: Several of the Monasca components are available as Java. OpenStack does not have any Java coding standards. We've adopted the Google Java Style at, https://google.github.io/styleguide/javaguide.html.
** The standard says either 80 or 100 length lines. We've adopted 100.

= Technologies =

Monasca uses a number of third-party technologies:

* Internal Processing and Middleware
** Apache Kafka (http://kafka.apache.org): Apache Kafka is publish-subscribe messaging rethought as a distributed commit log. Kafka is a highly performant, distributed, fault-tolerant, and scalable message queue with durability built-in.
** Apache Storm (http://storm.incubator.apache.org/): Apache Storm is a free and open source distributed realtime computation system. Storm makes it easy to reliably process unbounded streams of data, doing for realtime processing what Hadoop did for batch processing.
** ZooKeeper (http://zookeeper.apache.org/): Used by Kafka and Storm.
** Apache Spark: Used by Monasca Transform as an aggregation engine.

* Configuration database:
** MySQL: MySQL is supported as a Config Database.
** PostgreSQL: Support for POSTgres, via Hibernate and SQLAlchemy, for the Config Database.

* Vagrant (http://www.vagrantup.com/): Vagrant provides easy to configure, reproducible, and portable work environments built on top of industry-standard technology and controlled by a single consistent workflow to help maximize the productivity and flexibility of you and your team.

* Dropwizard (https://dropwizard.github.io/dropwizard/): Dropwizard pulls together stable, mature libraries from the Java ecosystem into a simple, light-weight package that lets you focus on getting things done. Dropwizard has out-of-the-box support for sophisticated configuration, application metrics, logging, operational tools, and much more, allowing you and your team to ship a production-quality web service in the shortest time possible.

* Time Series Database:
** InfluxDB (http://influxdb.com/): An open-source distributed time series database with no external dependencies. InfluxDB is supported for the Metrics Database.
** Vertica (http://www.vertica.com): A commercial Enterprise class SQL analytics database that is highly scalable. It offers built-in automatic high-availability and excels at in-database analytics and compressing and storing massive amounts of data. A free community version of Vertica is available that can store up to 1 TB of data with no time-limit is available at, https://my.vertica.com/community/. Vertica is supported for the Metrics Database, though no longer commonly used.
** Cassandra(https://cassandra.apache.org): Cassandra is supported for the Metrics Database.

= License =

Copyright (c) 2014, 2015 Hewlett-Packard Development Company, L.P.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied.
See the License for the specific language governing permissions and
limitations under the License.

[[File:yklogo.png|200px|thumb|left|Monasca uses YourKit Profiler for Java development]]

 
 
 
 

[https://www.yourkit.com/ Visit YourKit website for more information]

Monasca

2019-05-08T17:05:40Z

Joseph Davis: /* Technologies */ reformatting, add spark

== Overview ==

Monasca is a open-source multi-tenant, highly scalable, performant, fault-tolerant monitoring-as-a-service solution that integrates with OpenStack. It uses a REST API for high-speed metrics processing and querying and has a streaming alarm engine and notification engine.

[[File:OpenStack_Project_Monasca_vertical.png|thumb|right]]

==== Contribution ====
* [https://storyboard.openstack.org/#!/project_group/59 Project Group on StoryBoard]
* [https://storyboard.openstack.org/#!/board/111 Important activities in Stein release]
* [https://storyboard.openstack.org/#!/board/141 Important activities in Train release]
* [https://docs.openstack.org/monasca-api/latest/contributor/index.html Contribution Guidelines]

==== Communication and Meetings ====
* IRC: #openstack-monasca on freenode.net
* Weekly Meetings:
** http://eavesdrop.openstack.org/#Monasca_Team_Meeting
** [[Meetings/Monasca]]
* Use [Monasca] tag on [http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-discuss openstack-discuss mailing list].

===== PTG meetings =====
* [[MonascaSteinPTG|Stein PTG Summary (Denver)]]
* (coming soon - Train summary)

==== Documentation ====
: Monasca API Specification: https://github.com/openstack/monasca-api/blob/master/docs/monasca-api-spec.md
: Agent Documentation: https://github.com/openstack/monasca-agent

==== Presentations ====
There have been many interesting and excellent presentations on Monasca, many given at the OpenStack or OpenInfra Summits.
: [[Monasca/Presentations]] (updated with Denver 2019 links)

==== Repositories ====

: '''Code'''
:: Core: https://git.openstack.org/cgit/?q=monasca (old git link)
:: API: https://opendev.org/openstack/monasca-api

: '''Deployment'''
: The following repositories are available for deploying Monasca:
:: Docker: https://github.com/monasca/monasca-docker
:: Kubernetes: https://github.com/monasca/monasca-helm
:: Ansible: https://github.com/search?utf8=%E2%9C%93&q=ansible-monasca
:: Puppet: https://git.openstack.org/openstack/puppet-monasca

==== Bugs ====

===== All open bugs =====
: https://storyboard.openstack.org/#!/worklist/213

===== In Stein triaged bugs =====
: https://storyboard.openstack.org/#!/worklist/467

===== Bug-fixing progress =====
: https://storyboard.openstack.org/#!/board/114

== Requirements ==
[[Monasca/Requirements]].

See also https://opendev.org/openstack/monasca-specs/

== Features ==

This section describes the overall features.

* A highly performant, scalable, reliable and fault-tolerant Monitoring as a Service (MONaaS) solution that scales to service provider metrics levels of metrics throughput. Performance, scalability and high-availability have been designed in from the start. Can process 100s of thousands of metrics/sec as well as offer data retention periods of greater than a year with no data loss while still processing interactive queries.

* Rest API for storing and querying metrics and historical information. Most monitoring solution use special transports and protocols, such as CollectD or NSCA (Nagios). In our solution, http is the only protocol used. This simplifies the overall design and also allows for a much richer way of describing the data via dimensions.

* Multi-tenant and authenticated. Metrics are submitted and authenticated using Keystone and stored associated with a tenant ID.

* Metrics defined using a set of (key, value) pairs called dimensions.

* Real-time thresholding and alarming on metrics.

* Compound alarms described using a simple expressive grammar composed of alarm sub-expressions and logical operators.

* Monitoring agent that supports a number of built-in system and service checks and also supports Nagios checks and statsd.

* Open-source monitoring solution built on open-source technologies.

=== Comparisons to alternatives ===
[[Monasca/Comparison]]

== Architecture ==
[[File:monasca-arch-component-diagram.png|Monasca Architecture Component Diagram]]
* Monitoring Agent (monasca-agent): A modern Python based monitoring agent that consists of several sub-components and supports system metrics, such as cpu utilization and available memory, Nagios plugins, statsd and many built-in checks for services such as MySQL, RabbitMQ, and many others.
* Monitoring API (monasca-api): A well-defined and documented RESTful API for monitoring that is primarily focused on the following concepts and areas:
** Metrics: Store and query massive amounts of metrics in real-time.
** Statistics: Query statistics for metrics.
** Alarm Definitions: Create, update, query and delete alarm definitions.
** Alarms: Query and delete the alarm history.
***Simple expressive grammar for creating compound alarms composed of alarm subexpressions and logical operators.
***Alarm severities can be associated with alarms.
***The complete alarm state transition history is stored and queryable which allows for subsequent root cause analysis (RCA) or advanced analytics.
**Notification Methods: Create and delete notification methods and associate them with alarms, such as email. Supports the ability to notify users directly via email when an alarm state transitions occur.
** The Monasca API has both Java and Python implementations avaialble.
* Persister (monasca-persister): Consumes metrics and alarm state transitions from the MessageQ and stores them in the Metrics and Alarms database.
** The Persister has both Java and Python implementations.
* Transform and Aggregation Engine (monasca-transform): Transform metric names and values, such as delta or time-based derivative calculations, and creates new metrics that are published to the Message Queue. The Transform Engine is not available yet.
* Anomaly and Prediction Engine: Evaluates prediction and anomalies and generates predicted metrics as well as anomaly likelihood and anomaly scores. The Anomaly and Prediction Engine is currently in a prototype status.
* Threshold Engine (monasca-thresh): Computes thresholds on metrics and publishes alarms to the MessageQ when exceeded. Based on Apache Storm a free and open distributed real-time computation system.
* Notification Engine (monasca-notification): Consumes alarm state transition messages from the MessageQ and sends notifications, such as emails for alarms. The Notification Engine is Python based.
* Analytics Engine (monasca-analytics): Consumes alarm state transisitions and metrics from the MessageQ and does anomaly detection and alarm clustering/correlation.
* Message Queue: A third-party component that primarily receives published metrics from the Monitoring API and alarm state transition messages from the Threshold Engine that are consumed by other components, such as the Persister and Notification Engine. The Message Queue is also used to publish and consume other events in the system. Currently, a Kafka based MessageQ is supported. Kafka is a high performance, distributed, fault-tolerant, and scalable message queue with durability built-in. We will look at other alternatives, such as RabbitMQ and in-fact in our previous implementation RabbitMQ was supported, but due to performance, scale, durability and high-availability limitiations with RabbitMQ we have moved to Kafka.
* Metrics and Alarms Database: A third-party component that primarily stores metrics and the alarm state history. Currently, Vertica and InfluxDB are supported. Support for Cassandra is in progress.
* Config Database: A third-party component that stores a lot of the configuration and other information in the system. Currently, MySQL is supported. Support for PostgreSQL is in progress.
* Monitoring Client (python-monascaclient): A Python command line client and library that communicates and controls the Monitoring API. The Monitoring Client was written using the OpenStack Heat Python client as a framework. The Monitoring Client also has a Python library, "monascaclient" similar to the other OpenStack clients, that can be used to quickly build additional capabilities. The Monitoring Client library is used by the Monitoring UI, Ceilometer publisher, and other components.
* Monitoring UI: A Horizon dashboard for visualizing the overall health and status of an OpenStack cloud.
* Ceilometer publisher: A multi-publisher plugin for Ceilometer, not shown, that converts and publishes samples to the Monitoring API.
Most of the components are described in their respective repositories. However, there aren't any repositories for the third-party components used, so we describe some of the relevant details here.

Further historical context for the architecture can be found at [[Monasca/Architecture Details]]

=== Message Schema ===
[[Monasca/Message Schema]]

=== Message Queue ===
A distributed, performant, scalable, HA message queue for distributing metrics, alarms and events in the monitoring system. Currently, based on Kafka.

==== Messages ====
There are several messages that are published and consumed by various components in Monasca via the MessageQ. See [[Monasca/Message Schema|Message Schema]].

=== Metrics and Alarms Database ===

A high-performance analytics database that can store massive amounts of metrics and alarms in real-time and also support interactive queries. Currently Vertica and InfluxDB are supported.

The SQL schema that is used by Vertica is as follows:

* MonMetrics.Measurements: Stores the actual measurements that are sent.
** id: An integer ID for the measurement.
** definition_dimensions_id: A reference to DefinitionDimensions.
** time_stamp
** value
* MonMetrics.DefinitionDimensions
** id: A sha1 hash of (defintion_id, dimension_set_id)
** definition_id: A reference to the Definitions.id
** dimension_set_id: A reference to the Dimensions.dimension_set_id
* MonMetrics.Definitions
** id: A sha1 hash of the (name, tenant_id, region)
** name: Name of the metric.
** tenant_id: The tenant_id that submitted the metric.
** region: The region the metric was submitted under.
* MonMetric.Dimensions
** dimension_set_id: A sha1 hash of the set of dimenions for a metric.
** name: Name of dimension.
** value: Value of dimension.

=== Config Database ===
The config database store all the configuration information. Currently based on MySQL.

The SQL schema is as follows:

* alarm
** id
** tenant_id
** name
** description
** expression
** state
** actions_enabled
** created_at
** updated_at
** deleted_at
* alarm_action
** alarm_id
** alarm_state
** action_id
* notification_method
** id
** tenant_id
** name
** type
** address
** created_at
** updated_at
* sub_alarm
** id
** alarm_id
** function
** metric_name
** operator
** threshold
** period
** periods
** state
** created_at
** updated_at
* sub_alarm_dimension
** sub_alarm_id
** dimension_name
** value

== Events ==
Support for real-time event stream processing in Monasca is in progress. For more details see the link at, [[Monasca/Events]].

== Logging ==
Support for logging in Monasca is under discussion. For more details see the link at, [[Monasca/Logging]].

== Transform and Aggregation Engine ==
For more details see the link at, [[Monasca/Transform]].

== Analytics ==
Support for anomaly detection and alarm clustering/correlation is in progress. For more details see the link at, [[Monasca/Analytics]].

== Monitoring ==
Enablement and usage for monitoring the status of Monasca is under discussion. For more details see the link at, [[Monasca/Monitoring_Of_Monasca]]

== UI/UX Support ==
Adding more support for common UI/UX queries is under discussion. For more details see the link at, [[Monasca/UI_UX_Support]]

== Keystone Requirements ==
Monasca relies on keystone for running and there are requirements about which keystone configuration must exist.

* The endpoint for the api must be registered in keystone as the 'monasca' service.
* The api must have an admin token to use in verifying the keystone tokens it receives.
* For each project which uses Monasca two users must exist, one will be in the 'monasca-agent' role and be used by the monasca-agent's running on machines. The other should not be in that role and can be used logging into the UI, using the CLI or for direct queries against the API.

== Post Metric Sequence ==

This section describes the sequence of operations involved in posting a metric to the Monasca API.

[[File:monasca-arch-post-metric-diagram.png|Monasca Architecture Post Metric Diagram]]

# A metric is posted to the Monasca API.
# The Monasca API authenticates and validates the request and publishes the metric to the the Message Queue.
# The Persister consumes the metric from the Message Queue and stores in the Metrics Store.
# The Transform Engine consumes the metrics from the Message Queue, performs transform and aggregation operations on metrics, and publishes metrics that it creates back to Message Queue.
# The Threshold Engine consumes metrics from the Message Queue and evaluates alarms. If a state change occurs in an alarm, an "alarm-state-transitioned-event" is published to the Message Queue.
# The Notification Engine consumes "alarm-state-transitioned-events" from the Message Queue, evaluates whether they have a Notification Method associated with it, and sends the appropriate notification, such as email.
# The Persister consumes the "alarm-state-transitioned-event" from the Message Queue and stores it in the Alarm State History Store.

== Alarm Managers ==
This section describes the new features including alarm grouping, inhibition and silencing.

Examples:

* Silence
SilenceRule1 = '{"alarm-silencing-definition-created": {"name": "silence_rule_1", "matchers": {"severity": "LOW"}, "start_time": "2017-02-21 20:00:00", "end_time": "2017-02-21 22:00:00"}}'
Two alarm transitions: AT1 and AT2
AT1_severity = HIGH
AT2_severity = LOW

Output: AT2 get silenced and AT1 send a notification.
SilenceRule1 expires after "end_time"="2017-02-21 22:00:00".

* Inhibit
InhibitionRule1 = '{"alarm-inhibition-definition-created": {"name": "inhibit_rule_1","source":{"severity":"HIGH"},"target":{"severity":"LOW"}, "equals":["tenantId"], "exclusions": {"alarm_name": "vm is dead"}}}'
Three alarm transitions: AT1, AT2 and AT3
AT1_tenantId = "d42bc"
AT2_tenantId = "d42bc"
AT3_tenantId = "d42bc"
AT1_severity = HIGH
AT2_severity = LOW
AT3_severity = LOW
AT1_alarm_name = "cpu high"
AT2_alarm_name = "memory high"
AT3_alarm_name = "vm is dead"
AT1_state = ALARM
AT2_state = ALARM
AT3_state = ALARM

Output: AT1 is the source alarm which will send a notification. AT2 is the target alarm and will get inhibited. AT3 matches the exclusions and will send a notification immediately.

* Grouping Scenario 1

GroupingRule1 = '{"alarm-grouping-definition-created":
{"name": "group_rule_1", "matchers": ["hostname"], "id": "b7163","repeat_interval": "2h", "group_wait": "30s", "exclusions": {"alarm_name": "cpu_percent_high"}, "tenantId": " d42bc", "alarm_actions": ["cd892"], "ok_actions": ["ad892"], "undetermined_actions": ["cf892"]}}'

Three alarm transitions: AT1, AT2 and AT3

AT1_hostname = host1
AT2_hostname = host1
AT3_hostname = host2
AT1_alarm_name = cpu_percent_high
AT2_alarm_name = cpu_system_perc_high
AT3_alarm_name = cpu_percent_high
AT1_state = ALARM
AT2_state = ALARM
AT3_state = ALARM

Output: AT1 and AT3 match exclusions and send notifications immediately. Generate a grouped notification “group_notification_rule_1_host1_alarm[1]” and send out using alarm_actions ["cd892"]. There are no alarm_actions, ok_actions or undermined_actions associated with the AT1, AT2, AT3 alarm definitions.

* Grouping Scenario 2

GroupingRule2 = '{"alarm-grouping-definition-created":
{"name": "group_rule_2", "matchers": ["hostname"], "id": "b7163","repeat_interval": "2h", "group_wait": "30s", "tenantId": " d42bc", "alarm_actions": ["cd892"], "ok_actions": ["ad892"], "undetermined_actions": ["cf892"]}}'

Three alarm transitions: AT1 with "alarm_actions": ["123ab"], AT2 with "alarm_actions": ["cd839"] and AT3

AT1_hostname = host1
AT2_hostname = host1
AT3_hostname = host2
AT1_state = ALARM
AT2_state = ALARM
AT3_state = ALARM

Output: Generate two grouped notifications “group_notification_rule_2_host1_alarm[2]” and “group_notification_rule_2_host2_alarm[1]”. Both using alarm_actions ["cd892"]. Also since AT1 and AT2 has their own alarm actions associated with them, there will be two more notifications sent out.

* Silenced and Grouped

SilenceRule1 = '{"alarm-silencing-definition-created": {"name": "silence_rule_1", "matchers": {"severity": "LOW"}, "start_time": "1487269470498", "end_time": "1587269470498"}}'

GroupingRule2 = '{"alarm-grouping-definition-created":
{"name": "group_rule_2", "matchers": ["hostname"], "id": "b7163","repeat_interval": "2h", "group_wait": "30s", "tenantId": " d42bc", "alarm_actions": ["cd892"], "ok_actions": ["ad892"], "undetermined_actions": ["cf892"]}}'

Four alarm transitions: AT1, AT2, AT3 and AT4

Silencing rule:
AT1_severity = HIGH
AT2_severity = LOW
AT3_severity = HIGH
AT4_severity = HIGH

Grouping rule:
AT1_hostname = host1
AT2_hostname = host1
AT3_hostname = host2
AT4_hostname = host1
AT1_state = ALARM
AT2_state = ALARM
AT3_state = OK
AT4_state = ALARM

Output: Generate two grouped notifications “group_notification_rule_2_host1_alarm[2]” using alarm action "cd892" and “group_notification_rule_2_host2_ok[1]” using ok action "ad892". AT2 is silenced so it isn't include in group_notification_rule_2_host1 count. There are no alarm_actions, ok_actions or undermined_actions associated with the AT1, AT2, AT3, AT4 alarm definitions.

* Silenced and inhibited (source alarm get silenced)

SilenceRule2 = '{"alarm-silencing-definition-created": {"name": "silence_rule_2", "matchers": {"severity": "HIGH, "hostname": "host1"}, "start_time": "2017-02-21 15:00:00", "end_time": "2017-02-21 21:00:00"}}'

InhibitionRule1 = '{"alarm-inhibition-definition-created": {"name": "inhibit_rule_1","source":{"severity":"HIGH"},"target":{"severity":"LOW"}, "equals":["tenantId"]}}'

Alarm transitions: AT1, AT2

Inhibition rule:
AT1_state = ALARM
AT2_state = ALARM
AT1_tenantId = "d42bc"
AT2_tenantId = "d42bc"
AT1_severity = HIGH
AT2_severity = LOW

Silencing rule:
AT1_hostname = host1
AT2_hostname = host2
AT1_severity = HIGH
AT2_severity = LOW

Output: no notification sent out. For inhibition, AT1 is the source alarm, AT2 is the target alarm. But at the same time, AT1 get silenced because it matches the silence rule.

* Inhibited and grouped

InhibitionRule1 = '{"alarm-inhibition-definition-created": {"name": "inhibit_rule_1","source":{"severity":"HIGH"},"target":{"severity":"LOW"}, "equals":["tenantId"]}}'

GroupingRule1 = '{"alarm-grouping-definition-created": {"name": "group_rule_1", "matchers": ["hostname"], "id": "b7163", "repeat_interval": "2h", "group_wait": "30s", "tenantId": " d42bc", "alarm_actions": ["cd892"], "ok_actions": ["ad892"], "undetermined_actions": ["cf892"]}}'

Alarm transitions: AT1, AT2, AT3

Inhibition rule:
AT1_tenantId = "d42bc"
AT2_tenantId = "d42bc"
AT3_tenantId = "d42bc"
AT1_severity = HIGH
AT2_severity = LOW
AT3_severity = HIGH
AT1_state = OK
AT2_state = ALARM
AT3_state = ALARM

Grouping rule:
AT1_hostname = host1
AT2_hostname = host2
AT3_hostname = host1
AT1_state = OK
AT2_state = ALARM
AT3_state = ALARM

Output: AT2 gets inhibited because its severity is low. AT3 is the source alarm. Since AT1 is in OK state, it is not a source alarm. For grouping, AT1 and AT3 has the same host name but different state. So there will be two grouped notifications sent out: “group_notification_rule_1_host1_ok[1]” and “group_notification_rule_1_host1_alarm[1]”.

* Silenced, inhibited and grouped

SilenceRule2 = '{"alarm-silencing-definition-created": {"name": "silence_rule_2", "matchers": {"severity": "HIGH, "hostname": "host1"}, "start_time": "2017-02-21 15:00:00", "end_time": "2017-02-21 21:00:00"}}'

InhibitionRule1 = '{"alarm-inhibition-definition-created": {"name": "inhibit_rule_1","source":{"severity":"HIGH"},"target":{"severity":"LOW"}, "equals":["tenantId"]}}'

GroupingRule1 = '{"alarm-grouping-definition-created": {"name": "group_rule_1", "matchers": ["hostname"], "id": "b7163", "repeat_interval": "2h", "group_wait": "30s", "tenantId": " d42bc", "alarm_actions": ["cd892"], "ok_actions": ["ad892"], "undetermined_actions": ["cf892"]}}'

Alarm transitions: AT1, AT2, AT3 and AT5

Silence rule:
AT1_severity = HIGH AT1_hostname = host1 (silenced)
AT2_severity = LOW AT2_hostname = host2
AT3_severity = HIGH AT3_hostname = host1(silenced)
AT5_severity = HIGH AT5_hostname = host3

Inhibition rule:
AT1_tenantId = "d42bc"
AT2_tenantId = "d42bc"
AT3_tenantId = "d42bc"
AT5_tenantId = "d42bc"
AT1_state = ALARM
AT2_state = ALARM
AT3_state = OK
AT5_state = UNDETERMINED
AT1_severity = HIGH (source)
AT2_severity = LOW (target)
AT3_severity = HIGH
AT5_severity = HIGH

Grouping rule:
AT1_hostname = host1
AT2_hostname = host2
AT3_hostname = host1
AT5_hostname = host3

Output: AT1 is in "group_notification_rule_1_host1_alarm" group and silenced. AT3 is in "group_notification_rule_1_host1_ok" group and silenced. AT2 is in "group_notification_rule_1_host2_alarm" group and inhibited. AT5 is in "group_notification_rule_1_host3_undetermined" group and will send notification “group_notification_rule_1_host3_undetermined[1]” using undetermined action "cf892".

= Development Environment =

* Monasca DevStack Plugin:
** DevStack is the primary developmement environment for OpenStack.
***See http://docs.openstack.org/developer/devstack/
** The Monasca DevStack plugin installs the Monasca Service, Agent, Horizon Monitoring Panel, and Grafana
** README at, https://github.com/openstack/monasca-api/tree/master/devstack
** Best way to get started is to use Vagrant with the Vagrantfile at, https://github.com/openstack/monasca-api/blob/master/devstack/Vagrantfile.

* Monasca on LaunchPad. https://launchpad.net/monasca

* Monasca projects source code. https://git.openstack.org/cgit/?q=monasca

= Coding Standards =

* Python: All Python code conforms to the OpenStack standards at, http://docs.openstack.org/developer/hacking/.
** Note, all components in Monasca, except for the Threshold Engine, have been ported to Python.

* Java: Several of the Monasca components are available as Java. OpenStack does not have any Java coding standards. We've adopted the Google Java Style at, https://google.github.io/styleguide/javaguide.html.
** The standard says either 80 or 100 length lines. We've adopted 100.

= Technologies =

Monasca uses a number of third-party technologies:

* Internal Processing and Middleware
** Apache Kafka (http://kafka.apache.org): Apache Kafka is publish-subscribe messaging rethought as a distributed commit log. Kafka is a highly performant, distributed, fault-tolerant, and scalable message queue with durability built-in.
** Apache Storm (http://storm.incubator.apache.org/): Apache Storm is a free and open source distributed realtime computation system. Storm makes it easy to reliably process unbounded streams of data, doing for realtime processing what Hadoop did for batch processing.
** ZooKeeper (http://zookeeper.apache.org/): Used by Kafka and Storm.
** Apache Spark: Used by Monasca Transform as an aggregation engine.

* Configuration database:
** MySQL: MySQL is supported as a Config Database.
** PostgreSQL: Support for POSTgres, via Hibernate and SQLAlchemy, for the Config Database.

* Vagrant (http://www.vagrantup.com/): Vagrant provides easy to configure, reproducible, and portable work environments built on top of industry-standard technology and controlled by a single consistent workflow to help maximize the productivity and flexibility of you and your team.

* Dropwizard (https://dropwizard.github.io/dropwizard/): Dropwizard pulls together stable, mature libraries from the Java ecosystem into a simple, light-weight package that lets you focus on getting things done. Dropwizard has out-of-the-box support for sophisticated configuration, application metrics, logging, operational tools, and much more, allowing you and your team to ship a production-quality web service in the shortest time possible.

* Time Series Database:
** InfluxDB (http://influxdb.com/): An open-source distributed time series database with no external dependencies. InfluxDB is supported for the Metrics Database.
** Vertica (http://www.vertica.com): A commercial Enterprise class SQL analytics database that is highly scalable. It offers built-in automatic high-availability and excels at in-database analytics and compressing and storing massive amounts of data. A free community version of Vertica is available that can store up to 1 TB of data with no time-limit is available at, https://my.vertica.com/community/. Vertica is supported for the Metrics Database.
** Cassandra: Support for Cassandra for the Metrics Database is in progress.

= License =

Copyright (c) 2014, 2015 Hewlett-Packard Development Company, L.P.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied.
See the License for the specific language governing permissions and
limitations under the License.

[[File:yklogo.png|200px|thumb|left|Monasca uses YourKit Profiler for Java development]]

 
 
 
 

[https://www.yourkit.com/ Visit YourKit website for more information]

Monasca

2019-05-08T17:01:26Z

Joseph Davis: /* Comparisons to alternatives */

== Overview ==

Monasca is a open-source multi-tenant, highly scalable, performant, fault-tolerant monitoring-as-a-service solution that integrates with OpenStack. It uses a REST API for high-speed metrics processing and querying and has a streaming alarm engine and notification engine.

[[File:OpenStack_Project_Monasca_vertical.png|thumb|right]]

==== Contribution ====
* [https://storyboard.openstack.org/#!/project_group/59 Project Group on StoryBoard]
* [https://storyboard.openstack.org/#!/board/111 Important activities in Stein release]
* [https://storyboard.openstack.org/#!/board/141 Important activities in Train release]
* [https://docs.openstack.org/monasca-api/latest/contributor/index.html Contribution Guidelines]

==== Communication and Meetings ====
* IRC: #openstack-monasca on freenode.net
* Weekly Meetings:
** http://eavesdrop.openstack.org/#Monasca_Team_Meeting
** [[Meetings/Monasca]]
* Use [Monasca] tag on [http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-discuss openstack-discuss mailing list].

===== PTG meetings =====
* [[MonascaSteinPTG|Stein PTG Summary (Denver)]]
* (coming soon - Train summary)

==== Documentation ====
: Monasca API Specification: https://github.com/openstack/monasca-api/blob/master/docs/monasca-api-spec.md
: Agent Documentation: https://github.com/openstack/monasca-agent

==== Presentations ====
There have been many interesting and excellent presentations on Monasca, many given at the OpenStack or OpenInfra Summits.
: [[Monasca/Presentations]] (updated with Denver 2019 links)

==== Repositories ====

: '''Code'''
:: Core: https://git.openstack.org/cgit/?q=monasca (old git link)
:: API: https://opendev.org/openstack/monasca-api

: '''Deployment'''
: The following repositories are available for deploying Monasca:
:: Docker: https://github.com/monasca/monasca-docker
:: Kubernetes: https://github.com/monasca/monasca-helm
:: Ansible: https://github.com/search?utf8=%E2%9C%93&q=ansible-monasca
:: Puppet: https://git.openstack.org/openstack/puppet-monasca

==== Bugs ====

===== All open bugs =====
: https://storyboard.openstack.org/#!/worklist/213

===== In Stein triaged bugs =====
: https://storyboard.openstack.org/#!/worklist/467

===== Bug-fixing progress =====
: https://storyboard.openstack.org/#!/board/114

== Requirements ==
[[Monasca/Requirements]].

See also https://opendev.org/openstack/monasca-specs/

== Features ==

This section describes the overall features.

* A highly performant, scalable, reliable and fault-tolerant Monitoring as a Service (MONaaS) solution that scales to service provider metrics levels of metrics throughput. Performance, scalability and high-availability have been designed in from the start. Can process 100s of thousands of metrics/sec as well as offer data retention periods of greater than a year with no data loss while still processing interactive queries.

* Rest API for storing and querying metrics and historical information. Most monitoring solution use special transports and protocols, such as CollectD or NSCA (Nagios). In our solution, http is the only protocol used. This simplifies the overall design and also allows for a much richer way of describing the data via dimensions.

* Multi-tenant and authenticated. Metrics are submitted and authenticated using Keystone and stored associated with a tenant ID.

* Metrics defined using a set of (key, value) pairs called dimensions.

* Real-time thresholding and alarming on metrics.

* Compound alarms described using a simple expressive grammar composed of alarm sub-expressions and logical operators.

* Monitoring agent that supports a number of built-in system and service checks and also supports Nagios checks and statsd.

* Open-source monitoring solution built on open-source technologies.

=== Comparisons to alternatives ===
[[Monasca/Comparison]]

== Architecture ==
[[File:monasca-arch-component-diagram.png|Monasca Architecture Component Diagram]]
* Monitoring Agent (monasca-agent): A modern Python based monitoring agent that consists of several sub-components and supports system metrics, such as cpu utilization and available memory, Nagios plugins, statsd and many built-in checks for services such as MySQL, RabbitMQ, and many others.
* Monitoring API (monasca-api): A well-defined and documented RESTful API for monitoring that is primarily focused on the following concepts and areas:
** Metrics: Store and query massive amounts of metrics in real-time.
** Statistics: Query statistics for metrics.
** Alarm Definitions: Create, update, query and delete alarm definitions.
** Alarms: Query and delete the alarm history.
***Simple expressive grammar for creating compound alarms composed of alarm subexpressions and logical operators.
***Alarm severities can be associated with alarms.
***The complete alarm state transition history is stored and queryable which allows for subsequent root cause analysis (RCA) or advanced analytics.
**Notification Methods: Create and delete notification methods and associate them with alarms, such as email. Supports the ability to notify users directly via email when an alarm state transitions occur.
** The Monasca API has both Java and Python implementations avaialble.
* Persister (monasca-persister): Consumes metrics and alarm state transitions from the MessageQ and stores them in the Metrics and Alarms database.
** The Persister has both Java and Python implementations.
* Transform and Aggregation Engine (monasca-transform): Transform metric names and values, such as delta or time-based derivative calculations, and creates new metrics that are published to the Message Queue. The Transform Engine is not available yet.
* Anomaly and Prediction Engine: Evaluates prediction and anomalies and generates predicted metrics as well as anomaly likelihood and anomaly scores. The Anomaly and Prediction Engine is currently in a prototype status.
* Threshold Engine (monasca-thresh): Computes thresholds on metrics and publishes alarms to the MessageQ when exceeded. Based on Apache Storm a free and open distributed real-time computation system.
* Notification Engine (monasca-notification): Consumes alarm state transition messages from the MessageQ and sends notifications, such as emails for alarms. The Notification Engine is Python based.
* Analytics Engine (monasca-analytics): Consumes alarm state transisitions and metrics from the MessageQ and does anomaly detection and alarm clustering/correlation.
* Message Queue: A third-party component that primarily receives published metrics from the Monitoring API and alarm state transition messages from the Threshold Engine that are consumed by other components, such as the Persister and Notification Engine. The Message Queue is also used to publish and consume other events in the system. Currently, a Kafka based MessageQ is supported. Kafka is a high performance, distributed, fault-tolerant, and scalable message queue with durability built-in. We will look at other alternatives, such as RabbitMQ and in-fact in our previous implementation RabbitMQ was supported, but due to performance, scale, durability and high-availability limitiations with RabbitMQ we have moved to Kafka.
* Metrics and Alarms Database: A third-party component that primarily stores metrics and the alarm state history. Currently, Vertica and InfluxDB are supported. Support for Cassandra is in progress.
* Config Database: A third-party component that stores a lot of the configuration and other information in the system. Currently, MySQL is supported. Support for PostgreSQL is in progress.
* Monitoring Client (python-monascaclient): A Python command line client and library that communicates and controls the Monitoring API. The Monitoring Client was written using the OpenStack Heat Python client as a framework. The Monitoring Client also has a Python library, "monascaclient" similar to the other OpenStack clients, that can be used to quickly build additional capabilities. The Monitoring Client library is used by the Monitoring UI, Ceilometer publisher, and other components.
* Monitoring UI: A Horizon dashboard for visualizing the overall health and status of an OpenStack cloud.
* Ceilometer publisher: A multi-publisher plugin for Ceilometer, not shown, that converts and publishes samples to the Monitoring API.
Most of the components are described in their respective repositories. However, there aren't any repositories for the third-party components used, so we describe some of the relevant details here.

Further historical context for the architecture can be found at [[Monasca/Architecture Details]]

=== Message Schema ===
[[Monasca/Message Schema]]

=== Message Queue ===
A distributed, performant, scalable, HA message queue for distributing metrics, alarms and events in the monitoring system. Currently, based on Kafka.

==== Messages ====
There are several messages that are published and consumed by various components in Monasca via the MessageQ. See [[Monasca/Message Schema|Message Schema]].

=== Metrics and Alarms Database ===

A high-performance analytics database that can store massive amounts of metrics and alarms in real-time and also support interactive queries. Currently Vertica and InfluxDB are supported.

The SQL schema that is used by Vertica is as follows:

* MonMetrics.Measurements: Stores the actual measurements that are sent.
** id: An integer ID for the measurement.
** definition_dimensions_id: A reference to DefinitionDimensions.
** time_stamp
** value
* MonMetrics.DefinitionDimensions
** id: A sha1 hash of (defintion_id, dimension_set_id)
** definition_id: A reference to the Definitions.id
** dimension_set_id: A reference to the Dimensions.dimension_set_id
* MonMetrics.Definitions
** id: A sha1 hash of the (name, tenant_id, region)
** name: Name of the metric.
** tenant_id: The tenant_id that submitted the metric.
** region: The region the metric was submitted under.
* MonMetric.Dimensions
** dimension_set_id: A sha1 hash of the set of dimenions for a metric.
** name: Name of dimension.
** value: Value of dimension.

=== Config Database ===
The config database store all the configuration information. Currently based on MySQL.

The SQL schema is as follows:

* alarm
** id
** tenant_id
** name
** description
** expression
** state
** actions_enabled
** created_at
** updated_at
** deleted_at
* alarm_action
** alarm_id
** alarm_state
** action_id
* notification_method
** id
** tenant_id
** name
** type
** address
** created_at
** updated_at
* sub_alarm
** id
** alarm_id
** function
** metric_name
** operator
** threshold
** period
** periods
** state
** created_at
** updated_at
* sub_alarm_dimension
** sub_alarm_id
** dimension_name
** value

== Events ==
Support for real-time event stream processing in Monasca is in progress. For more details see the link at, [[Monasca/Events]].

== Logging ==
Support for logging in Monasca is under discussion. For more details see the link at, [[Monasca/Logging]].

== Transform and Aggregation Engine ==
For more details see the link at, [[Monasca/Transform]].

== Analytics ==
Support for anomaly detection and alarm clustering/correlation is in progress. For more details see the link at, [[Monasca/Analytics]].

== Monitoring ==
Enablement and usage for monitoring the status of Monasca is under discussion. For more details see the link at, [[Monasca/Monitoring_Of_Monasca]]

== UI/UX Support ==
Adding more support for common UI/UX queries is under discussion. For more details see the link at, [[Monasca/UI_UX_Support]]

== Keystone Requirements ==
Monasca relies on keystone for running and there are requirements about which keystone configuration must exist.

* The endpoint for the api must be registered in keystone as the 'monasca' service.
* The api must have an admin token to use in verifying the keystone tokens it receives.
* For each project which uses Monasca two users must exist, one will be in the 'monasca-agent' role and be used by the monasca-agent's running on machines. The other should not be in that role and can be used logging into the UI, using the CLI or for direct queries against the API.

== Post Metric Sequence ==

This section describes the sequence of operations involved in posting a metric to the Monasca API.

[[File:monasca-arch-post-metric-diagram.png|Monasca Architecture Post Metric Diagram]]

# A metric is posted to the Monasca API.
# The Monasca API authenticates and validates the request and publishes the metric to the the Message Queue.
# The Persister consumes the metric from the Message Queue and stores in the Metrics Store.
# The Transform Engine consumes the metrics from the Message Queue, performs transform and aggregation operations on metrics, and publishes metrics that it creates back to Message Queue.
# The Threshold Engine consumes metrics from the Message Queue and evaluates alarms. If a state change occurs in an alarm, an "alarm-state-transitioned-event" is published to the Message Queue.
# The Notification Engine consumes "alarm-state-transitioned-events" from the Message Queue, evaluates whether they have a Notification Method associated with it, and sends the appropriate notification, such as email.
# The Persister consumes the "alarm-state-transitioned-event" from the Message Queue and stores it in the Alarm State History Store.

== Alarm Managers ==
This section describes the new features including alarm grouping, inhibition and silencing.

Examples:

* Silence
SilenceRule1 = '{"alarm-silencing-definition-created": {"name": "silence_rule_1", "matchers": {"severity": "LOW"}, "start_time": "2017-02-21 20:00:00", "end_time": "2017-02-21 22:00:00"}}'
Two alarm transitions: AT1 and AT2
AT1_severity = HIGH
AT2_severity = LOW

Output: AT2 get silenced and AT1 send a notification.
SilenceRule1 expires after "end_time"="2017-02-21 22:00:00".

* Inhibit
InhibitionRule1 = '{"alarm-inhibition-definition-created": {"name": "inhibit_rule_1","source":{"severity":"HIGH"},"target":{"severity":"LOW"}, "equals":["tenantId"], "exclusions": {"alarm_name": "vm is dead"}}}'
Three alarm transitions: AT1, AT2 and AT3
AT1_tenantId = "d42bc"
AT2_tenantId = "d42bc"
AT3_tenantId = "d42bc"
AT1_severity = HIGH
AT2_severity = LOW
AT3_severity = LOW
AT1_alarm_name = "cpu high"
AT2_alarm_name = "memory high"
AT3_alarm_name = "vm is dead"
AT1_state = ALARM
AT2_state = ALARM
AT3_state = ALARM

Output: AT1 is the source alarm which will send a notification. AT2 is the target alarm and will get inhibited. AT3 matches the exclusions and will send a notification immediately.

* Grouping Scenario 1

GroupingRule1 = '{"alarm-grouping-definition-created":
{"name": "group_rule_1", "matchers": ["hostname"], "id": "b7163","repeat_interval": "2h", "group_wait": "30s", "exclusions": {"alarm_name": "cpu_percent_high"}, "tenantId": " d42bc", "alarm_actions": ["cd892"], "ok_actions": ["ad892"], "undetermined_actions": ["cf892"]}}'

Three alarm transitions: AT1, AT2 and AT3

AT1_hostname = host1
AT2_hostname = host1
AT3_hostname = host2
AT1_alarm_name = cpu_percent_high
AT2_alarm_name = cpu_system_perc_high
AT3_alarm_name = cpu_percent_high
AT1_state = ALARM
AT2_state = ALARM
AT3_state = ALARM

Output: AT1 and AT3 match exclusions and send notifications immediately. Generate a grouped notification “group_notification_rule_1_host1_alarm[1]” and send out using alarm_actions ["cd892"]. There are no alarm_actions, ok_actions or undermined_actions associated with the AT1, AT2, AT3 alarm definitions.

* Grouping Scenario 2

GroupingRule2 = '{"alarm-grouping-definition-created":
{"name": "group_rule_2", "matchers": ["hostname"], "id": "b7163","repeat_interval": "2h", "group_wait": "30s", "tenantId": " d42bc", "alarm_actions": ["cd892"], "ok_actions": ["ad892"], "undetermined_actions": ["cf892"]}}'

Three alarm transitions: AT1 with "alarm_actions": ["123ab"], AT2 with "alarm_actions": ["cd839"] and AT3

AT1_hostname = host1
AT2_hostname = host1
AT3_hostname = host2
AT1_state = ALARM
AT2_state = ALARM
AT3_state = ALARM

Output: Generate two grouped notifications “group_notification_rule_2_host1_alarm[2]” and “group_notification_rule_2_host2_alarm[1]”. Both using alarm_actions ["cd892"]. Also since AT1 and AT2 has their own alarm actions associated with them, there will be two more notifications sent out.

* Silenced and Grouped

SilenceRule1 = '{"alarm-silencing-definition-created": {"name": "silence_rule_1", "matchers": {"severity": "LOW"}, "start_time": "1487269470498", "end_time": "1587269470498"}}'

GroupingRule2 = '{"alarm-grouping-definition-created":
{"name": "group_rule_2", "matchers": ["hostname"], "id": "b7163","repeat_interval": "2h", "group_wait": "30s", "tenantId": " d42bc", "alarm_actions": ["cd892"], "ok_actions": ["ad892"], "undetermined_actions": ["cf892"]}}'

Four alarm transitions: AT1, AT2, AT3 and AT4

Silencing rule:
AT1_severity = HIGH
AT2_severity = LOW
AT3_severity = HIGH
AT4_severity = HIGH

Grouping rule:
AT1_hostname = host1
AT2_hostname = host1
AT3_hostname = host2
AT4_hostname = host1
AT1_state = ALARM
AT2_state = ALARM
AT3_state = OK
AT4_state = ALARM

Output: Generate two grouped notifications “group_notification_rule_2_host1_alarm[2]” using alarm action "cd892" and “group_notification_rule_2_host2_ok[1]” using ok action "ad892". AT2 is silenced so it isn't include in group_notification_rule_2_host1 count. There are no alarm_actions, ok_actions or undermined_actions associated with the AT1, AT2, AT3, AT4 alarm definitions.

* Silenced and inhibited (source alarm get silenced)

SilenceRule2 = '{"alarm-silencing-definition-created": {"name": "silence_rule_2", "matchers": {"severity": "HIGH, "hostname": "host1"}, "start_time": "2017-02-21 15:00:00", "end_time": "2017-02-21 21:00:00"}}'

InhibitionRule1 = '{"alarm-inhibition-definition-created": {"name": "inhibit_rule_1","source":{"severity":"HIGH"},"target":{"severity":"LOW"}, "equals":["tenantId"]}}'

Alarm transitions: AT1, AT2

Inhibition rule:
AT1_state = ALARM
AT2_state = ALARM
AT1_tenantId = "d42bc"
AT2_tenantId = "d42bc"
AT1_severity = HIGH
AT2_severity = LOW

Silencing rule:
AT1_hostname = host1
AT2_hostname = host2
AT1_severity = HIGH
AT2_severity = LOW

Output: no notification sent out. For inhibition, AT1 is the source alarm, AT2 is the target alarm. But at the same time, AT1 get silenced because it matches the silence rule.

* Inhibited and grouped

InhibitionRule1 = '{"alarm-inhibition-definition-created": {"name": "inhibit_rule_1","source":{"severity":"HIGH"},"target":{"severity":"LOW"}, "equals":["tenantId"]}}'

GroupingRule1 = '{"alarm-grouping-definition-created": {"name": "group_rule_1", "matchers": ["hostname"], "id": "b7163", "repeat_interval": "2h", "group_wait": "30s", "tenantId": " d42bc", "alarm_actions": ["cd892"], "ok_actions": ["ad892"], "undetermined_actions": ["cf892"]}}'

Alarm transitions: AT1, AT2, AT3

Inhibition rule:
AT1_tenantId = "d42bc"
AT2_tenantId = "d42bc"
AT3_tenantId = "d42bc"
AT1_severity = HIGH
AT2_severity = LOW
AT3_severity = HIGH
AT1_state = OK
AT2_state = ALARM
AT3_state = ALARM

Grouping rule:
AT1_hostname = host1
AT2_hostname = host2
AT3_hostname = host1
AT1_state = OK
AT2_state = ALARM
AT3_state = ALARM

Output: AT2 gets inhibited because its severity is low. AT3 is the source alarm. Since AT1 is in OK state, it is not a source alarm. For grouping, AT1 and AT3 has the same host name but different state. So there will be two grouped notifications sent out: “group_notification_rule_1_host1_ok[1]” and “group_notification_rule_1_host1_alarm[1]”.

* Silenced, inhibited and grouped

SilenceRule2 = '{"alarm-silencing-definition-created": {"name": "silence_rule_2", "matchers": {"severity": "HIGH, "hostname": "host1"}, "start_time": "2017-02-21 15:00:00", "end_time": "2017-02-21 21:00:00"}}'

InhibitionRule1 = '{"alarm-inhibition-definition-created": {"name": "inhibit_rule_1","source":{"severity":"HIGH"},"target":{"severity":"LOW"}, "equals":["tenantId"]}}'

GroupingRule1 = '{"alarm-grouping-definition-created": {"name": "group_rule_1", "matchers": ["hostname"], "id": "b7163", "repeat_interval": "2h", "group_wait": "30s", "tenantId": " d42bc", "alarm_actions": ["cd892"], "ok_actions": ["ad892"], "undetermined_actions": ["cf892"]}}'

Alarm transitions: AT1, AT2, AT3 and AT5

Silence rule:
AT1_severity = HIGH AT1_hostname = host1 (silenced)
AT2_severity = LOW AT2_hostname = host2
AT3_severity = HIGH AT3_hostname = host1(silenced)
AT5_severity = HIGH AT5_hostname = host3

Inhibition rule:
AT1_tenantId = "d42bc"
AT2_tenantId = "d42bc"
AT3_tenantId = "d42bc"
AT5_tenantId = "d42bc"
AT1_state = ALARM
AT2_state = ALARM
AT3_state = OK
AT5_state = UNDETERMINED
AT1_severity = HIGH (source)
AT2_severity = LOW (target)
AT3_severity = HIGH
AT5_severity = HIGH

Grouping rule:
AT1_hostname = host1
AT2_hostname = host2
AT3_hostname = host1
AT5_hostname = host3

Output: AT1 is in "group_notification_rule_1_host1_alarm" group and silenced. AT3 is in "group_notification_rule_1_host1_ok" group and silenced. AT2 is in "group_notification_rule_1_host2_alarm" group and inhibited. AT5 is in "group_notification_rule_1_host3_undetermined" group and will send notification “group_notification_rule_1_host3_undetermined[1]” using undetermined action "cf892".

= Development Environment =

* Monasca DevStack Plugin:
** DevStack is the primary developmement environment for OpenStack.
***See http://docs.openstack.org/developer/devstack/
** The Monasca DevStack plugin installs the Monasca Service, Agent, Horizon Monitoring Panel, and Grafana
** README at, https://github.com/openstack/monasca-api/tree/master/devstack
** Best way to get started is to use Vagrant with the Vagrantfile at, https://github.com/openstack/monasca-api/blob/master/devstack/Vagrantfile.

* Monasca on LaunchPad. https://launchpad.net/monasca

* Monasca projects source code. https://git.openstack.org/cgit/?q=monasca

= Coding Standards =

* Python: All Python code conforms to the OpenStack standards at, http://docs.openstack.org/developer/hacking/.
** Note, all components in Monasca, except for the Threshold Engine, have been ported to Python.

* Java: Several of the Monasca components are available as Java. OpenStack does not have any Java coding standards. We've adopted the Google Java Style at, https://google.github.io/styleguide/javaguide.html.
** The standard says either 80 or 100 length lines. We've adopted 100.

= Technologies =

Monasca uses a number of third-party technologies:

* Apache Kafka (http://kafka.apache.org): Apache Kafka is publish-subscribe messaging rethought as a distributed commit log. Kafka is a highly performant, distributed, fault-tolerant, and scalable message queue with durability built-in.

* Apache Storm (http://storm.incubator.apache.org/): Apache Storm is a free and open source distributed realtime computation system. Storm makes it easy to reliably process unbounded streams of data, doing for realtime processing what Hadoop did for batch processing.

* ZooKeeper (http://zookeeper.apache.org/): Used by Kafka and Storm.

* MySQL: MySQL is supported as a Config Database.

* POSTgres: Support for POSTgres, via Hibernate and SQLAlchemy, for the Config Database.

* Vagrant (http://www.vagrantup.com/): Vagrant provides easy to configure, reproducible, and portable work environments built on top of industry-standard technology and controlled by a single consistent workflow to help maximize the productivity and flexibility of you and your team.

* Dropwizard (https://dropwizard.github.io/dropwizard/): Dropwizard pulls together stable, mature libraries from the Java ecosystem into a simple, light-weight package that lets you focus on getting things done. Dropwizard has out-of-the-box support for sophisticated configuration, application metrics, logging, operational tools, and much more, allowing you and your team to ship a production-quality web service in the shortest time possible.

* InfluxDB (http://influxdb.com/): An open-source distributed time series database with no external dependencies. InfluxDB is supported for the Metrics Database.

* Vertica (http://www.vertica.com): A commercial Enterprise class SQL analytics database that is highly scalable. It offers built-in automatic high-availability and excels at in-database analytics and compressing and storing massive amounts of data. A free community version of Vertica is available that can store up to 1 TB of data with no time-limit is available at, https://my.vertica.com/community/. Vertica is supported for the Metrics Database.

* Cassandra: Support for Cassandra for the Metrics Database is in progress.

= License =

Copyright (c) 2014, 2015 Hewlett-Packard Development Company, L.P.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied.
See the License for the specific language governing permissions and
limitations under the License.

[[File:yklogo.png|200px|thumb|left|Monasca uses YourKit Profiler for Java development]]

 
 
 
 

[https://www.yourkit.com/ Visit YourKit website for more information]

Monasca

2019-05-08T17:00:58Z

Joseph Davis: /* Requirements */

== Overview ==

Monasca is a open-source multi-tenant, highly scalable, performant, fault-tolerant monitoring-as-a-service solution that integrates with OpenStack. It uses a REST API for high-speed metrics processing and querying and has a streaming alarm engine and notification engine.

[[File:OpenStack_Project_Monasca_vertical.png|thumb|right]]

==== Contribution ====
* [https://storyboard.openstack.org/#!/project_group/59 Project Group on StoryBoard]
* [https://storyboard.openstack.org/#!/board/111 Important activities in Stein release]
* [https://storyboard.openstack.org/#!/board/141 Important activities in Train release]
* [https://docs.openstack.org/monasca-api/latest/contributor/index.html Contribution Guidelines]

==== Communication and Meetings ====
* IRC: #openstack-monasca on freenode.net
* Weekly Meetings:
** http://eavesdrop.openstack.org/#Monasca_Team_Meeting
** [[Meetings/Monasca]]
* Use [Monasca] tag on [http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-discuss openstack-discuss mailing list].

===== PTG meetings =====
* [[MonascaSteinPTG|Stein PTG Summary (Denver)]]
* (coming soon - Train summary)

==== Documentation ====
: Monasca API Specification: https://github.com/openstack/monasca-api/blob/master/docs/monasca-api-spec.md
: Agent Documentation: https://github.com/openstack/monasca-agent

==== Presentations ====
There have been many interesting and excellent presentations on Monasca, many given at the OpenStack or OpenInfra Summits.
: [[Monasca/Presentations]] (updated with Denver 2019 links)

==== Repositories ====

: '''Code'''
:: Core: https://git.openstack.org/cgit/?q=monasca (old git link)
:: API: https://opendev.org/openstack/monasca-api

: '''Deployment'''
: The following repositories are available for deploying Monasca:
:: Docker: https://github.com/monasca/monasca-docker
:: Kubernetes: https://github.com/monasca/monasca-helm
:: Ansible: https://github.com/search?utf8=%E2%9C%93&q=ansible-monasca
:: Puppet: https://git.openstack.org/openstack/puppet-monasca

==== Bugs ====

===== All open bugs =====
: https://storyboard.openstack.org/#!/worklist/213

===== In Stein triaged bugs =====
: https://storyboard.openstack.org/#!/worklist/467

===== Bug-fixing progress =====
: https://storyboard.openstack.org/#!/board/114

== Requirements ==
[[Monasca/Requirements]].

See also https://opendev.org/openstack/monasca-specs/

== Features ==

This section describes the overall features.

* A highly performant, scalable, reliable and fault-tolerant Monitoring as a Service (MONaaS) solution that scales to service provider metrics levels of metrics throughput. Performance, scalability and high-availability have been designed in from the start. Can process 100s of thousands of metrics/sec as well as offer data retention periods of greater than a year with no data loss while still processing interactive queries.

* Rest API for storing and querying metrics and historical information. Most monitoring solution use special transports and protocols, such as CollectD or NSCA (Nagios). In our solution, http is the only protocol used. This simplifies the overall design and also allows for a much richer way of describing the data via dimensions.

* Multi-tenant and authenticated. Metrics are submitted and authenticated using Keystone and stored associated with a tenant ID.

* Metrics defined using a set of (key, value) pairs called dimensions.

* Real-time thresholding and alarming on metrics.

* Compound alarms described using a simple expressive grammar composed of alarm sub-expressions and logical operators.

* Monitoring agent that supports a number of built-in system and service checks and also supports Nagios checks and statsd.

* Open-source monitoring solution built on open-source technologies.

== Comparisons to alternatives ==
[[Monasca/Comparison]]

== Architecture ==
[[File:monasca-arch-component-diagram.png|Monasca Architecture Component Diagram]]
* Monitoring Agent (monasca-agent): A modern Python based monitoring agent that consists of several sub-components and supports system metrics, such as cpu utilization and available memory, Nagios plugins, statsd and many built-in checks for services such as MySQL, RabbitMQ, and many others.
* Monitoring API (monasca-api): A well-defined and documented RESTful API for monitoring that is primarily focused on the following concepts and areas:
** Metrics: Store and query massive amounts of metrics in real-time.
** Statistics: Query statistics for metrics.
** Alarm Definitions: Create, update, query and delete alarm definitions.
** Alarms: Query and delete the alarm history.
***Simple expressive grammar for creating compound alarms composed of alarm subexpressions and logical operators.
***Alarm severities can be associated with alarms.
***The complete alarm state transition history is stored and queryable which allows for subsequent root cause analysis (RCA) or advanced analytics.
**Notification Methods: Create and delete notification methods and associate them with alarms, such as email. Supports the ability to notify users directly via email when an alarm state transitions occur.
** The Monasca API has both Java and Python implementations avaialble.
* Persister (monasca-persister): Consumes metrics and alarm state transitions from the MessageQ and stores them in the Metrics and Alarms database.
** The Persister has both Java and Python implementations.
* Transform and Aggregation Engine (monasca-transform): Transform metric names and values, such as delta or time-based derivative calculations, and creates new metrics that are published to the Message Queue. The Transform Engine is not available yet.
* Anomaly and Prediction Engine: Evaluates prediction and anomalies and generates predicted metrics as well as anomaly likelihood and anomaly scores. The Anomaly and Prediction Engine is currently in a prototype status.
* Threshold Engine (monasca-thresh): Computes thresholds on metrics and publishes alarms to the MessageQ when exceeded. Based on Apache Storm a free and open distributed real-time computation system.
* Notification Engine (monasca-notification): Consumes alarm state transition messages from the MessageQ and sends notifications, such as emails for alarms. The Notification Engine is Python based.
* Analytics Engine (monasca-analytics): Consumes alarm state transisitions and metrics from the MessageQ and does anomaly detection and alarm clustering/correlation.
* Message Queue: A third-party component that primarily receives published metrics from the Monitoring API and alarm state transition messages from the Threshold Engine that are consumed by other components, such as the Persister and Notification Engine. The Message Queue is also used to publish and consume other events in the system. Currently, a Kafka based MessageQ is supported. Kafka is a high performance, distributed, fault-tolerant, and scalable message queue with durability built-in. We will look at other alternatives, such as RabbitMQ and in-fact in our previous implementation RabbitMQ was supported, but due to performance, scale, durability and high-availability limitiations with RabbitMQ we have moved to Kafka.
* Metrics and Alarms Database: A third-party component that primarily stores metrics and the alarm state history. Currently, Vertica and InfluxDB are supported. Support for Cassandra is in progress.
* Config Database: A third-party component that stores a lot of the configuration and other information in the system. Currently, MySQL is supported. Support for PostgreSQL is in progress.
* Monitoring Client (python-monascaclient): A Python command line client and library that communicates and controls the Monitoring API. The Monitoring Client was written using the OpenStack Heat Python client as a framework. The Monitoring Client also has a Python library, "monascaclient" similar to the other OpenStack clients, that can be used to quickly build additional capabilities. The Monitoring Client library is used by the Monitoring UI, Ceilometer publisher, and other components.
* Monitoring UI: A Horizon dashboard for visualizing the overall health and status of an OpenStack cloud.
* Ceilometer publisher: A multi-publisher plugin for Ceilometer, not shown, that converts and publishes samples to the Monitoring API.
Most of the components are described in their respective repositories. However, there aren't any repositories for the third-party components used, so we describe some of the relevant details here.

Further historical context for the architecture can be found at [[Monasca/Architecture Details]]

=== Message Schema ===
[[Monasca/Message Schema]]

=== Message Queue ===
A distributed, performant, scalable, HA message queue for distributing metrics, alarms and events in the monitoring system. Currently, based on Kafka.

==== Messages ====
There are several messages that are published and consumed by various components in Monasca via the MessageQ. See [[Monasca/Message Schema|Message Schema]].

=== Metrics and Alarms Database ===

A high-performance analytics database that can store massive amounts of metrics and alarms in real-time and also support interactive queries. Currently Vertica and InfluxDB are supported.

The SQL schema that is used by Vertica is as follows:

* MonMetrics.Measurements: Stores the actual measurements that are sent.
** id: An integer ID for the measurement.
** definition_dimensions_id: A reference to DefinitionDimensions.
** time_stamp
** value
* MonMetrics.DefinitionDimensions
** id: A sha1 hash of (defintion_id, dimension_set_id)
** definition_id: A reference to the Definitions.id
** dimension_set_id: A reference to the Dimensions.dimension_set_id
* MonMetrics.Definitions
** id: A sha1 hash of the (name, tenant_id, region)
** name: Name of the metric.
** tenant_id: The tenant_id that submitted the metric.
** region: The region the metric was submitted under.
* MonMetric.Dimensions
** dimension_set_id: A sha1 hash of the set of dimenions for a metric.
** name: Name of dimension.
** value: Value of dimension.

=== Config Database ===
The config database store all the configuration information. Currently based on MySQL.

The SQL schema is as follows:

* alarm
** id
** tenant_id
** name
** description
** expression
** state
** actions_enabled
** created_at
** updated_at
** deleted_at
* alarm_action
** alarm_id
** alarm_state
** action_id
* notification_method
** id
** tenant_id
** name
** type
** address
** created_at
** updated_at
* sub_alarm
** id
** alarm_id
** function
** metric_name
** operator
** threshold
** period
** periods
** state
** created_at
** updated_at
* sub_alarm_dimension
** sub_alarm_id
** dimension_name
** value

== Events ==
Support for real-time event stream processing in Monasca is in progress. For more details see the link at, [[Monasca/Events]].

== Logging ==
Support for logging in Monasca is under discussion. For more details see the link at, [[Monasca/Logging]].

== Transform and Aggregation Engine ==
For more details see the link at, [[Monasca/Transform]].

== Analytics ==
Support for anomaly detection and alarm clustering/correlation is in progress. For more details see the link at, [[Monasca/Analytics]].

== Monitoring ==
Enablement and usage for monitoring the status of Monasca is under discussion. For more details see the link at, [[Monasca/Monitoring_Of_Monasca]]

== UI/UX Support ==
Adding more support for common UI/UX queries is under discussion. For more details see the link at, [[Monasca/UI_UX_Support]]

== Keystone Requirements ==
Monasca relies on keystone for running and there are requirements about which keystone configuration must exist.

* The endpoint for the api must be registered in keystone as the 'monasca' service.
* The api must have an admin token to use in verifying the keystone tokens it receives.
* For each project which uses Monasca two users must exist, one will be in the 'monasca-agent' role and be used by the monasca-agent's running on machines. The other should not be in that role and can be used logging into the UI, using the CLI or for direct queries against the API.

== Post Metric Sequence ==

This section describes the sequence of operations involved in posting a metric to the Monasca API.

[[File:monasca-arch-post-metric-diagram.png|Monasca Architecture Post Metric Diagram]]

# A metric is posted to the Monasca API.
# The Monasca API authenticates and validates the request and publishes the metric to the the Message Queue.
# The Persister consumes the metric from the Message Queue and stores in the Metrics Store.
# The Transform Engine consumes the metrics from the Message Queue, performs transform and aggregation operations on metrics, and publishes metrics that it creates back to Message Queue.
# The Threshold Engine consumes metrics from the Message Queue and evaluates alarms. If a state change occurs in an alarm, an "alarm-state-transitioned-event" is published to the Message Queue.
# The Notification Engine consumes "alarm-state-transitioned-events" from the Message Queue, evaluates whether they have a Notification Method associated with it, and sends the appropriate notification, such as email.
# The Persister consumes the "alarm-state-transitioned-event" from the Message Queue and stores it in the Alarm State History Store.

== Alarm Managers ==
This section describes the new features including alarm grouping, inhibition and silencing.

Examples:

* Silence
SilenceRule1 = '{"alarm-silencing-definition-created": {"name": "silence_rule_1", "matchers": {"severity": "LOW"}, "start_time": "2017-02-21 20:00:00", "end_time": "2017-02-21 22:00:00"}}'
Two alarm transitions: AT1 and AT2
AT1_severity = HIGH
AT2_severity = LOW

Output: AT2 get silenced and AT1 send a notification.
SilenceRule1 expires after "end_time"="2017-02-21 22:00:00".

* Inhibit
InhibitionRule1 = '{"alarm-inhibition-definition-created": {"name": "inhibit_rule_1","source":{"severity":"HIGH"},"target":{"severity":"LOW"}, "equals":["tenantId"], "exclusions": {"alarm_name": "vm is dead"}}}'
Three alarm transitions: AT1, AT2 and AT3
AT1_tenantId = "d42bc"
AT2_tenantId = "d42bc"
AT3_tenantId = "d42bc"
AT1_severity = HIGH
AT2_severity = LOW
AT3_severity = LOW
AT1_alarm_name = "cpu high"
AT2_alarm_name = "memory high"
AT3_alarm_name = "vm is dead"
AT1_state = ALARM
AT2_state = ALARM
AT3_state = ALARM

Output: AT1 is the source alarm which will send a notification. AT2 is the target alarm and will get inhibited. AT3 matches the exclusions and will send a notification immediately.

* Grouping Scenario 1

GroupingRule1 = '{"alarm-grouping-definition-created":
{"name": "group_rule_1", "matchers": ["hostname"], "id": "b7163","repeat_interval": "2h", "group_wait": "30s", "exclusions": {"alarm_name": "cpu_percent_high"}, "tenantId": " d42bc", "alarm_actions": ["cd892"], "ok_actions": ["ad892"], "undetermined_actions": ["cf892"]}}'

Three alarm transitions: AT1, AT2 and AT3

AT1_hostname = host1
AT2_hostname = host1
AT3_hostname = host2
AT1_alarm_name = cpu_percent_high
AT2_alarm_name = cpu_system_perc_high
AT3_alarm_name = cpu_percent_high
AT1_state = ALARM
AT2_state = ALARM
AT3_state = ALARM

Output: AT1 and AT3 match exclusions and send notifications immediately. Generate a grouped notification “group_notification_rule_1_host1_alarm[1]” and send out using alarm_actions ["cd892"]. There are no alarm_actions, ok_actions or undermined_actions associated with the AT1, AT2, AT3 alarm definitions.

* Grouping Scenario 2

GroupingRule2 = '{"alarm-grouping-definition-created":
{"name": "group_rule_2", "matchers": ["hostname"], "id": "b7163","repeat_interval": "2h", "group_wait": "30s", "tenantId": " d42bc", "alarm_actions": ["cd892"], "ok_actions": ["ad892"], "undetermined_actions": ["cf892"]}}'

Three alarm transitions: AT1 with "alarm_actions": ["123ab"], AT2 with "alarm_actions": ["cd839"] and AT3

AT1_hostname = host1
AT2_hostname = host1
AT3_hostname = host2
AT1_state = ALARM
AT2_state = ALARM
AT3_state = ALARM

Output: Generate two grouped notifications “group_notification_rule_2_host1_alarm[2]” and “group_notification_rule_2_host2_alarm[1]”. Both using alarm_actions ["cd892"]. Also since AT1 and AT2 has their own alarm actions associated with them, there will be two more notifications sent out.

* Silenced and Grouped

SilenceRule1 = '{"alarm-silencing-definition-created": {"name": "silence_rule_1", "matchers": {"severity": "LOW"}, "start_time": "1487269470498", "end_time": "1587269470498"}}'

GroupingRule2 = '{"alarm-grouping-definition-created":
{"name": "group_rule_2", "matchers": ["hostname"], "id": "b7163","repeat_interval": "2h", "group_wait": "30s", "tenantId": " d42bc", "alarm_actions": ["cd892"], "ok_actions": ["ad892"], "undetermined_actions": ["cf892"]}}'

Four alarm transitions: AT1, AT2, AT3 and AT4

Silencing rule:
AT1_severity = HIGH
AT2_severity = LOW
AT3_severity = HIGH
AT4_severity = HIGH

Grouping rule:
AT1_hostname = host1
AT2_hostname = host1
AT3_hostname = host2
AT4_hostname = host1
AT1_state = ALARM
AT2_state = ALARM
AT3_state = OK
AT4_state = ALARM

Output: Generate two grouped notifications “group_notification_rule_2_host1_alarm[2]” using alarm action "cd892" and “group_notification_rule_2_host2_ok[1]” using ok action "ad892". AT2 is silenced so it isn't include in group_notification_rule_2_host1 count. There are no alarm_actions, ok_actions or undermined_actions associated with the AT1, AT2, AT3, AT4 alarm definitions.

* Silenced and inhibited (source alarm get silenced)

SilenceRule2 = '{"alarm-silencing-definition-created": {"name": "silence_rule_2", "matchers": {"severity": "HIGH, "hostname": "host1"}, "start_time": "2017-02-21 15:00:00", "end_time": "2017-02-21 21:00:00"}}'

InhibitionRule1 = '{"alarm-inhibition-definition-created": {"name": "inhibit_rule_1","source":{"severity":"HIGH"},"target":{"severity":"LOW"}, "equals":["tenantId"]}}'

Alarm transitions: AT1, AT2

Inhibition rule:
AT1_state = ALARM
AT2_state = ALARM
AT1_tenantId = "d42bc"
AT2_tenantId = "d42bc"
AT1_severity = HIGH
AT2_severity = LOW

Silencing rule:
AT1_hostname = host1
AT2_hostname = host2
AT1_severity = HIGH
AT2_severity = LOW

Output: no notification sent out. For inhibition, AT1 is the source alarm, AT2 is the target alarm. But at the same time, AT1 get silenced because it matches the silence rule.

* Inhibited and grouped

InhibitionRule1 = '{"alarm-inhibition-definition-created": {"name": "inhibit_rule_1","source":{"severity":"HIGH"},"target":{"severity":"LOW"}, "equals":["tenantId"]}}'

GroupingRule1 = '{"alarm-grouping-definition-created": {"name": "group_rule_1", "matchers": ["hostname"], "id": "b7163", "repeat_interval": "2h", "group_wait": "30s", "tenantId": " d42bc", "alarm_actions": ["cd892"], "ok_actions": ["ad892"], "undetermined_actions": ["cf892"]}}'

Alarm transitions: AT1, AT2, AT3

Inhibition rule:
AT1_tenantId = "d42bc"
AT2_tenantId = "d42bc"
AT3_tenantId = "d42bc"
AT1_severity = HIGH
AT2_severity = LOW
AT3_severity = HIGH
AT1_state = OK
AT2_state = ALARM
AT3_state = ALARM

Grouping rule:
AT1_hostname = host1
AT2_hostname = host2
AT3_hostname = host1
AT1_state = OK
AT2_state = ALARM
AT3_state = ALARM

Output: AT2 gets inhibited because its severity is low. AT3 is the source alarm. Since AT1 is in OK state, it is not a source alarm. For grouping, AT1 and AT3 has the same host name but different state. So there will be two grouped notifications sent out: “group_notification_rule_1_host1_ok[1]” and “group_notification_rule_1_host1_alarm[1]”.

* Silenced, inhibited and grouped

SilenceRule2 = '{"alarm-silencing-definition-created": {"name": "silence_rule_2", "matchers": {"severity": "HIGH, "hostname": "host1"}, "start_time": "2017-02-21 15:00:00", "end_time": "2017-02-21 21:00:00"}}'

InhibitionRule1 = '{"alarm-inhibition-definition-created": {"name": "inhibit_rule_1","source":{"severity":"HIGH"},"target":{"severity":"LOW"}, "equals":["tenantId"]}}'

GroupingRule1 = '{"alarm-grouping-definition-created": {"name": "group_rule_1", "matchers": ["hostname"], "id": "b7163", "repeat_interval": "2h", "group_wait": "30s", "tenantId": " d42bc", "alarm_actions": ["cd892"], "ok_actions": ["ad892"], "undetermined_actions": ["cf892"]}}'

Alarm transitions: AT1, AT2, AT3 and AT5

Silence rule:
AT1_severity = HIGH AT1_hostname = host1 (silenced)
AT2_severity = LOW AT2_hostname = host2
AT3_severity = HIGH AT3_hostname = host1(silenced)
AT5_severity = HIGH AT5_hostname = host3

Inhibition rule:
AT1_tenantId = "d42bc"
AT2_tenantId = "d42bc"
AT3_tenantId = "d42bc"
AT5_tenantId = "d42bc"
AT1_state = ALARM
AT2_state = ALARM
AT3_state = OK
AT5_state = UNDETERMINED
AT1_severity = HIGH (source)
AT2_severity = LOW (target)
AT3_severity = HIGH
AT5_severity = HIGH

Grouping rule:
AT1_hostname = host1
AT2_hostname = host2
AT3_hostname = host1
AT5_hostname = host3

Output: AT1 is in "group_notification_rule_1_host1_alarm" group and silenced. AT3 is in "group_notification_rule_1_host1_ok" group and silenced. AT2 is in "group_notification_rule_1_host2_alarm" group and inhibited. AT5 is in "group_notification_rule_1_host3_undetermined" group and will send notification “group_notification_rule_1_host3_undetermined[1]” using undetermined action "cf892".

= Development Environment =

* Monasca DevStack Plugin:
** DevStack is the primary developmement environment for OpenStack.
***See http://docs.openstack.org/developer/devstack/
** The Monasca DevStack plugin installs the Monasca Service, Agent, Horizon Monitoring Panel, and Grafana
** README at, https://github.com/openstack/monasca-api/tree/master/devstack
** Best way to get started is to use Vagrant with the Vagrantfile at, https://github.com/openstack/monasca-api/blob/master/devstack/Vagrantfile.

* Monasca on LaunchPad. https://launchpad.net/monasca

* Monasca projects source code. https://git.openstack.org/cgit/?q=monasca

= Coding Standards =

* Python: All Python code conforms to the OpenStack standards at, http://docs.openstack.org/developer/hacking/.
** Note, all components in Monasca, except for the Threshold Engine, have been ported to Python.

* Java: Several of the Monasca components are available as Java. OpenStack does not have any Java coding standards. We've adopted the Google Java Style at, https://google.github.io/styleguide/javaguide.html.
** The standard says either 80 or 100 length lines. We've adopted 100.

= Technologies =

Monasca uses a number of third-party technologies:

* Apache Kafka (http://kafka.apache.org): Apache Kafka is publish-subscribe messaging rethought as a distributed commit log. Kafka is a highly performant, distributed, fault-tolerant, and scalable message queue with durability built-in.

* Apache Storm (http://storm.incubator.apache.org/): Apache Storm is a free and open source distributed realtime computation system. Storm makes it easy to reliably process unbounded streams of data, doing for realtime processing what Hadoop did for batch processing.

* ZooKeeper (http://zookeeper.apache.org/): Used by Kafka and Storm.

* MySQL: MySQL is supported as a Config Database.

* POSTgres: Support for POSTgres, via Hibernate and SQLAlchemy, for the Config Database.

* Vagrant (http://www.vagrantup.com/): Vagrant provides easy to configure, reproducible, and portable work environments built on top of industry-standard technology and controlled by a single consistent workflow to help maximize the productivity and flexibility of you and your team.

* Dropwizard (https://dropwizard.github.io/dropwizard/): Dropwizard pulls together stable, mature libraries from the Java ecosystem into a simple, light-weight package that lets you focus on getting things done. Dropwizard has out-of-the-box support for sophisticated configuration, application metrics, logging, operational tools, and much more, allowing you and your team to ship a production-quality web service in the shortest time possible.

* InfluxDB (http://influxdb.com/): An open-source distributed time series database with no external dependencies. InfluxDB is supported for the Metrics Database.

* Vertica (http://www.vertica.com): A commercial Enterprise class SQL analytics database that is highly scalable. It offers built-in automatic high-availability and excels at in-database analytics and compressing and storing massive amounts of data. A free community version of Vertica is available that can store up to 1 TB of data with no time-limit is available at, https://my.vertica.com/community/. Vertica is supported for the Metrics Database.

* Cassandra: Support for Cassandra for the Metrics Database is in progress.

= License =

Copyright (c) 2014, 2015 Hewlett-Packard Development Company, L.P.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied.
See the License for the specific language governing permissions and
limitations under the License.

[[File:yklogo.png|200px|thumb|left|Monasca uses YourKit Profiler for Java development]]

 
 
 
 

[https://www.yourkit.com/ Visit YourKit website for more information]

Monasca

2019-05-08T16:58:37Z

Joseph Davis: /* Repositories */

== Overview ==

Monasca is a open-source multi-tenant, highly scalable, performant, fault-tolerant monitoring-as-a-service solution that integrates with OpenStack. It uses a REST API for high-speed metrics processing and querying and has a streaming alarm engine and notification engine.

[[File:OpenStack_Project_Monasca_vertical.png|thumb|right]]

==== Contribution ====
* [https://storyboard.openstack.org/#!/project_group/59 Project Group on StoryBoard]
* [https://storyboard.openstack.org/#!/board/111 Important activities in Stein release]
* [https://storyboard.openstack.org/#!/board/141 Important activities in Train release]
* [https://docs.openstack.org/monasca-api/latest/contributor/index.html Contribution Guidelines]

==== Communication and Meetings ====
* IRC: #openstack-monasca on freenode.net
* Weekly Meetings:
** http://eavesdrop.openstack.org/#Monasca_Team_Meeting
** [[Meetings/Monasca]]
* Use [Monasca] tag on [http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-discuss openstack-discuss mailing list].

===== PTG meetings =====
* [[MonascaSteinPTG|Stein PTG Summary (Denver)]]
* (coming soon - Train summary)

==== Documentation ====
: Monasca API Specification: https://github.com/openstack/monasca-api/blob/master/docs/monasca-api-spec.md
: Agent Documentation: https://github.com/openstack/monasca-agent

==== Presentations ====
There have been many interesting and excellent presentations on Monasca, many given at the OpenStack or OpenInfra Summits.
: [[Monasca/Presentations]] (updated with Denver 2019 links)

==== Repositories ====

: '''Code'''
:: Core: https://git.openstack.org/cgit/?q=monasca (old git link)
:: API: https://opendev.org/openstack/monasca-api

: '''Deployment'''
: The following repositories are available for deploying Monasca:
:: Docker: https://github.com/monasca/monasca-docker
:: Kubernetes: https://github.com/monasca/monasca-helm
:: Ansible: https://github.com/search?utf8=%E2%9C%93&q=ansible-monasca
:: Puppet: https://git.openstack.org/openstack/puppet-monasca

==== Bugs ====

===== All open bugs =====
: https://storyboard.openstack.org/#!/worklist/213

===== In Stein triaged bugs =====
: https://storyboard.openstack.org/#!/worklist/467

===== Bug-fixing progress =====
: https://storyboard.openstack.org/#!/board/114

== Requirements ==
[[Monasca/Requirements]].

== Features ==

This section describes the overall features.

* A highly performant, scalable, reliable and fault-tolerant Monitoring as a Service (MONaaS) solution that scales to service provider metrics levels of metrics throughput. Performance, scalability and high-availability have been designed in from the start. Can process 100s of thousands of metrics/sec as well as offer data retention periods of greater than a year with no data loss while still processing interactive queries.

* Rest API for storing and querying metrics and historical information. Most monitoring solution use special transports and protocols, such as CollectD or NSCA (Nagios). In our solution, http is the only protocol used. This simplifies the overall design and also allows for a much richer way of describing the data via dimensions.

* Multi-tenant and authenticated. Metrics are submitted and authenticated using Keystone and stored associated with a tenant ID.

* Metrics defined using a set of (key, value) pairs called dimensions.

* Real-time thresholding and alarming on metrics.

* Compound alarms described using a simple expressive grammar composed of alarm sub-expressions and logical operators.

* Monitoring agent that supports a number of built-in system and service checks and also supports Nagios checks and statsd.

* Open-source monitoring solution built on open-source technologies.

== Comparisons to alternatives ==
[[Monasca/Comparison]]

== Architecture ==
[[File:monasca-arch-component-diagram.png|Monasca Architecture Component Diagram]]
* Monitoring Agent (monasca-agent): A modern Python based monitoring agent that consists of several sub-components and supports system metrics, such as cpu utilization and available memory, Nagios plugins, statsd and many built-in checks for services such as MySQL, RabbitMQ, and many others.
* Monitoring API (monasca-api): A well-defined and documented RESTful API for monitoring that is primarily focused on the following concepts and areas:
** Metrics: Store and query massive amounts of metrics in real-time.
** Statistics: Query statistics for metrics.
** Alarm Definitions: Create, update, query and delete alarm definitions.
** Alarms: Query and delete the alarm history.
***Simple expressive grammar for creating compound alarms composed of alarm subexpressions and logical operators.
***Alarm severities can be associated with alarms.
***The complete alarm state transition history is stored and queryable which allows for subsequent root cause analysis (RCA) or advanced analytics.
**Notification Methods: Create and delete notification methods and associate them with alarms, such as email. Supports the ability to notify users directly via email when an alarm state transitions occur.
** The Monasca API has both Java and Python implementations avaialble.
* Persister (monasca-persister): Consumes metrics and alarm state transitions from the MessageQ and stores them in the Metrics and Alarms database.
** The Persister has both Java and Python implementations.
* Transform and Aggregation Engine (monasca-transform): Transform metric names and values, such as delta or time-based derivative calculations, and creates new metrics that are published to the Message Queue. The Transform Engine is not available yet.
* Anomaly and Prediction Engine: Evaluates prediction and anomalies and generates predicted metrics as well as anomaly likelihood and anomaly scores. The Anomaly and Prediction Engine is currently in a prototype status.
* Threshold Engine (monasca-thresh): Computes thresholds on metrics and publishes alarms to the MessageQ when exceeded. Based on Apache Storm a free and open distributed real-time computation system.
* Notification Engine (monasca-notification): Consumes alarm state transition messages from the MessageQ and sends notifications, such as emails for alarms. The Notification Engine is Python based.
* Analytics Engine (monasca-analytics): Consumes alarm state transisitions and metrics from the MessageQ and does anomaly detection and alarm clustering/correlation.
* Message Queue: A third-party component that primarily receives published metrics from the Monitoring API and alarm state transition messages from the Threshold Engine that are consumed by other components, such as the Persister and Notification Engine. The Message Queue is also used to publish and consume other events in the system. Currently, a Kafka based MessageQ is supported. Kafka is a high performance, distributed, fault-tolerant, and scalable message queue with durability built-in. We will look at other alternatives, such as RabbitMQ and in-fact in our previous implementation RabbitMQ was supported, but due to performance, scale, durability and high-availability limitiations with RabbitMQ we have moved to Kafka.
* Metrics and Alarms Database: A third-party component that primarily stores metrics and the alarm state history. Currently, Vertica and InfluxDB are supported. Support for Cassandra is in progress.
* Config Database: A third-party component that stores a lot of the configuration and other information in the system. Currently, MySQL is supported. Support for PostgreSQL is in progress.
* Monitoring Client (python-monascaclient): A Python command line client and library that communicates and controls the Monitoring API. The Monitoring Client was written using the OpenStack Heat Python client as a framework. The Monitoring Client also has a Python library, "monascaclient" similar to the other OpenStack clients, that can be used to quickly build additional capabilities. The Monitoring Client library is used by the Monitoring UI, Ceilometer publisher, and other components.
* Monitoring UI: A Horizon dashboard for visualizing the overall health and status of an OpenStack cloud.
* Ceilometer publisher: A multi-publisher plugin for Ceilometer, not shown, that converts and publishes samples to the Monitoring API.
Most of the components are described in their respective repositories. However, there aren't any repositories for the third-party components used, so we describe some of the relevant details here.

Further historical context for the architecture can be found at [[Monasca/Architecture Details]]

=== Message Schema ===
[[Monasca/Message Schema]]

=== Message Queue ===
A distributed, performant, scalable, HA message queue for distributing metrics, alarms and events in the monitoring system. Currently, based on Kafka.

==== Messages ====
There are several messages that are published and consumed by various components in Monasca via the MessageQ. See [[Monasca/Message Schema|Message Schema]].

=== Metrics and Alarms Database ===

A high-performance analytics database that can store massive amounts of metrics and alarms in real-time and also support interactive queries. Currently Vertica and InfluxDB are supported.

The SQL schema that is used by Vertica is as follows:

* MonMetrics.Measurements: Stores the actual measurements that are sent.
** id: An integer ID for the measurement.
** definition_dimensions_id: A reference to DefinitionDimensions.
** time_stamp
** value
* MonMetrics.DefinitionDimensions
** id: A sha1 hash of (defintion_id, dimension_set_id)
** definition_id: A reference to the Definitions.id
** dimension_set_id: A reference to the Dimensions.dimension_set_id
* MonMetrics.Definitions
** id: A sha1 hash of the (name, tenant_id, region)
** name: Name of the metric.
** tenant_id: The tenant_id that submitted the metric.
** region: The region the metric was submitted under.
* MonMetric.Dimensions
** dimension_set_id: A sha1 hash of the set of dimenions for a metric.
** name: Name of dimension.
** value: Value of dimension.

=== Config Database ===
The config database store all the configuration information. Currently based on MySQL.

The SQL schema is as follows:

* alarm
** id
** tenant_id
** name
** description
** expression
** state
** actions_enabled
** created_at
** updated_at
** deleted_at
* alarm_action
** alarm_id
** alarm_state
** action_id
* notification_method
** id
** tenant_id
** name
** type
** address
** created_at
** updated_at
* sub_alarm
** id
** alarm_id
** function
** metric_name
** operator
** threshold
** period
** periods
** state
** created_at
** updated_at
* sub_alarm_dimension
** sub_alarm_id
** dimension_name
** value

== Events ==
Support for real-time event stream processing in Monasca is in progress. For more details see the link at, [[Monasca/Events]].

== Logging ==
Support for logging in Monasca is under discussion. For more details see the link at, [[Monasca/Logging]].

== Transform and Aggregation Engine ==
For more details see the link at, [[Monasca/Transform]].

== Analytics ==
Support for anomaly detection and alarm clustering/correlation is in progress. For more details see the link at, [[Monasca/Analytics]].

== Monitoring ==
Enablement and usage for monitoring the status of Monasca is under discussion. For more details see the link at, [[Monasca/Monitoring_Of_Monasca]]

== UI/UX Support ==
Adding more support for common UI/UX queries is under discussion. For more details see the link at, [[Monasca/UI_UX_Support]]

== Keystone Requirements ==
Monasca relies on keystone for running and there are requirements about which keystone configuration must exist.

* The endpoint for the api must be registered in keystone as the 'monasca' service.
* The api must have an admin token to use in verifying the keystone tokens it receives.
* For each project which uses Monasca two users must exist, one will be in the 'monasca-agent' role and be used by the monasca-agent's running on machines. The other should not be in that role and can be used logging into the UI, using the CLI or for direct queries against the API.

== Post Metric Sequence ==

This section describes the sequence of operations involved in posting a metric to the Monasca API.

[[File:monasca-arch-post-metric-diagram.png|Monasca Architecture Post Metric Diagram]]

# A metric is posted to the Monasca API.
# The Monasca API authenticates and validates the request and publishes the metric to the the Message Queue.
# The Persister consumes the metric from the Message Queue and stores in the Metrics Store.
# The Transform Engine consumes the metrics from the Message Queue, performs transform and aggregation operations on metrics, and publishes metrics that it creates back to Message Queue.
# The Threshold Engine consumes metrics from the Message Queue and evaluates alarms. If a state change occurs in an alarm, an "alarm-state-transitioned-event" is published to the Message Queue.
# The Notification Engine consumes "alarm-state-transitioned-events" from the Message Queue, evaluates whether they have a Notification Method associated with it, and sends the appropriate notification, such as email.
# The Persister consumes the "alarm-state-transitioned-event" from the Message Queue and stores it in the Alarm State History Store.

== Alarm Managers ==
This section describes the new features including alarm grouping, inhibition and silencing.

Examples:

* Silence
SilenceRule1 = '{"alarm-silencing-definition-created": {"name": "silence_rule_1", "matchers": {"severity": "LOW"}, "start_time": "2017-02-21 20:00:00", "end_time": "2017-02-21 22:00:00"}}'
Two alarm transitions: AT1 and AT2
AT1_severity = HIGH
AT2_severity = LOW

Output: AT2 get silenced and AT1 send a notification.
SilenceRule1 expires after "end_time"="2017-02-21 22:00:00".

* Inhibit
InhibitionRule1 = '{"alarm-inhibition-definition-created": {"name": "inhibit_rule_1","source":{"severity":"HIGH"},"target":{"severity":"LOW"}, "equals":["tenantId"], "exclusions": {"alarm_name": "vm is dead"}}}'
Three alarm transitions: AT1, AT2 and AT3
AT1_tenantId = "d42bc"
AT2_tenantId = "d42bc"
AT3_tenantId = "d42bc"
AT1_severity = HIGH
AT2_severity = LOW
AT3_severity = LOW
AT1_alarm_name = "cpu high"
AT2_alarm_name = "memory high"
AT3_alarm_name = "vm is dead"
AT1_state = ALARM
AT2_state = ALARM
AT3_state = ALARM

Output: AT1 is the source alarm which will send a notification. AT2 is the target alarm and will get inhibited. AT3 matches the exclusions and will send a notification immediately.

* Grouping Scenario 1

GroupingRule1 = '{"alarm-grouping-definition-created":
{"name": "group_rule_1", "matchers": ["hostname"], "id": "b7163","repeat_interval": "2h", "group_wait": "30s", "exclusions": {"alarm_name": "cpu_percent_high"}, "tenantId": " d42bc", "alarm_actions": ["cd892"], "ok_actions": ["ad892"], "undetermined_actions": ["cf892"]}}'

Three alarm transitions: AT1, AT2 and AT3

AT1_hostname = host1
AT2_hostname = host1
AT3_hostname = host2
AT1_alarm_name = cpu_percent_high
AT2_alarm_name = cpu_system_perc_high
AT3_alarm_name = cpu_percent_high
AT1_state = ALARM
AT2_state = ALARM
AT3_state = ALARM

Output: AT1 and AT3 match exclusions and send notifications immediately. Generate a grouped notification “group_notification_rule_1_host1_alarm[1]” and send out using alarm_actions ["cd892"]. There are no alarm_actions, ok_actions or undermined_actions associated with the AT1, AT2, AT3 alarm definitions.

* Grouping Scenario 2

GroupingRule2 = '{"alarm-grouping-definition-created":
{"name": "group_rule_2", "matchers": ["hostname"], "id": "b7163","repeat_interval": "2h", "group_wait": "30s", "tenantId": " d42bc", "alarm_actions": ["cd892"], "ok_actions": ["ad892"], "undetermined_actions": ["cf892"]}}'

Three alarm transitions: AT1 with "alarm_actions": ["123ab"], AT2 with "alarm_actions": ["cd839"] and AT3

AT1_hostname = host1
AT2_hostname = host1
AT3_hostname = host2
AT1_state = ALARM
AT2_state = ALARM
AT3_state = ALARM

Output: Generate two grouped notifications “group_notification_rule_2_host1_alarm[2]” and “group_notification_rule_2_host2_alarm[1]”. Both using alarm_actions ["cd892"]. Also since AT1 and AT2 has their own alarm actions associated with them, there will be two more notifications sent out.

* Silenced and Grouped

SilenceRule1 = '{"alarm-silencing-definition-created": {"name": "silence_rule_1", "matchers": {"severity": "LOW"}, "start_time": "1487269470498", "end_time": "1587269470498"}}'

GroupingRule2 = '{"alarm-grouping-definition-created":
{"name": "group_rule_2", "matchers": ["hostname"], "id": "b7163","repeat_interval": "2h", "group_wait": "30s", "tenantId": " d42bc", "alarm_actions": ["cd892"], "ok_actions": ["ad892"], "undetermined_actions": ["cf892"]}}'

Four alarm transitions: AT1, AT2, AT3 and AT4

Silencing rule:
AT1_severity = HIGH
AT2_severity = LOW
AT3_severity = HIGH
AT4_severity = HIGH

Grouping rule:
AT1_hostname = host1
AT2_hostname = host1
AT3_hostname = host2
AT4_hostname = host1
AT1_state = ALARM
AT2_state = ALARM
AT3_state = OK
AT4_state = ALARM

Output: Generate two grouped notifications “group_notification_rule_2_host1_alarm[2]” using alarm action "cd892" and “group_notification_rule_2_host2_ok[1]” using ok action "ad892". AT2 is silenced so it isn't include in group_notification_rule_2_host1 count. There are no alarm_actions, ok_actions or undermined_actions associated with the AT1, AT2, AT3, AT4 alarm definitions.

* Silenced and inhibited (source alarm get silenced)

SilenceRule2 = '{"alarm-silencing-definition-created": {"name": "silence_rule_2", "matchers": {"severity": "HIGH, "hostname": "host1"}, "start_time": "2017-02-21 15:00:00", "end_time": "2017-02-21 21:00:00"}}'

InhibitionRule1 = '{"alarm-inhibition-definition-created": {"name": "inhibit_rule_1","source":{"severity":"HIGH"},"target":{"severity":"LOW"}, "equals":["tenantId"]}}'

Alarm transitions: AT1, AT2

Inhibition rule:
AT1_state = ALARM
AT2_state = ALARM
AT1_tenantId = "d42bc"
AT2_tenantId = "d42bc"
AT1_severity = HIGH
AT2_severity = LOW

Silencing rule:
AT1_hostname = host1
AT2_hostname = host2
AT1_severity = HIGH
AT2_severity = LOW

Output: no notification sent out. For inhibition, AT1 is the source alarm, AT2 is the target alarm. But at the same time, AT1 get silenced because it matches the silence rule.

* Inhibited and grouped

InhibitionRule1 = '{"alarm-inhibition-definition-created": {"name": "inhibit_rule_1","source":{"severity":"HIGH"},"target":{"severity":"LOW"}, "equals":["tenantId"]}}'

GroupingRule1 = '{"alarm-grouping-definition-created": {"name": "group_rule_1", "matchers": ["hostname"], "id": "b7163", "repeat_interval": "2h", "group_wait": "30s", "tenantId": " d42bc", "alarm_actions": ["cd892"], "ok_actions": ["ad892"], "undetermined_actions": ["cf892"]}}'

Alarm transitions: AT1, AT2, AT3

Inhibition rule:
AT1_tenantId = "d42bc"
AT2_tenantId = "d42bc"
AT3_tenantId = "d42bc"
AT1_severity = HIGH
AT2_severity = LOW
AT3_severity = HIGH
AT1_state = OK
AT2_state = ALARM
AT3_state = ALARM

Grouping rule:
AT1_hostname = host1
AT2_hostname = host2
AT3_hostname = host1
AT1_state = OK
AT2_state = ALARM
AT3_state = ALARM

Output: AT2 gets inhibited because its severity is low. AT3 is the source alarm. Since AT1 is in OK state, it is not a source alarm. For grouping, AT1 and AT3 has the same host name but different state. So there will be two grouped notifications sent out: “group_notification_rule_1_host1_ok[1]” and “group_notification_rule_1_host1_alarm[1]”.

* Silenced, inhibited and grouped

SilenceRule2 = '{"alarm-silencing-definition-created": {"name": "silence_rule_2", "matchers": {"severity": "HIGH, "hostname": "host1"}, "start_time": "2017-02-21 15:00:00", "end_time": "2017-02-21 21:00:00"}}'

InhibitionRule1 = '{"alarm-inhibition-definition-created": {"name": "inhibit_rule_1","source":{"severity":"HIGH"},"target":{"severity":"LOW"}, "equals":["tenantId"]}}'

GroupingRule1 = '{"alarm-grouping-definition-created": {"name": "group_rule_1", "matchers": ["hostname"], "id": "b7163", "repeat_interval": "2h", "group_wait": "30s", "tenantId": " d42bc", "alarm_actions": ["cd892"], "ok_actions": ["ad892"], "undetermined_actions": ["cf892"]}}'

Alarm transitions: AT1, AT2, AT3 and AT5

Silence rule:
AT1_severity = HIGH AT1_hostname = host1 (silenced)
AT2_severity = LOW AT2_hostname = host2
AT3_severity = HIGH AT3_hostname = host1(silenced)
AT5_severity = HIGH AT5_hostname = host3

Inhibition rule:
AT1_tenantId = "d42bc"
AT2_tenantId = "d42bc"
AT3_tenantId = "d42bc"
AT5_tenantId = "d42bc"
AT1_state = ALARM
AT2_state = ALARM
AT3_state = OK
AT5_state = UNDETERMINED
AT1_severity = HIGH (source)
AT2_severity = LOW (target)
AT3_severity = HIGH
AT5_severity = HIGH

Grouping rule:
AT1_hostname = host1
AT2_hostname = host2
AT3_hostname = host1
AT5_hostname = host3

Output: AT1 is in "group_notification_rule_1_host1_alarm" group and silenced. AT3 is in "group_notification_rule_1_host1_ok" group and silenced. AT2 is in "group_notification_rule_1_host2_alarm" group and inhibited. AT5 is in "group_notification_rule_1_host3_undetermined" group and will send notification “group_notification_rule_1_host3_undetermined[1]” using undetermined action "cf892".

= Development Environment =

* Monasca DevStack Plugin:
** DevStack is the primary developmement environment for OpenStack.
***See http://docs.openstack.org/developer/devstack/
** The Monasca DevStack plugin installs the Monasca Service, Agent, Horizon Monitoring Panel, and Grafana
** README at, https://github.com/openstack/monasca-api/tree/master/devstack
** Best way to get started is to use Vagrant with the Vagrantfile at, https://github.com/openstack/monasca-api/blob/master/devstack/Vagrantfile.

* Monasca on LaunchPad. https://launchpad.net/monasca

* Monasca projects source code. https://git.openstack.org/cgit/?q=monasca

= Coding Standards =

* Python: All Python code conforms to the OpenStack standards at, http://docs.openstack.org/developer/hacking/.
** Note, all components in Monasca, except for the Threshold Engine, have been ported to Python.

* Java: Several of the Monasca components are available as Java. OpenStack does not have any Java coding standards. We've adopted the Google Java Style at, https://google.github.io/styleguide/javaguide.html.
** The standard says either 80 or 100 length lines. We've adopted 100.

= Technologies =

Monasca uses a number of third-party technologies:

* Apache Kafka (http://kafka.apache.org): Apache Kafka is publish-subscribe messaging rethought as a distributed commit log. Kafka is a highly performant, distributed, fault-tolerant, and scalable message queue with durability built-in.

* Apache Storm (http://storm.incubator.apache.org/): Apache Storm is a free and open source distributed realtime computation system. Storm makes it easy to reliably process unbounded streams of data, doing for realtime processing what Hadoop did for batch processing.

* ZooKeeper (http://zookeeper.apache.org/): Used by Kafka and Storm.

* MySQL: MySQL is supported as a Config Database.

* POSTgres: Support for POSTgres, via Hibernate and SQLAlchemy, for the Config Database.

* Vagrant (http://www.vagrantup.com/): Vagrant provides easy to configure, reproducible, and portable work environments built on top of industry-standard technology and controlled by a single consistent workflow to help maximize the productivity and flexibility of you and your team.

* Dropwizard (https://dropwizard.github.io/dropwizard/): Dropwizard pulls together stable, mature libraries from the Java ecosystem into a simple, light-weight package that lets you focus on getting things done. Dropwizard has out-of-the-box support for sophisticated configuration, application metrics, logging, operational tools, and much more, allowing you and your team to ship a production-quality web service in the shortest time possible.

* InfluxDB (http://influxdb.com/): An open-source distributed time series database with no external dependencies. InfluxDB is supported for the Metrics Database.

* Vertica (http://www.vertica.com): A commercial Enterprise class SQL analytics database that is highly scalable. It offers built-in automatic high-availability and excels at in-database analytics and compressing and storing massive amounts of data. A free community version of Vertica is available that can store up to 1 TB of data with no time-limit is available at, https://my.vertica.com/community/. Vertica is supported for the Metrics Database.

* Cassandra: Support for Cassandra for the Metrics Database is in progress.

= License =

Copyright (c) 2014, 2015 Hewlett-Packard Development Company, L.P.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied.
See the License for the specific language governing permissions and
limitations under the License.

[[File:yklogo.png|200px|thumb|left|Monasca uses YourKit Profiler for Java development]]

 
 
 
 

[https://www.yourkit.com/ Visit YourKit website for more information]

Monasca

2019-05-08T16:49:08Z

Joseph Davis: /* Logo */ remove logo section (taking too much space)

== Overview ==

Monasca is a open-source multi-tenant, highly scalable, performant, fault-tolerant monitoring-as-a-service solution that integrates with OpenStack. It uses a REST API for high-speed metrics processing and querying and has a streaming alarm engine and notification engine.

[[File:OpenStack_Project_Monasca_vertical.png|thumb|right]]

==== Contribution ====
* [https://storyboard.openstack.org/#!/project_group/59 Project Group on StoryBoard]
* [https://storyboard.openstack.org/#!/board/111 Important activities in Stein release]
* [https://storyboard.openstack.org/#!/board/141 Important activities in Train release]
* [https://docs.openstack.org/monasca-api/latest/contributor/index.html Contribution Guidelines]

==== Communication and Meetings ====
* IRC: #openstack-monasca on freenode.net
* Weekly Meetings:
** http://eavesdrop.openstack.org/#Monasca_Team_Meeting
** [[Meetings/Monasca]]
* Use [Monasca] tag on [http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-discuss openstack-discuss mailing list].

===== PTG meetings =====
* [[MonascaSteinPTG|Stein PTG Summary (Denver)]]
* (coming soon - Train summary)

==== Documentation ====
: Monasca API Specification: https://github.com/openstack/monasca-api/blob/master/docs/monasca-api-spec.md
: Agent Documentation: https://github.com/openstack/monasca-agent

==== Presentations ====
There have been many interesting and excellent presentations on Monasca, many given at the OpenStack or OpenInfra Summits.
: [[Monasca/Presentations]] (updated with Denver 2019 links)

==== Repositories ====

: '''Core'''
:: Core: https://git.openstack.org/cgit/?q=monasca

: '''Deployment'''
The following repositories are available for deploying Monasca:
* Docker: https://github.com/monasca/monasca-docker
* Kubernetes: https://github.com/monasca/monasca-helm
* Ansible: https://github.com/search?utf8=%E2%9C%93&q=ansible-monasca
* Puppet: https://git.openstack.org/openstack/puppet-monasca

==== Bugs ====

===== All open bugs =====
: https://storyboard.openstack.org/#!/worklist/213

===== In Stein triaged bugs =====
: https://storyboard.openstack.org/#!/worklist/467

===== Bug-fixing progress =====
: https://storyboard.openstack.org/#!/board/114

== Requirements ==
[[Monasca/Requirements]].

== Features ==

This section describes the overall features.

* A highly performant, scalable, reliable and fault-tolerant Monitoring as a Service (MONaaS) solution that scales to service provider metrics levels of metrics throughput. Performance, scalability and high-availability have been designed in from the start. Can process 100s of thousands of metrics/sec as well as offer data retention periods of greater than a year with no data loss while still processing interactive queries.

* Rest API for storing and querying metrics and historical information. Most monitoring solution use special transports and protocols, such as CollectD or NSCA (Nagios). In our solution, http is the only protocol used. This simplifies the overall design and also allows for a much richer way of describing the data via dimensions.

* Multi-tenant and authenticated. Metrics are submitted and authenticated using Keystone and stored associated with a tenant ID.

* Metrics defined using a set of (key, value) pairs called dimensions.

* Real-time thresholding and alarming on metrics.

* Compound alarms described using a simple expressive grammar composed of alarm sub-expressions and logical operators.

* Monitoring agent that supports a number of built-in system and service checks and also supports Nagios checks and statsd.

* Open-source monitoring solution built on open-source technologies.

== Comparisons to alternatives ==
[[Monasca/Comparison]]

== Architecture ==
[[File:monasca-arch-component-diagram.png|Monasca Architecture Component Diagram]]
* Monitoring Agent (monasca-agent): A modern Python based monitoring agent that consists of several sub-components and supports system metrics, such as cpu utilization and available memory, Nagios plugins, statsd and many built-in checks for services such as MySQL, RabbitMQ, and many others.
* Monitoring API (monasca-api): A well-defined and documented RESTful API for monitoring that is primarily focused on the following concepts and areas:
** Metrics: Store and query massive amounts of metrics in real-time.
** Statistics: Query statistics for metrics.
** Alarm Definitions: Create, update, query and delete alarm definitions.
** Alarms: Query and delete the alarm history.
***Simple expressive grammar for creating compound alarms composed of alarm subexpressions and logical operators.
***Alarm severities can be associated with alarms.
***The complete alarm state transition history is stored and queryable which allows for subsequent root cause analysis (RCA) or advanced analytics.
**Notification Methods: Create and delete notification methods and associate them with alarms, such as email. Supports the ability to notify users directly via email when an alarm state transitions occur.
** The Monasca API has both Java and Python implementations avaialble.
* Persister (monasca-persister): Consumes metrics and alarm state transitions from the MessageQ and stores them in the Metrics and Alarms database.
** The Persister has both Java and Python implementations.
* Transform and Aggregation Engine (monasca-transform): Transform metric names and values, such as delta or time-based derivative calculations, and creates new metrics that are published to the Message Queue. The Transform Engine is not available yet.
* Anomaly and Prediction Engine: Evaluates prediction and anomalies and generates predicted metrics as well as anomaly likelihood and anomaly scores. The Anomaly and Prediction Engine is currently in a prototype status.
* Threshold Engine (monasca-thresh): Computes thresholds on metrics and publishes alarms to the MessageQ when exceeded. Based on Apache Storm a free and open distributed real-time computation system.
* Notification Engine (monasca-notification): Consumes alarm state transition messages from the MessageQ and sends notifications, such as emails for alarms. The Notification Engine is Python based.
* Analytics Engine (monasca-analytics): Consumes alarm state transisitions and metrics from the MessageQ and does anomaly detection and alarm clustering/correlation.
* Message Queue: A third-party component that primarily receives published metrics from the Monitoring API and alarm state transition messages from the Threshold Engine that are consumed by other components, such as the Persister and Notification Engine. The Message Queue is also used to publish and consume other events in the system. Currently, a Kafka based MessageQ is supported. Kafka is a high performance, distributed, fault-tolerant, and scalable message queue with durability built-in. We will look at other alternatives, such as RabbitMQ and in-fact in our previous implementation RabbitMQ was supported, but due to performance, scale, durability and high-availability limitiations with RabbitMQ we have moved to Kafka.
* Metrics and Alarms Database: A third-party component that primarily stores metrics and the alarm state history. Currently, Vertica and InfluxDB are supported. Support for Cassandra is in progress.
* Config Database: A third-party component that stores a lot of the configuration and other information in the system. Currently, MySQL is supported. Support for PostgreSQL is in progress.
* Monitoring Client (python-monascaclient): A Python command line client and library that communicates and controls the Monitoring API. The Monitoring Client was written using the OpenStack Heat Python client as a framework. The Monitoring Client also has a Python library, "monascaclient" similar to the other OpenStack clients, that can be used to quickly build additional capabilities. The Monitoring Client library is used by the Monitoring UI, Ceilometer publisher, and other components.
* Monitoring UI: A Horizon dashboard for visualizing the overall health and status of an OpenStack cloud.
* Ceilometer publisher: A multi-publisher plugin for Ceilometer, not shown, that converts and publishes samples to the Monitoring API.
Most of the components are described in their respective repositories. However, there aren't any repositories for the third-party components used, so we describe some of the relevant details here.

Further historical context for the architecture can be found at [[Monasca/Architecture Details]]

=== Message Schema ===
[[Monasca/Message Schema]]

=== Message Queue ===
A distributed, performant, scalable, HA message queue for distributing metrics, alarms and events in the monitoring system. Currently, based on Kafka.

==== Messages ====
There are several messages that are published and consumed by various components in Monasca via the MessageQ. See [[Monasca/Message Schema|Message Schema]].

=== Metrics and Alarms Database ===

A high-performance analytics database that can store massive amounts of metrics and alarms in real-time and also support interactive queries. Currently Vertica and InfluxDB are supported.

The SQL schema that is used by Vertica is as follows:

* MonMetrics.Measurements: Stores the actual measurements that are sent.
** id: An integer ID for the measurement.
** definition_dimensions_id: A reference to DefinitionDimensions.
** time_stamp
** value
* MonMetrics.DefinitionDimensions
** id: A sha1 hash of (defintion_id, dimension_set_id)
** definition_id: A reference to the Definitions.id
** dimension_set_id: A reference to the Dimensions.dimension_set_id
* MonMetrics.Definitions
** id: A sha1 hash of the (name, tenant_id, region)
** name: Name of the metric.
** tenant_id: The tenant_id that submitted the metric.
** region: The region the metric was submitted under.
* MonMetric.Dimensions
** dimension_set_id: A sha1 hash of the set of dimenions for a metric.
** name: Name of dimension.
** value: Value of dimension.

=== Config Database ===
The config database store all the configuration information. Currently based on MySQL.

The SQL schema is as follows:

* alarm
** id
** tenant_id
** name
** description
** expression
** state
** actions_enabled
** created_at
** updated_at
** deleted_at
* alarm_action
** alarm_id
** alarm_state
** action_id
* notification_method
** id
** tenant_id
** name
** type
** address
** created_at
** updated_at
* sub_alarm
** id
** alarm_id
** function
** metric_name
** operator
** threshold
** period
** periods
** state
** created_at
** updated_at
* sub_alarm_dimension
** sub_alarm_id
** dimension_name
** value

== Events ==
Support for real-time event stream processing in Monasca is in progress. For more details see the link at, [[Monasca/Events]].

== Logging ==
Support for logging in Monasca is under discussion. For more details see the link at, [[Monasca/Logging]].

== Transform and Aggregation Engine ==
For more details see the link at, [[Monasca/Transform]].

== Analytics ==
Support for anomaly detection and alarm clustering/correlation is in progress. For more details see the link at, [[Monasca/Analytics]].

== Monitoring ==
Enablement and usage for monitoring the status of Monasca is under discussion. For more details see the link at, [[Monasca/Monitoring_Of_Monasca]]

== UI/UX Support ==
Adding more support for common UI/UX queries is under discussion. For more details see the link at, [[Monasca/UI_UX_Support]]

== Keystone Requirements ==
Monasca relies on keystone for running and there are requirements about which keystone configuration must exist.

* The endpoint for the api must be registered in keystone as the 'monasca' service.
* The api must have an admin token to use in verifying the keystone tokens it receives.
* For each project which uses Monasca two users must exist, one will be in the 'monasca-agent' role and be used by the monasca-agent's running on machines. The other should not be in that role and can be used logging into the UI, using the CLI or for direct queries against the API.

== Post Metric Sequence ==

This section describes the sequence of operations involved in posting a metric to the Monasca API.

[[File:monasca-arch-post-metric-diagram.png|Monasca Architecture Post Metric Diagram]]

# A metric is posted to the Monasca API.
# The Monasca API authenticates and validates the request and publishes the metric to the the Message Queue.
# The Persister consumes the metric from the Message Queue and stores in the Metrics Store.
# The Transform Engine consumes the metrics from the Message Queue, performs transform and aggregation operations on metrics, and publishes metrics that it creates back to Message Queue.
# The Threshold Engine consumes metrics from the Message Queue and evaluates alarms. If a state change occurs in an alarm, an "alarm-state-transitioned-event" is published to the Message Queue.
# The Notification Engine consumes "alarm-state-transitioned-events" from the Message Queue, evaluates whether they have a Notification Method associated with it, and sends the appropriate notification, such as email.
# The Persister consumes the "alarm-state-transitioned-event" from the Message Queue and stores it in the Alarm State History Store.

== Alarm Managers ==
This section describes the new features including alarm grouping, inhibition and silencing.

Examples:

* Silence
SilenceRule1 = '{"alarm-silencing-definition-created": {"name": "silence_rule_1", "matchers": {"severity": "LOW"}, "start_time": "2017-02-21 20:00:00", "end_time": "2017-02-21 22:00:00"}}'
Two alarm transitions: AT1 and AT2
AT1_severity = HIGH
AT2_severity = LOW

Output: AT2 get silenced and AT1 send a notification.
SilenceRule1 expires after "end_time"="2017-02-21 22:00:00".

* Inhibit
InhibitionRule1 = '{"alarm-inhibition-definition-created": {"name": "inhibit_rule_1","source":{"severity":"HIGH"},"target":{"severity":"LOW"}, "equals":["tenantId"], "exclusions": {"alarm_name": "vm is dead"}}}'
Three alarm transitions: AT1, AT2 and AT3
AT1_tenantId = "d42bc"
AT2_tenantId = "d42bc"
AT3_tenantId = "d42bc"
AT1_severity = HIGH
AT2_severity = LOW
AT3_severity = LOW
AT1_alarm_name = "cpu high"
AT2_alarm_name = "memory high"
AT3_alarm_name = "vm is dead"
AT1_state = ALARM
AT2_state = ALARM
AT3_state = ALARM

Output: AT1 is the source alarm which will send a notification. AT2 is the target alarm and will get inhibited. AT3 matches the exclusions and will send a notification immediately.

* Grouping Scenario 1

GroupingRule1 = '{"alarm-grouping-definition-created":
{"name": "group_rule_1", "matchers": ["hostname"], "id": "b7163","repeat_interval": "2h", "group_wait": "30s", "exclusions": {"alarm_name": "cpu_percent_high"}, "tenantId": " d42bc", "alarm_actions": ["cd892"], "ok_actions": ["ad892"], "undetermined_actions": ["cf892"]}}'

Three alarm transitions: AT1, AT2 and AT3

AT1_hostname = host1
AT2_hostname = host1
AT3_hostname = host2
AT1_alarm_name = cpu_percent_high
AT2_alarm_name = cpu_system_perc_high
AT3_alarm_name = cpu_percent_high
AT1_state = ALARM
AT2_state = ALARM
AT3_state = ALARM

Output: AT1 and AT3 match exclusions and send notifications immediately. Generate a grouped notification “group_notification_rule_1_host1_alarm[1]” and send out using alarm_actions ["cd892"]. There are no alarm_actions, ok_actions or undermined_actions associated with the AT1, AT2, AT3 alarm definitions.

* Grouping Scenario 2

GroupingRule2 = '{"alarm-grouping-definition-created":
{"name": "group_rule_2", "matchers": ["hostname"], "id": "b7163","repeat_interval": "2h", "group_wait": "30s", "tenantId": " d42bc", "alarm_actions": ["cd892"], "ok_actions": ["ad892"], "undetermined_actions": ["cf892"]}}'

Three alarm transitions: AT1 with "alarm_actions": ["123ab"], AT2 with "alarm_actions": ["cd839"] and AT3

AT1_hostname = host1
AT2_hostname = host1
AT3_hostname = host2
AT1_state = ALARM
AT2_state = ALARM
AT3_state = ALARM

Output: Generate two grouped notifications “group_notification_rule_2_host1_alarm[2]” and “group_notification_rule_2_host2_alarm[1]”. Both using alarm_actions ["cd892"]. Also since AT1 and AT2 has their own alarm actions associated with them, there will be two more notifications sent out.

* Silenced and Grouped

SilenceRule1 = '{"alarm-silencing-definition-created": {"name": "silence_rule_1", "matchers": {"severity": "LOW"}, "start_time": "1487269470498", "end_time": "1587269470498"}}'

GroupingRule2 = '{"alarm-grouping-definition-created":
{"name": "group_rule_2", "matchers": ["hostname"], "id": "b7163","repeat_interval": "2h", "group_wait": "30s", "tenantId": " d42bc", "alarm_actions": ["cd892"], "ok_actions": ["ad892"], "undetermined_actions": ["cf892"]}}'

Four alarm transitions: AT1, AT2, AT3 and AT4

Silencing rule:
AT1_severity = HIGH
AT2_severity = LOW
AT3_severity = HIGH
AT4_severity = HIGH

Grouping rule:
AT1_hostname = host1
AT2_hostname = host1
AT3_hostname = host2
AT4_hostname = host1
AT1_state = ALARM
AT2_state = ALARM
AT3_state = OK
AT4_state = ALARM

Output: Generate two grouped notifications “group_notification_rule_2_host1_alarm[2]” using alarm action "cd892" and “group_notification_rule_2_host2_ok[1]” using ok action "ad892". AT2 is silenced so it isn't include in group_notification_rule_2_host1 count. There are no alarm_actions, ok_actions or undermined_actions associated with the AT1, AT2, AT3, AT4 alarm definitions.

* Silenced and inhibited (source alarm get silenced)

SilenceRule2 = '{"alarm-silencing-definition-created": {"name": "silence_rule_2", "matchers": {"severity": "HIGH, "hostname": "host1"}, "start_time": "2017-02-21 15:00:00", "end_time": "2017-02-21 21:00:00"}}'

InhibitionRule1 = '{"alarm-inhibition-definition-created": {"name": "inhibit_rule_1","source":{"severity":"HIGH"},"target":{"severity":"LOW"}, "equals":["tenantId"]}}'

Alarm transitions: AT1, AT2

Inhibition rule:
AT1_state = ALARM
AT2_state = ALARM
AT1_tenantId = "d42bc"
AT2_tenantId = "d42bc"
AT1_severity = HIGH
AT2_severity = LOW

Silencing rule:
AT1_hostname = host1
AT2_hostname = host2
AT1_severity = HIGH
AT2_severity = LOW

Output: no notification sent out. For inhibition, AT1 is the source alarm, AT2 is the target alarm. But at the same time, AT1 get silenced because it matches the silence rule.

* Inhibited and grouped

InhibitionRule1 = '{"alarm-inhibition-definition-created": {"name": "inhibit_rule_1","source":{"severity":"HIGH"},"target":{"severity":"LOW"}, "equals":["tenantId"]}}'

GroupingRule1 = '{"alarm-grouping-definition-created": {"name": "group_rule_1", "matchers": ["hostname"], "id": "b7163", "repeat_interval": "2h", "group_wait": "30s", "tenantId": " d42bc", "alarm_actions": ["cd892"], "ok_actions": ["ad892"], "undetermined_actions": ["cf892"]}}'

Alarm transitions: AT1, AT2, AT3

Inhibition rule:
AT1_tenantId = "d42bc"
AT2_tenantId = "d42bc"
AT3_tenantId = "d42bc"
AT1_severity = HIGH
AT2_severity = LOW
AT3_severity = HIGH
AT1_state = OK
AT2_state = ALARM
AT3_state = ALARM

Grouping rule:
AT1_hostname = host1
AT2_hostname = host2
AT3_hostname = host1
AT1_state = OK
AT2_state = ALARM
AT3_state = ALARM

Output: AT2 gets inhibited because its severity is low. AT3 is the source alarm. Since AT1 is in OK state, it is not a source alarm. For grouping, AT1 and AT3 has the same host name but different state. So there will be two grouped notifications sent out: “group_notification_rule_1_host1_ok[1]” and “group_notification_rule_1_host1_alarm[1]”.

* Silenced, inhibited and grouped

SilenceRule2 = '{"alarm-silencing-definition-created": {"name": "silence_rule_2", "matchers": {"severity": "HIGH, "hostname": "host1"}, "start_time": "2017-02-21 15:00:00", "end_time": "2017-02-21 21:00:00"}}'

InhibitionRule1 = '{"alarm-inhibition-definition-created": {"name": "inhibit_rule_1","source":{"severity":"HIGH"},"target":{"severity":"LOW"}, "equals":["tenantId"]}}'

GroupingRule1 = '{"alarm-grouping-definition-created": {"name": "group_rule_1", "matchers": ["hostname"], "id": "b7163", "repeat_interval": "2h", "group_wait": "30s", "tenantId": " d42bc", "alarm_actions": ["cd892"], "ok_actions": ["ad892"], "undetermined_actions": ["cf892"]}}'

Alarm transitions: AT1, AT2, AT3 and AT5

Silence rule:
AT1_severity = HIGH AT1_hostname = host1 (silenced)
AT2_severity = LOW AT2_hostname = host2
AT3_severity = HIGH AT3_hostname = host1(silenced)
AT5_severity = HIGH AT5_hostname = host3

Inhibition rule:
AT1_tenantId = "d42bc"
AT2_tenantId = "d42bc"
AT3_tenantId = "d42bc"
AT5_tenantId = "d42bc"
AT1_state = ALARM
AT2_state = ALARM
AT3_state = OK
AT5_state = UNDETERMINED
AT1_severity = HIGH (source)
AT2_severity = LOW (target)
AT3_severity = HIGH
AT5_severity = HIGH

Grouping rule:
AT1_hostname = host1
AT2_hostname = host2
AT3_hostname = host1
AT5_hostname = host3

Output: AT1 is in "group_notification_rule_1_host1_alarm" group and silenced. AT3 is in "group_notification_rule_1_host1_ok" group and silenced. AT2 is in "group_notification_rule_1_host2_alarm" group and inhibited. AT5 is in "group_notification_rule_1_host3_undetermined" group and will send notification “group_notification_rule_1_host3_undetermined[1]” using undetermined action "cf892".

= Development Environment =

* Monasca DevStack Plugin:
** DevStack is the primary developmement environment for OpenStack.
***See http://docs.openstack.org/developer/devstack/
** The Monasca DevStack plugin installs the Monasca Service, Agent, Horizon Monitoring Panel, and Grafana
** README at, https://github.com/openstack/monasca-api/tree/master/devstack
** Best way to get started is to use Vagrant with the Vagrantfile at, https://github.com/openstack/monasca-api/blob/master/devstack/Vagrantfile.

* Monasca on LaunchPad. https://launchpad.net/monasca

* Monasca projects source code. https://git.openstack.org/cgit/?q=monasca

= Coding Standards =

* Python: All Python code conforms to the OpenStack standards at, http://docs.openstack.org/developer/hacking/.
** Note, all components in Monasca, except for the Threshold Engine, have been ported to Python.

* Java: Several of the Monasca components are available as Java. OpenStack does not have any Java coding standards. We've adopted the Google Java Style at, https://google.github.io/styleguide/javaguide.html.
** The standard says either 80 or 100 length lines. We've adopted 100.

= Technologies =

Monasca uses a number of third-party technologies:

* Apache Kafka (http://kafka.apache.org): Apache Kafka is publish-subscribe messaging rethought as a distributed commit log. Kafka is a highly performant, distributed, fault-tolerant, and scalable message queue with durability built-in.

* Apache Storm (http://storm.incubator.apache.org/): Apache Storm is a free and open source distributed realtime computation system. Storm makes it easy to reliably process unbounded streams of data, doing for realtime processing what Hadoop did for batch processing.

* ZooKeeper (http://zookeeper.apache.org/): Used by Kafka and Storm.

* MySQL: MySQL is supported as a Config Database.

* POSTgres: Support for POSTgres, via Hibernate and SQLAlchemy, for the Config Database.

* Vagrant (http://www.vagrantup.com/): Vagrant provides easy to configure, reproducible, and portable work environments built on top of industry-standard technology and controlled by a single consistent workflow to help maximize the productivity and flexibility of you and your team.

* Dropwizard (https://dropwizard.github.io/dropwizard/): Dropwizard pulls together stable, mature libraries from the Java ecosystem into a simple, light-weight package that lets you focus on getting things done. Dropwizard has out-of-the-box support for sophisticated configuration, application metrics, logging, operational tools, and much more, allowing you and your team to ship a production-quality web service in the shortest time possible.

* InfluxDB (http://influxdb.com/): An open-source distributed time series database with no external dependencies. InfluxDB is supported for the Metrics Database.

* Vertica (http://www.vertica.com): A commercial Enterprise class SQL analytics database that is highly scalable. It offers built-in automatic high-availability and excels at in-database analytics and compressing and storing massive amounts of data. A free community version of Vertica is available that can store up to 1 TB of data with no time-limit is available at, https://my.vertica.com/community/. Vertica is supported for the Metrics Database.

* Cassandra: Support for Cassandra for the Metrics Database is in progress.

= License =

Copyright (c) 2014, 2015 Hewlett-Packard Development Company, L.P.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied.
See the License for the specific language governing permissions and
limitations under the License.

[[File:yklogo.png|200px|thumb|left|Monasca uses YourKit Profiler for Java development]]

 
 
 
 

[https://www.yourkit.com/ Visit YourKit website for more information]

Monasca

2019-05-08T16:46:28Z

Joseph Davis: /* Contribution */

== Overview ==

Monasca is a open-source multi-tenant, highly scalable, performant, fault-tolerant monitoring-as-a-service solution that integrates with OpenStack. It uses a REST API for high-speed metrics processing and querying and has a streaming alarm engine and notification engine.

[[File:OpenStack_Project_Monasca_vertical.png|thumb|right]]

==== Contribution ====
* [https://storyboard.openstack.org/#!/project_group/59 Project Group on StoryBoard]
* [https://storyboard.openstack.org/#!/board/111 Important activities in Stein release]
* [https://storyboard.openstack.org/#!/board/141 Important activities in Train release]
* [https://docs.openstack.org/monasca-api/latest/contributor/index.html Contribution Guidelines]

==== Communication and Meetings ====
* IRC: #openstack-monasca on freenode.net
* Weekly Meetings:
** http://eavesdrop.openstack.org/#Monasca_Team_Meeting
** [[Meetings/Monasca]]
* Use [Monasca] tag on [http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-discuss openstack-discuss mailing list].

===== PTG meetings =====
* [[MonascaSteinPTG|Stein PTG Summary (Denver)]]
* (coming soon - Train summary)

==== Documentation ====
: Monasca API Specification: https://github.com/openstack/monasca-api/blob/master/docs/monasca-api-spec.md
: Agent Documentation: https://github.com/openstack/monasca-agent

==== Logo ====
: https://www.openstack.org/themes/openstack/images/project-mascots/Monasca/OpenStack_Project_Monasca_horizontal_1color.eps
: https://www.openstack.org/themes/openstack/images/project-mascots/Monasca/OpenStack_Project_Monasca_horizontal.eps
: https://www.openstack.org/themes/openstack/images/project-mascots/Monasca/OpenStack_Project_Monasca_horizontal.jpg
: https://www.openstack.org/themes/openstack/images/project-mascots/Monasca/OpenStack_Project_Monasca_horizontal.png
: https://www.openstack.org/themes/openstack/images/project-mascots/Monasca/OpenStack_Project_Monasca_mascot.jpg
: https://www.openstack.org/themes/openstack/images/project-mascots/Monasca/OpenStack_Project_Monasca_mascot.png
: https://www.openstack.org/themes/openstack/images/project-mascots/Monasca/OpenStack_Project_Monasca_vertical_1color.eps
: https://www.openstack.org/themes/openstack/images/project-mascots/Monasca/OpenStack_Project_Monasca_vertical.eps
: https://www.openstack.org/themes/openstack/images/project-mascots/Monasca/OpenStack_Project_Monasca_vertical.jpg
: https://www.openstack.org/themes/openstack/images/project-mascots/Monasca/OpenStack_Project_Monasca_vertical.png

==== Presentations ====
There have been many interesting and excellent presentations on Monasca, many given at the OpenStack or OpenInfra Summits.
: [[Monasca/Presentations]] (updated with Denver 2019 links)

==== Repositories ====

: '''Core'''
:: Core: https://git.openstack.org/cgit/?q=monasca

: '''Deployment'''
The following repositories are available for deploying Monasca:
* Docker: https://github.com/monasca/monasca-docker
* Kubernetes: https://github.com/monasca/monasca-helm
* Ansible: https://github.com/search?utf8=%E2%9C%93&q=ansible-monasca
* Puppet: https://git.openstack.org/openstack/puppet-monasca

==== Bugs ====

===== All open bugs =====
: https://storyboard.openstack.org/#!/worklist/213

===== In Stein triaged bugs =====
: https://storyboard.openstack.org/#!/worklist/467

===== Bug-fixing progress =====
: https://storyboard.openstack.org/#!/board/114

== Requirements ==
[[Monasca/Requirements]].

== Features ==

This section describes the overall features.

* A highly performant, scalable, reliable and fault-tolerant Monitoring as a Service (MONaaS) solution that scales to service provider metrics levels of metrics throughput. Performance, scalability and high-availability have been designed in from the start. Can process 100s of thousands of metrics/sec as well as offer data retention periods of greater than a year with no data loss while still processing interactive queries.

* Rest API for storing and querying metrics and historical information. Most monitoring solution use special transports and protocols, such as CollectD or NSCA (Nagios). In our solution, http is the only protocol used. This simplifies the overall design and also allows for a much richer way of describing the data via dimensions.

* Multi-tenant and authenticated. Metrics are submitted and authenticated using Keystone and stored associated with a tenant ID.

* Metrics defined using a set of (key, value) pairs called dimensions.

* Real-time thresholding and alarming on metrics.

* Compound alarms described using a simple expressive grammar composed of alarm sub-expressions and logical operators.

* Monitoring agent that supports a number of built-in system and service checks and also supports Nagios checks and statsd.

* Open-source monitoring solution built on open-source technologies.

== Comparisons to alternatives ==
[[Monasca/Comparison]]

== Architecture ==
[[File:monasca-arch-component-diagram.png|Monasca Architecture Component Diagram]]
* Monitoring Agent (monasca-agent): A modern Python based monitoring agent that consists of several sub-components and supports system metrics, such as cpu utilization and available memory, Nagios plugins, statsd and many built-in checks for services such as MySQL, RabbitMQ, and many others.
* Monitoring API (monasca-api): A well-defined and documented RESTful API for monitoring that is primarily focused on the following concepts and areas:
** Metrics: Store and query massive amounts of metrics in real-time.
** Statistics: Query statistics for metrics.
** Alarm Definitions: Create, update, query and delete alarm definitions.
** Alarms: Query and delete the alarm history.
***Simple expressive grammar for creating compound alarms composed of alarm subexpressions and logical operators.
***Alarm severities can be associated with alarms.
***The complete alarm state transition history is stored and queryable which allows for subsequent root cause analysis (RCA) or advanced analytics.
**Notification Methods: Create and delete notification methods and associate them with alarms, such as email. Supports the ability to notify users directly via email when an alarm state transitions occur.
** The Monasca API has both Java and Python implementations avaialble.
* Persister (monasca-persister): Consumes metrics and alarm state transitions from the MessageQ and stores them in the Metrics and Alarms database.
** The Persister has both Java and Python implementations.
* Transform and Aggregation Engine (monasca-transform): Transform metric names and values, such as delta or time-based derivative calculations, and creates new metrics that are published to the Message Queue. The Transform Engine is not available yet.
* Anomaly and Prediction Engine: Evaluates prediction and anomalies and generates predicted metrics as well as anomaly likelihood and anomaly scores. The Anomaly and Prediction Engine is currently in a prototype status.
* Threshold Engine (monasca-thresh): Computes thresholds on metrics and publishes alarms to the MessageQ when exceeded. Based on Apache Storm a free and open distributed real-time computation system.
* Notification Engine (monasca-notification): Consumes alarm state transition messages from the MessageQ and sends notifications, such as emails for alarms. The Notification Engine is Python based.
* Analytics Engine (monasca-analytics): Consumes alarm state transisitions and metrics from the MessageQ and does anomaly detection and alarm clustering/correlation.
* Message Queue: A third-party component that primarily receives published metrics from the Monitoring API and alarm state transition messages from the Threshold Engine that are consumed by other components, such as the Persister and Notification Engine. The Message Queue is also used to publish and consume other events in the system. Currently, a Kafka based MessageQ is supported. Kafka is a high performance, distributed, fault-tolerant, and scalable message queue with durability built-in. We will look at other alternatives, such as RabbitMQ and in-fact in our previous implementation RabbitMQ was supported, but due to performance, scale, durability and high-availability limitiations with RabbitMQ we have moved to Kafka.
* Metrics and Alarms Database: A third-party component that primarily stores metrics and the alarm state history. Currently, Vertica and InfluxDB are supported. Support for Cassandra is in progress.
* Config Database: A third-party component that stores a lot of the configuration and other information in the system. Currently, MySQL is supported. Support for PostgreSQL is in progress.
* Monitoring Client (python-monascaclient): A Python command line client and library that communicates and controls the Monitoring API. The Monitoring Client was written using the OpenStack Heat Python client as a framework. The Monitoring Client also has a Python library, "monascaclient" similar to the other OpenStack clients, that can be used to quickly build additional capabilities. The Monitoring Client library is used by the Monitoring UI, Ceilometer publisher, and other components.
* Monitoring UI: A Horizon dashboard for visualizing the overall health and status of an OpenStack cloud.
* Ceilometer publisher: A multi-publisher plugin for Ceilometer, not shown, that converts and publishes samples to the Monitoring API.
Most of the components are described in their respective repositories. However, there aren't any repositories for the third-party components used, so we describe some of the relevant details here.

Further historical context for the architecture can be found at [[Monasca/Architecture Details]]

=== Message Schema ===
[[Monasca/Message Schema]]

=== Message Queue ===
A distributed, performant, scalable, HA message queue for distributing metrics, alarms and events in the monitoring system. Currently, based on Kafka.

==== Messages ====
There are several messages that are published and consumed by various components in Monasca via the MessageQ. See [[Monasca/Message Schema|Message Schema]].

=== Metrics and Alarms Database ===

A high-performance analytics database that can store massive amounts of metrics and alarms in real-time and also support interactive queries. Currently Vertica and InfluxDB are supported.

The SQL schema that is used by Vertica is as follows:

* MonMetrics.Measurements: Stores the actual measurements that are sent.
** id: An integer ID for the measurement.
** definition_dimensions_id: A reference to DefinitionDimensions.
** time_stamp
** value
* MonMetrics.DefinitionDimensions
** id: A sha1 hash of (defintion_id, dimension_set_id)
** definition_id: A reference to the Definitions.id
** dimension_set_id: A reference to the Dimensions.dimension_set_id
* MonMetrics.Definitions
** id: A sha1 hash of the (name, tenant_id, region)
** name: Name of the metric.
** tenant_id: The tenant_id that submitted the metric.
** region: The region the metric was submitted under.
* MonMetric.Dimensions
** dimension_set_id: A sha1 hash of the set of dimenions for a metric.
** name: Name of dimension.
** value: Value of dimension.

=== Config Database ===
The config database store all the configuration information. Currently based on MySQL.

The SQL schema is as follows:

* alarm
** id
** tenant_id
** name
** description
** expression
** state
** actions_enabled
** created_at
** updated_at
** deleted_at
* alarm_action
** alarm_id
** alarm_state
** action_id
* notification_method
** id
** tenant_id
** name
** type
** address
** created_at
** updated_at
* sub_alarm
** id
** alarm_id
** function
** metric_name
** operator
** threshold
** period
** periods
** state
** created_at
** updated_at
* sub_alarm_dimension
** sub_alarm_id
** dimension_name
** value

== Events ==
Support for real-time event stream processing in Monasca is in progress. For more details see the link at, [[Monasca/Events]].

== Logging ==
Support for logging in Monasca is under discussion. For more details see the link at, [[Monasca/Logging]].

== Transform and Aggregation Engine ==
For more details see the link at, [[Monasca/Transform]].

== Analytics ==
Support for anomaly detection and alarm clustering/correlation is in progress. For more details see the link at, [[Monasca/Analytics]].

== Monitoring ==
Enablement and usage for monitoring the status of Monasca is under discussion. For more details see the link at, [[Monasca/Monitoring_Of_Monasca]]

== UI/UX Support ==
Adding more support for common UI/UX queries is under discussion. For more details see the link at, [[Monasca/UI_UX_Support]]

== Keystone Requirements ==
Monasca relies on keystone for running and there are requirements about which keystone configuration must exist.

* The endpoint for the api must be registered in keystone as the 'monasca' service.
* The api must have an admin token to use in verifying the keystone tokens it receives.
* For each project which uses Monasca two users must exist, one will be in the 'monasca-agent' role and be used by the monasca-agent's running on machines. The other should not be in that role and can be used logging into the UI, using the CLI or for direct queries against the API.

== Post Metric Sequence ==

This section describes the sequence of operations involved in posting a metric to the Monasca API.

[[File:monasca-arch-post-metric-diagram.png|Monasca Architecture Post Metric Diagram]]

# A metric is posted to the Monasca API.
# The Monasca API authenticates and validates the request and publishes the metric to the the Message Queue.
# The Persister consumes the metric from the Message Queue and stores in the Metrics Store.
# The Transform Engine consumes the metrics from the Message Queue, performs transform and aggregation operations on metrics, and publishes metrics that it creates back to Message Queue.
# The Threshold Engine consumes metrics from the Message Queue and evaluates alarms. If a state change occurs in an alarm, an "alarm-state-transitioned-event" is published to the Message Queue.
# The Notification Engine consumes "alarm-state-transitioned-events" from the Message Queue, evaluates whether they have a Notification Method associated with it, and sends the appropriate notification, such as email.
# The Persister consumes the "alarm-state-transitioned-event" from the Message Queue and stores it in the Alarm State History Store.

== Alarm Managers ==
This section describes the new features including alarm grouping, inhibition and silencing.

Examples:

* Silence
SilenceRule1 = '{"alarm-silencing-definition-created": {"name": "silence_rule_1", "matchers": {"severity": "LOW"}, "start_time": "2017-02-21 20:00:00", "end_time": "2017-02-21 22:00:00"}}'
Two alarm transitions: AT1 and AT2
AT1_severity = HIGH
AT2_severity = LOW

Output: AT2 get silenced and AT1 send a notification.
SilenceRule1 expires after "end_time"="2017-02-21 22:00:00".

* Inhibit
InhibitionRule1 = '{"alarm-inhibition-definition-created": {"name": "inhibit_rule_1","source":{"severity":"HIGH"},"target":{"severity":"LOW"}, "equals":["tenantId"], "exclusions": {"alarm_name": "vm is dead"}}}'
Three alarm transitions: AT1, AT2 and AT3
AT1_tenantId = "d42bc"
AT2_tenantId = "d42bc"
AT3_tenantId = "d42bc"
AT1_severity = HIGH
AT2_severity = LOW
AT3_severity = LOW
AT1_alarm_name = "cpu high"
AT2_alarm_name = "memory high"
AT3_alarm_name = "vm is dead"
AT1_state = ALARM
AT2_state = ALARM
AT3_state = ALARM

Output: AT1 is the source alarm which will send a notification. AT2 is the target alarm and will get inhibited. AT3 matches the exclusions and will send a notification immediately.

* Grouping Scenario 1

GroupingRule1 = '{"alarm-grouping-definition-created":
{"name": "group_rule_1", "matchers": ["hostname"], "id": "b7163","repeat_interval": "2h", "group_wait": "30s", "exclusions": {"alarm_name": "cpu_percent_high"}, "tenantId": " d42bc", "alarm_actions": ["cd892"], "ok_actions": ["ad892"], "undetermined_actions": ["cf892"]}}'

Three alarm transitions: AT1, AT2 and AT3

AT1_hostname = host1
AT2_hostname = host1
AT3_hostname = host2
AT1_alarm_name = cpu_percent_high
AT2_alarm_name = cpu_system_perc_high
AT3_alarm_name = cpu_percent_high
AT1_state = ALARM
AT2_state = ALARM
AT3_state = ALARM

Output: AT1 and AT3 match exclusions and send notifications immediately. Generate a grouped notification “group_notification_rule_1_host1_alarm[1]” and send out using alarm_actions ["cd892"]. There are no alarm_actions, ok_actions or undermined_actions associated with the AT1, AT2, AT3 alarm definitions.

* Grouping Scenario 2

GroupingRule2 = '{"alarm-grouping-definition-created":
{"name": "group_rule_2", "matchers": ["hostname"], "id": "b7163","repeat_interval": "2h", "group_wait": "30s", "tenantId": " d42bc", "alarm_actions": ["cd892"], "ok_actions": ["ad892"], "undetermined_actions": ["cf892"]}}'

Three alarm transitions: AT1 with "alarm_actions": ["123ab"], AT2 with "alarm_actions": ["cd839"] and AT3

AT1_hostname = host1
AT2_hostname = host1
AT3_hostname = host2
AT1_state = ALARM
AT2_state = ALARM
AT3_state = ALARM

Output: Generate two grouped notifications “group_notification_rule_2_host1_alarm[2]” and “group_notification_rule_2_host2_alarm[1]”. Both using alarm_actions ["cd892"]. Also since AT1 and AT2 has their own alarm actions associated with them, there will be two more notifications sent out.

* Silenced and Grouped

SilenceRule1 = '{"alarm-silencing-definition-created": {"name": "silence_rule_1", "matchers": {"severity": "LOW"}, "start_time": "1487269470498", "end_time": "1587269470498"}}'

GroupingRule2 = '{"alarm-grouping-definition-created":
{"name": "group_rule_2", "matchers": ["hostname"], "id": "b7163","repeat_interval": "2h", "group_wait": "30s", "tenantId": " d42bc", "alarm_actions": ["cd892"], "ok_actions": ["ad892"], "undetermined_actions": ["cf892"]}}'

Four alarm transitions: AT1, AT2, AT3 and AT4

Silencing rule:
AT1_severity = HIGH
AT2_severity = LOW
AT3_severity = HIGH
AT4_severity = HIGH

Grouping rule:
AT1_hostname = host1
AT2_hostname = host1
AT3_hostname = host2
AT4_hostname = host1
AT1_state = ALARM
AT2_state = ALARM
AT3_state = OK
AT4_state = ALARM

Output: Generate two grouped notifications “group_notification_rule_2_host1_alarm[2]” using alarm action "cd892" and “group_notification_rule_2_host2_ok[1]” using ok action "ad892". AT2 is silenced so it isn't include in group_notification_rule_2_host1 count. There are no alarm_actions, ok_actions or undermined_actions associated with the AT1, AT2, AT3, AT4 alarm definitions.

* Silenced and inhibited (source alarm get silenced)

SilenceRule2 = '{"alarm-silencing-definition-created": {"name": "silence_rule_2", "matchers": {"severity": "HIGH, "hostname": "host1"}, "start_time": "2017-02-21 15:00:00", "end_time": "2017-02-21 21:00:00"}}'

InhibitionRule1 = '{"alarm-inhibition-definition-created": {"name": "inhibit_rule_1","source":{"severity":"HIGH"},"target":{"severity":"LOW"}, "equals":["tenantId"]}}'

Alarm transitions: AT1, AT2

Inhibition rule:
AT1_state = ALARM
AT2_state = ALARM
AT1_tenantId = "d42bc"
AT2_tenantId = "d42bc"
AT1_severity = HIGH
AT2_severity = LOW

Silencing rule:
AT1_hostname = host1
AT2_hostname = host2
AT1_severity = HIGH
AT2_severity = LOW

Output: no notification sent out. For inhibition, AT1 is the source alarm, AT2 is the target alarm. But at the same time, AT1 get silenced because it matches the silence rule.

* Inhibited and grouped

InhibitionRule1 = '{"alarm-inhibition-definition-created": {"name": "inhibit_rule_1","source":{"severity":"HIGH"},"target":{"severity":"LOW"}, "equals":["tenantId"]}}'

GroupingRule1 = '{"alarm-grouping-definition-created": {"name": "group_rule_1", "matchers": ["hostname"], "id": "b7163", "repeat_interval": "2h", "group_wait": "30s", "tenantId": " d42bc", "alarm_actions": ["cd892"], "ok_actions": ["ad892"], "undetermined_actions": ["cf892"]}}'

Alarm transitions: AT1, AT2, AT3

Inhibition rule:
AT1_tenantId = "d42bc"
AT2_tenantId = "d42bc"
AT3_tenantId = "d42bc"
AT1_severity = HIGH
AT2_severity = LOW
AT3_severity = HIGH
AT1_state = OK
AT2_state = ALARM
AT3_state = ALARM

Grouping rule:
AT1_hostname = host1
AT2_hostname = host2
AT3_hostname = host1
AT1_state = OK
AT2_state = ALARM
AT3_state = ALARM

Output: AT2 gets inhibited because its severity is low. AT3 is the source alarm. Since AT1 is in OK state, it is not a source alarm. For grouping, AT1 and AT3 has the same host name but different state. So there will be two grouped notifications sent out: “group_notification_rule_1_host1_ok[1]” and “group_notification_rule_1_host1_alarm[1]”.

* Silenced, inhibited and grouped

SilenceRule2 = '{"alarm-silencing-definition-created": {"name": "silence_rule_2", "matchers": {"severity": "HIGH, "hostname": "host1"}, "start_time": "2017-02-21 15:00:00", "end_time": "2017-02-21 21:00:00"}}'

InhibitionRule1 = '{"alarm-inhibition-definition-created": {"name": "inhibit_rule_1","source":{"severity":"HIGH"},"target":{"severity":"LOW"}, "equals":["tenantId"]}}'

GroupingRule1 = '{"alarm-grouping-definition-created": {"name": "group_rule_1", "matchers": ["hostname"], "id": "b7163", "repeat_interval": "2h", "group_wait": "30s", "tenantId": " d42bc", "alarm_actions": ["cd892"], "ok_actions": ["ad892"], "undetermined_actions": ["cf892"]}}'

Alarm transitions: AT1, AT2, AT3 and AT5

Silence rule:
AT1_severity = HIGH AT1_hostname = host1 (silenced)
AT2_severity = LOW AT2_hostname = host2
AT3_severity = HIGH AT3_hostname = host1(silenced)
AT5_severity = HIGH AT5_hostname = host3

Inhibition rule:
AT1_tenantId = "d42bc"
AT2_tenantId = "d42bc"
AT3_tenantId = "d42bc"
AT5_tenantId = "d42bc"
AT1_state = ALARM
AT2_state = ALARM
AT3_state = OK
AT5_state = UNDETERMINED
AT1_severity = HIGH (source)
AT2_severity = LOW (target)
AT3_severity = HIGH
AT5_severity = HIGH

Grouping rule:
AT1_hostname = host1
AT2_hostname = host2
AT3_hostname = host1
AT5_hostname = host3

Output: AT1 is in "group_notification_rule_1_host1_alarm" group and silenced. AT3 is in "group_notification_rule_1_host1_ok" group and silenced. AT2 is in "group_notification_rule_1_host2_alarm" group and inhibited. AT5 is in "group_notification_rule_1_host3_undetermined" group and will send notification “group_notification_rule_1_host3_undetermined[1]” using undetermined action "cf892".

= Development Environment =

* Monasca DevStack Plugin:
** DevStack is the primary developmement environment for OpenStack.
***See http://docs.openstack.org/developer/devstack/
** The Monasca DevStack plugin installs the Monasca Service, Agent, Horizon Monitoring Panel, and Grafana
** README at, https://github.com/openstack/monasca-api/tree/master/devstack
** Best way to get started is to use Vagrant with the Vagrantfile at, https://github.com/openstack/monasca-api/blob/master/devstack/Vagrantfile.

* Monasca on LaunchPad. https://launchpad.net/monasca

* Monasca projects source code. https://git.openstack.org/cgit/?q=monasca

= Coding Standards =

* Python: All Python code conforms to the OpenStack standards at, http://docs.openstack.org/developer/hacking/.
** Note, all components in Monasca, except for the Threshold Engine, have been ported to Python.

* Java: Several of the Monasca components are available as Java. OpenStack does not have any Java coding standards. We've adopted the Google Java Style at, https://google.github.io/styleguide/javaguide.html.
** The standard says either 80 or 100 length lines. We've adopted 100.

= Technologies =

Monasca uses a number of third-party technologies:

* Apache Kafka (http://kafka.apache.org): Apache Kafka is publish-subscribe messaging rethought as a distributed commit log. Kafka is a highly performant, distributed, fault-tolerant, and scalable message queue with durability built-in.

* Apache Storm (http://storm.incubator.apache.org/): Apache Storm is a free and open source distributed realtime computation system. Storm makes it easy to reliably process unbounded streams of data, doing for realtime processing what Hadoop did for batch processing.

* ZooKeeper (http://zookeeper.apache.org/): Used by Kafka and Storm.

* MySQL: MySQL is supported as a Config Database.

* POSTgres: Support for POSTgres, via Hibernate and SQLAlchemy, for the Config Database.

* Vagrant (http://www.vagrantup.com/): Vagrant provides easy to configure, reproducible, and portable work environments built on top of industry-standard technology and controlled by a single consistent workflow to help maximize the productivity and flexibility of you and your team.

* Dropwizard (https://dropwizard.github.io/dropwizard/): Dropwizard pulls together stable, mature libraries from the Java ecosystem into a simple, light-weight package that lets you focus on getting things done. Dropwizard has out-of-the-box support for sophisticated configuration, application metrics, logging, operational tools, and much more, allowing you and your team to ship a production-quality web service in the shortest time possible.

* InfluxDB (http://influxdb.com/): An open-source distributed time series database with no external dependencies. InfluxDB is supported for the Metrics Database.

* Vertica (http://www.vertica.com): A commercial Enterprise class SQL analytics database that is highly scalable. It offers built-in automatic high-availability and excels at in-database analytics and compressing and storing massive amounts of data. A free community version of Vertica is available that can store up to 1 TB of data with no time-limit is available at, https://my.vertica.com/community/. Vertica is supported for the Metrics Database.

* Cassandra: Support for Cassandra for the Metrics Database is in progress.

= License =

Copyright (c) 2014, 2015 Hewlett-Packard Development Company, L.P.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied.
See the License for the specific language governing permissions and
limitations under the License.

[[File:yklogo.png|200px|thumb|left|Monasca uses YourKit Profiler for Java development]]

 
 
 
 

[https://www.yourkit.com/ Visit YourKit website for more information]

Monasca/Architecture Details

2019-05-08T16:45:32Z

Joseph Davis: copy over from main page

== Overview ==

Monasca is a open-source multi-tenant, highly scalable, performant, fault-tolerant monitoring-as-a-service solution that integrates with OpenStack. It uses a REST API for high-speed metrics processing and querying and has a streaming alarm engine and notification engine.

'''<big>The following content is from the older version of the wiki, circa 2015, and is kept for historical reference.</big>'''

------

== Architecture (circal 2015) ==
[[File:monasca-arch-component-diagram.png|Monasca Architecture Component Diagram]]
* Monitoring Agent (monasca-agent): A modern Python based monitoring agent that consists of several sub-components and supports system metrics, such as cpu utilization and available memory, Nagios plugins, statsd and many built-in checks for services such as MySQL, RabbitMQ, and many others.
* Monitoring API (monasca-api): A well-defined and documented RESTful API for monitoring that is primarily focused on the following concepts and areas:
** Metrics: Store and query massive amounts of metrics in real-time.
** Statistics: Query statistics for metrics.
** Alarm Definitions: Create, update, query and delete alarm definitions.
** Alarms: Query and delete the alarm history.
***Simple expressive grammar for creating compound alarms composed of alarm subexpressions and logical operators.
***Alarm severities can be associated with alarms.
***The complete alarm state transition history is stored and queryable which allows for subsequent root cause analysis (RCA) or advanced analytics.
**Notification Methods: Create and delete notification methods and associate them with alarms, such as email. Supports the ability to notify users directly via email when an alarm state transitions occur.
** The Monasca API has both Java and Python implementations avaialble.
* Persister (monasca-persister): Consumes metrics and alarm state transitions from the MessageQ and stores them in the Metrics and Alarms database.
** The Persister has both Java and Python implementations.
* Transform and Aggregation Engine (monasca-transform): Transform metric names and values, such as delta or time-based derivative calculations, and creates new metrics that are published to the Message Queue. The Transform Engine is not available yet.
* Anomaly and Prediction Engine: Evaluates prediction and anomalies and generates predicted metrics as well as anomaly likelihood and anomaly scores. The Anomaly and Prediction Engine is currently in a prototype status.
* Threshold Engine (monasca-thresh): Computes thresholds on metrics and publishes alarms to the MessageQ when exceeded. Based on Apache Storm a free and open distributed real-time computation system.
* Notification Engine (monasca-notification): Consumes alarm state transition messages from the MessageQ and sends notifications, such as emails for alarms. The Notification Engine is Python based.
* Analytics Engine (monasca-analytics): Consumes alarm state transisitions and metrics from the MessageQ and does anomaly detection and alarm clustering/correlation.
* Message Queue: A third-party component that primarily receives published metrics from the Monitoring API and alarm state transition messages from the Threshold Engine that are consumed by other components, such as the Persister and Notification Engine. The Message Queue is also used to publish and consume other events in the system. Currently, a Kafka based MessageQ is supported. Kafka is a high performance, distributed, fault-tolerant, and scalable message queue with durability built-in. We will look at other alternatives, such as RabbitMQ and in-fact in our previous implementation RabbitMQ was supported, but due to performance, scale, durability and high-availability limitiations with RabbitMQ we have moved to Kafka.
* Metrics and Alarms Database: A third-party component that primarily stores metrics and the alarm state history. Currently, Vertica and InfluxDB are supported. Support for Cassandra is in progress.
* Config Database: A third-party component that stores a lot of the configuration and other information in the system. Currently, MySQL is supported. Support for PostgreSQL is in progress.
* Monitoring Client (python-monascaclient): A Python command line client and library that communicates and controls the Monitoring API. The Monitoring Client was written using the OpenStack Heat Python client as a framework. The Monitoring Client also has a Python library, "monascaclient" similar to the other OpenStack clients, that can be used to quickly build additional capabilities. The Monitoring Client library is used by the Monitoring UI, Ceilometer publisher, and other components.
* Monitoring UI: A Horizon dashboard for visualizing the overall health and status of an OpenStack cloud.
* Ceilometer publisher: A multi-publisher plugin for Ceilometer, not shown, that converts and publishes samples to the Monitoring API.
Most of the components are described in their respective repositories. However, there aren't any repositories for the third-party components used, so we describe some of the relevant details here.

=== Message Schema ===
[[Monasca/Message Schema]]

=== Message Queue ===
A distributed, performant, scalable, HA message queue for distributing metrics, alarms and events in the monitoring system. Currently, based on Kafka.

==== Messages ====
There are several messages that are published and consumed by various components in Monasca via the MessageQ. See [[Monasca/Message Schema|Message Schema]].

=== Metrics and Alarms Database ===

A high-performance analytics database that can store massive amounts of metrics and alarms in real-time and also support interactive queries. Currently Vertica and InfluxDB are supported.

The SQL schema that is used by Vertica is as follows:

* MonMetrics.Measurements: Stores the actual measurements that are sent.
** id: An integer ID for the measurement.
** definition_dimensions_id: A reference to DefinitionDimensions.
** time_stamp
** value
* MonMetrics.DefinitionDimensions
** id: A sha1 hash of (defintion_id, dimension_set_id)
** definition_id: A reference to the Definitions.id
** dimension_set_id: A reference to the Dimensions.dimension_set_id
* MonMetrics.Definitions
** id: A sha1 hash of the (name, tenant_id, region)
** name: Name of the metric.
** tenant_id: The tenant_id that submitted the metric.
** region: The region the metric was submitted under.
* MonMetric.Dimensions
** dimension_set_id: A sha1 hash of the set of dimenions for a metric.
** name: Name of dimension.
** value: Value of dimension.

=== Config Database ===
The config database store all the configuration information. Currently based on MySQL.

The SQL schema is as follows:

* alarm
** id
** tenant_id
** name
** description
** expression
** state
** actions_enabled
** created_at
** updated_at
** deleted_at
* alarm_action
** alarm_id
** alarm_state
** action_id
* notification_method
** id
** tenant_id
** name
** type
** address
** created_at
** updated_at
* sub_alarm
** id
** alarm_id
** function
** metric_name
** operator
** threshold
** period
** periods
** state
** created_at
** updated_at
* sub_alarm_dimension
** sub_alarm_id
** dimension_name
** value

== Events ==
Support for real-time event stream processing in Monasca is in progress. For more details see the link at, [[Monasca/Events]].

== Logging ==
Support for logging in Monasca is under discussion. For more details see the link at, [[Monasca/Logging]].

== Transform and Aggregation Engine ==
For more details see the link at, [[Monasca/Transform]].

== Analytics ==
Support for anomaly detection and alarm clustering/correlation is in progress. For more details see the link at, [[Monasca/Analytics]].

== Monitoring ==
Enablement and usage for monitoring the status of Monasca is under discussion. For more details see the link at, [[Monasca/Monitoring_Of_Monasca]]

== UI/UX Support ==
Adding more support for common UI/UX queries is under discussion. For more details see the link at, [[Monasca/UI_UX_Support]]

== Keystone Requirements ==
Monasca relies on keystone for running and there are requirements about which keystone configuration must exist.

* The endpoint for the api must be registered in keystone as the 'monasca' service.
* The api must have an admin token to use in verifying the keystone tokens it receives.
* For each project which uses Monasca two users must exist, one will be in the 'monasca-agent' role and be used by the monasca-agent's running on machines. The other should not be in that role and can be used logging into the UI, using the CLI or for direct queries against the API.

== Post Metric Sequence ==

This section describes the sequence of operations involved in posting a metric to the Monasca API.

[[File:monasca-arch-post-metric-diagram.png|Monasca Architecture Post Metric Diagram]]

# A metric is posted to the Monasca API.
# The Monasca API authenticates and validates the request and publishes the metric to the the Message Queue.
# The Persister consumes the metric from the Message Queue and stores in the Metrics Store.
# The Transform Engine consumes the metrics from the Message Queue, performs transform and aggregation operations on metrics, and publishes metrics that it creates back to Message Queue.
# The Threshold Engine consumes metrics from the Message Queue and evaluates alarms. If a state change occurs in an alarm, an "alarm-state-transitioned-event" is published to the Message Queue.
# The Notification Engine consumes "alarm-state-transitioned-events" from the Message Queue, evaluates whether they have a Notification Method associated with it, and sends the appropriate notification, such as email.
# The Persister consumes the "alarm-state-transitioned-event" from the Message Queue and stores it in the Alarm State History Store.

== Alarm Managers ==
This section describes the new features including alarm grouping, inhibition and silencing.

Examples:

* Silence
SilenceRule1 = '{"alarm-silencing-definition-created": {"name": "silence_rule_1", "matchers": {"severity": "LOW"}, "start_time": "2017-02-21 20:00:00", "end_time": "2017-02-21 22:00:00"}}'
Two alarm transitions: AT1 and AT2
AT1_severity = HIGH
AT2_severity = LOW

Output: AT2 get silenced and AT1 send a notification.
SilenceRule1 expires after "end_time"="2017-02-21 22:00:00".

* Inhibit
InhibitionRule1 = '{"alarm-inhibition-definition-created": {"name": "inhibit_rule_1","source":{"severity":"HIGH"},"target":{"severity":"LOW"}, "equals":["tenantId"], "exclusions": {"alarm_name": "vm is dead"}}}'
Three alarm transitions: AT1, AT2 and AT3
AT1_tenantId = "d42bc"
AT2_tenantId = "d42bc"
AT3_tenantId = "d42bc"
AT1_severity = HIGH
AT2_severity = LOW
AT3_severity = LOW
AT1_alarm_name = "cpu high"
AT2_alarm_name = "memory high"
AT3_alarm_name = "vm is dead"
AT1_state = ALARM
AT2_state = ALARM
AT3_state = ALARM

Output: AT1 is the source alarm which will send a notification. AT2 is the target alarm and will get inhibited. AT3 matches the exclusions and will send a notification immediately.

* Grouping Scenario 1

GroupingRule1 = '{"alarm-grouping-definition-created":
{"name": "group_rule_1", "matchers": ["hostname"], "id": "b7163","repeat_interval": "2h", "group_wait": "30s", "exclusions": {"alarm_name": "cpu_percent_high"}, "tenantId": " d42bc", "alarm_actions": ["cd892"], "ok_actions": ["ad892"], "undetermined_actions": ["cf892"]}}'

Three alarm transitions: AT1, AT2 and AT3

AT1_hostname = host1
AT2_hostname = host1
AT3_hostname = host2
AT1_alarm_name = cpu_percent_high
AT2_alarm_name = cpu_system_perc_high
AT3_alarm_name = cpu_percent_high
AT1_state = ALARM
AT2_state = ALARM
AT3_state = ALARM

Output: AT1 and AT3 match exclusions and send notifications immediately. Generate a grouped notification “group_notification_rule_1_host1_alarm[1]” and send out using alarm_actions ["cd892"]. There are no alarm_actions, ok_actions or undermined_actions associated with the AT1, AT2, AT3 alarm definitions.

* Grouping Scenario 2

GroupingRule2 = '{"alarm-grouping-definition-created":
{"name": "group_rule_2", "matchers": ["hostname"], "id": "b7163","repeat_interval": "2h", "group_wait": "30s", "tenantId": " d42bc", "alarm_actions": ["cd892"], "ok_actions": ["ad892"], "undetermined_actions": ["cf892"]}}'

Three alarm transitions: AT1 with "alarm_actions": ["123ab"], AT2 with "alarm_actions": ["cd839"] and AT3

AT1_hostname = host1
AT2_hostname = host1
AT3_hostname = host2
AT1_state = ALARM
AT2_state = ALARM
AT3_state = ALARM

Output: Generate two grouped notifications “group_notification_rule_2_host1_alarm[2]” and “group_notification_rule_2_host2_alarm[1]”. Both using alarm_actions ["cd892"]. Also since AT1 and AT2 has their own alarm actions associated with them, there will be two more notifications sent out.

* Silenced and Grouped

SilenceRule1 = '{"alarm-silencing-definition-created": {"name": "silence_rule_1", "matchers": {"severity": "LOW"}, "start_time": "1487269470498", "end_time": "1587269470498"}}'

GroupingRule2 = '{"alarm-grouping-definition-created":
{"name": "group_rule_2", "matchers": ["hostname"], "id": "b7163","repeat_interval": "2h", "group_wait": "30s", "tenantId": " d42bc", "alarm_actions": ["cd892"], "ok_actions": ["ad892"], "undetermined_actions": ["cf892"]}}'

Four alarm transitions: AT1, AT2, AT3 and AT4

Silencing rule:
AT1_severity = HIGH
AT2_severity = LOW
AT3_severity = HIGH
AT4_severity = HIGH

Grouping rule:
AT1_hostname = host1
AT2_hostname = host1
AT3_hostname = host2
AT4_hostname = host1
AT1_state = ALARM
AT2_state = ALARM
AT3_state = OK
AT4_state = ALARM

Output: Generate two grouped notifications “group_notification_rule_2_host1_alarm[2]” using alarm action "cd892" and “group_notification_rule_2_host2_ok[1]” using ok action "ad892". AT2 is silenced so it isn't include in group_notification_rule_2_host1 count. There are no alarm_actions, ok_actions or undermined_actions associated with the AT1, AT2, AT3, AT4 alarm definitions.

* Silenced and inhibited (source alarm get silenced)

SilenceRule2 = '{"alarm-silencing-definition-created": {"name": "silence_rule_2", "matchers": {"severity": "HIGH, "hostname": "host1"}, "start_time": "2017-02-21 15:00:00", "end_time": "2017-02-21 21:00:00"}}'

InhibitionRule1 = '{"alarm-inhibition-definition-created": {"name": "inhibit_rule_1","source":{"severity":"HIGH"},"target":{"severity":"LOW"}, "equals":["tenantId"]}}'

Alarm transitions: AT1, AT2

Inhibition rule:
AT1_state = ALARM
AT2_state = ALARM
AT1_tenantId = "d42bc"
AT2_tenantId = "d42bc"
AT1_severity = HIGH
AT2_severity = LOW

Silencing rule:
AT1_hostname = host1
AT2_hostname = host2
AT1_severity = HIGH
AT2_severity = LOW

Output: no notification sent out. For inhibition, AT1 is the source alarm, AT2 is the target alarm. But at the same time, AT1 get silenced because it matches the silence rule.

* Inhibited and grouped

InhibitionRule1 = '{"alarm-inhibition-definition-created": {"name": "inhibit_rule_1","source":{"severity":"HIGH"},"target":{"severity":"LOW"}, "equals":["tenantId"]}}'

GroupingRule1 = '{"alarm-grouping-definition-created": {"name": "group_rule_1", "matchers": ["hostname"], "id": "b7163", "repeat_interval": "2h", "group_wait": "30s", "tenantId": " d42bc", "alarm_actions": ["cd892"], "ok_actions": ["ad892"], "undetermined_actions": ["cf892"]}}'

Alarm transitions: AT1, AT2, AT3

Inhibition rule:
AT1_tenantId = "d42bc"
AT2_tenantId = "d42bc"
AT3_tenantId = "d42bc"
AT1_severity = HIGH
AT2_severity = LOW
AT3_severity = HIGH
AT1_state = OK
AT2_state = ALARM
AT3_state = ALARM

Grouping rule:
AT1_hostname = host1
AT2_hostname = host2
AT3_hostname = host1
AT1_state = OK
AT2_state = ALARM
AT3_state = ALARM

Output: AT2 gets inhibited because its severity is low. AT3 is the source alarm. Since AT1 is in OK state, it is not a source alarm. For grouping, AT1 and AT3 has the same host name but different state. So there will be two grouped notifications sent out: “group_notification_rule_1_host1_ok[1]” and “group_notification_rule_1_host1_alarm[1]”.

* Silenced, inhibited and grouped

SilenceRule2 = '{"alarm-silencing-definition-created": {"name": "silence_rule_2", "matchers": {"severity": "HIGH, "hostname": "host1"}, "start_time": "2017-02-21 15:00:00", "end_time": "2017-02-21 21:00:00"}}'

InhibitionRule1 = '{"alarm-inhibition-definition-created": {"name": "inhibit_rule_1","source":{"severity":"HIGH"},"target":{"severity":"LOW"}, "equals":["tenantId"]}}'

GroupingRule1 = '{"alarm-grouping-definition-created": {"name": "group_rule_1", "matchers": ["hostname"], "id": "b7163", "repeat_interval": "2h", "group_wait": "30s", "tenantId": " d42bc", "alarm_actions": ["cd892"], "ok_actions": ["ad892"], "undetermined_actions": ["cf892"]}}'

Alarm transitions: AT1, AT2, AT3 and AT5

Silence rule:
AT1_severity = HIGH AT1_hostname = host1 (silenced)
AT2_severity = LOW AT2_hostname = host2
AT3_severity = HIGH AT3_hostname = host1(silenced)
AT5_severity = HIGH AT5_hostname = host3

Inhibition rule:
AT1_tenantId = "d42bc"
AT2_tenantId = "d42bc"
AT3_tenantId = "d42bc"
AT5_tenantId = "d42bc"
AT1_state = ALARM
AT2_state = ALARM
AT3_state = OK
AT5_state = UNDETERMINED
AT1_severity = HIGH (source)
AT2_severity = LOW (target)
AT3_severity = HIGH
AT5_severity = HIGH

Grouping rule:
AT1_hostname = host1
AT2_hostname = host2
AT3_hostname = host1
AT5_hostname = host3

Output: AT1 is in "group_notification_rule_1_host1_alarm" group and silenced. AT3 is in "group_notification_rule_1_host1_ok" group and silenced. AT2 is in "group_notification_rule_1_host2_alarm" group and inhibited. AT5 is in "group_notification_rule_1_host3_undetermined" group and will send notification “group_notification_rule_1_host3_undetermined[1]” using undetermined action "cf892".

= Development Environment =

* Monasca DevStack Plugin:
** DevStack is the primary developmement environment for OpenStack.
***See http://docs.openstack.org/developer/devstack/
** The Monasca DevStack plugin installs the Monasca Service, Agent, Horizon Monitoring Panel, and Grafana
** README at, https://github.com/openstack/monasca-api/tree/master/devstack
** Best way to get started is to use Vagrant with the Vagrantfile at, https://github.com/openstack/monasca-api/blob/master/devstack/Vagrantfile.

* Monasca on LaunchPad. https://launchpad.net/monasca

* Monasca projects source code. https://git.openstack.org/cgit/?q=monasca

= Coding Standards =

* Python: All Python code conforms to the OpenStack standards at, http://docs.openstack.org/developer/hacking/.
** Note, all components in Monasca, except for the Threshold Engine, have been ported to Python.

* Java: Several of the Monasca components are available as Java. OpenStack does not have any Java coding standards. We've adopted the Google Java Style at, https://google.github.io/styleguide/javaguide.html.
** The standard says either 80 or 100 length lines. We've adopted 100.

= Technologies =

Monasca uses a number of third-party technologies:

* Apache Kafka (http://kafka.apache.org): Apache Kafka is publish-subscribe messaging rethought as a distributed commit log. Kafka is a highly performant, distributed, fault-tolerant, and scalable message queue with durability built-in.

* Apache Storm (http://storm.incubator.apache.org/): Apache Storm is a free and open source distributed realtime computation system. Storm makes it easy to reliably process unbounded streams of data, doing for realtime processing what Hadoop did for batch processing.

* ZooKeeper (http://zookeeper.apache.org/): Used by Kafka and Storm.

* MySQL: MySQL is supported as a Config Database.

* POSTgres: Support for POSTgres, via Hibernate and SQLAlchemy, for the Config Database.

* Vagrant (http://www.vagrantup.com/): Vagrant provides easy to configure, reproducible, and portable work environments built on top of industry-standard technology and controlled by a single consistent workflow to help maximize the productivity and flexibility of you and your team.

* Dropwizard (https://dropwizard.github.io/dropwizard/): Dropwizard pulls together stable, mature libraries from the Java ecosystem into a simple, light-weight package that lets you focus on getting things done. Dropwizard has out-of-the-box support for sophisticated configuration, application metrics, logging, operational tools, and much more, allowing you and your team to ship a production-quality web service in the shortest time possible.

* InfluxDB (http://influxdb.com/): An open-source distributed time series database with no external dependencies. InfluxDB is supported for the Metrics Database.

* Vertica (http://www.vertica.com): A commercial Enterprise class SQL analytics database that is highly scalable. It offers built-in automatic high-availability and excels at in-database analytics and compressing and storing massive amounts of data. A free community version of Vertica is available that can store up to 1 TB of data with no time-limit is available at, https://my.vertica.com/community/. Vertica is supported for the Metrics Database.

* Cassandra: Support for Cassandra for the Metrics Database is in progress.

= License =

Copyright (c) 2014, 2015 Hewlett-Packard Development Company, L.P.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied.
See the License for the specific language governing permissions and
limitations under the License.

[[File:yklogo.png|200px|thumb|left|Monasca uses YourKit Profiler for Java development]]

 
 
 
 

[https://www.yourkit.com/ Visit YourKit website for more information]

Monasca

2019-05-08T16:41:32Z

Joseph Davis: /* Architecture */ add link to Architecture Details page

== Overview ==

Monasca is a open-source multi-tenant, highly scalable, performant, fault-tolerant monitoring-as-a-service solution that integrates with OpenStack. It uses a REST API for high-speed metrics processing and querying and has a streaming alarm engine and notification engine.

[[File:OpenStack_Project_Monasca_vertical.png|thumb|right]]

==== Contribution ====
* [https://storyboard.openstack.org/#!/project_group/59 Project Group on StoryBoard]
* [https://storyboard.openstack.org/#!/board/111 Important activities in Stein release]
* {https://storyboard.openstack.org/#!/board/141 Important activities in Train release]
* [https://docs.openstack.org/monasca-api/latest/contributor/index.html Contribution Guidelines]

==== Communication and Meetings ====
* IRC: #openstack-monasca on freenode.net
* Weekly Meetings:
** http://eavesdrop.openstack.org/#Monasca_Team_Meeting
** [[Meetings/Monasca]]
* Use [Monasca] tag on [http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-discuss openstack-discuss mailing list].

===== PTG meetings =====
* [[MonascaSteinPTG|Stein PTG Summary (Denver)]]
* (coming soon - Train summary)

==== Documentation ====
: Monasca API Specification: https://github.com/openstack/monasca-api/blob/master/docs/monasca-api-spec.md
: Agent Documentation: https://github.com/openstack/monasca-agent

==== Logo ====
: https://www.openstack.org/themes/openstack/images/project-mascots/Monasca/OpenStack_Project_Monasca_horizontal_1color.eps
: https://www.openstack.org/themes/openstack/images/project-mascots/Monasca/OpenStack_Project_Monasca_horizontal.eps
: https://www.openstack.org/themes/openstack/images/project-mascots/Monasca/OpenStack_Project_Monasca_horizontal.jpg
: https://www.openstack.org/themes/openstack/images/project-mascots/Monasca/OpenStack_Project_Monasca_horizontal.png
: https://www.openstack.org/themes/openstack/images/project-mascots/Monasca/OpenStack_Project_Monasca_mascot.jpg
: https://www.openstack.org/themes/openstack/images/project-mascots/Monasca/OpenStack_Project_Monasca_mascot.png
: https://www.openstack.org/themes/openstack/images/project-mascots/Monasca/OpenStack_Project_Monasca_vertical_1color.eps
: https://www.openstack.org/themes/openstack/images/project-mascots/Monasca/OpenStack_Project_Monasca_vertical.eps
: https://www.openstack.org/themes/openstack/images/project-mascots/Monasca/OpenStack_Project_Monasca_vertical.jpg
: https://www.openstack.org/themes/openstack/images/project-mascots/Monasca/OpenStack_Project_Monasca_vertical.png

==== Presentations ====
There have been many interesting and excellent presentations on Monasca, many given at the OpenStack or OpenInfra Summits.
: [[Monasca/Presentations]] (updated with Denver 2019 links)

==== Repositories ====

: '''Core'''
:: Core: https://git.openstack.org/cgit/?q=monasca

: '''Deployment'''
The following repositories are available for deploying Monasca:
* Docker: https://github.com/monasca/monasca-docker
* Kubernetes: https://github.com/monasca/monasca-helm
* Ansible: https://github.com/search?utf8=%E2%9C%93&q=ansible-monasca
* Puppet: https://git.openstack.org/openstack/puppet-monasca

==== Bugs ====

===== All open bugs =====
: https://storyboard.openstack.org/#!/worklist/213

===== In Stein triaged bugs =====
: https://storyboard.openstack.org/#!/worklist/467

===== Bug-fixing progress =====
: https://storyboard.openstack.org/#!/board/114

== Requirements ==
[[Monasca/Requirements]].

== Features ==

This section describes the overall features.

* A highly performant, scalable, reliable and fault-tolerant Monitoring as a Service (MONaaS) solution that scales to service provider metrics levels of metrics throughput. Performance, scalability and high-availability have been designed in from the start. Can process 100s of thousands of metrics/sec as well as offer data retention periods of greater than a year with no data loss while still processing interactive queries.

* Rest API for storing and querying metrics and historical information. Most monitoring solution use special transports and protocols, such as CollectD or NSCA (Nagios). In our solution, http is the only protocol used. This simplifies the overall design and also allows for a much richer way of describing the data via dimensions.

* Multi-tenant and authenticated. Metrics are submitted and authenticated using Keystone and stored associated with a tenant ID.

* Metrics defined using a set of (key, value) pairs called dimensions.

* Real-time thresholding and alarming on metrics.

* Compound alarms described using a simple expressive grammar composed of alarm sub-expressions and logical operators.

* Monitoring agent that supports a number of built-in system and service checks and also supports Nagios checks and statsd.

* Open-source monitoring solution built on open-source technologies.

== Comparisons to alternatives ==
[[Monasca/Comparison]]

== Architecture ==
[[File:monasca-arch-component-diagram.png|Monasca Architecture Component Diagram]]
* Monitoring Agent (monasca-agent): A modern Python based monitoring agent that consists of several sub-components and supports system metrics, such as cpu utilization and available memory, Nagios plugins, statsd and many built-in checks for services such as MySQL, RabbitMQ, and many others.
* Monitoring API (monasca-api): A well-defined and documented RESTful API for monitoring that is primarily focused on the following concepts and areas:
** Metrics: Store and query massive amounts of metrics in real-time.
** Statistics: Query statistics for metrics.
** Alarm Definitions: Create, update, query and delete alarm definitions.
** Alarms: Query and delete the alarm history.
***Simple expressive grammar for creating compound alarms composed of alarm subexpressions and logical operators.
***Alarm severities can be associated with alarms.
***The complete alarm state transition history is stored and queryable which allows for subsequent root cause analysis (RCA) or advanced analytics.
**Notification Methods: Create and delete notification methods and associate them with alarms, such as email. Supports the ability to notify users directly via email when an alarm state transitions occur.
** The Monasca API has both Java and Python implementations avaialble.
* Persister (monasca-persister): Consumes metrics and alarm state transitions from the MessageQ and stores them in the Metrics and Alarms database.
** The Persister has both Java and Python implementations.
* Transform and Aggregation Engine (monasca-transform): Transform metric names and values, such as delta or time-based derivative calculations, and creates new metrics that are published to the Message Queue. The Transform Engine is not available yet.
* Anomaly and Prediction Engine: Evaluates prediction and anomalies and generates predicted metrics as well as anomaly likelihood and anomaly scores. The Anomaly and Prediction Engine is currently in a prototype status.
* Threshold Engine (monasca-thresh): Computes thresholds on metrics and publishes alarms to the MessageQ when exceeded. Based on Apache Storm a free and open distributed real-time computation system.
* Notification Engine (monasca-notification): Consumes alarm state transition messages from the MessageQ and sends notifications, such as emails for alarms. The Notification Engine is Python based.
* Analytics Engine (monasca-analytics): Consumes alarm state transisitions and metrics from the MessageQ and does anomaly detection and alarm clustering/correlation.
* Message Queue: A third-party component that primarily receives published metrics from the Monitoring API and alarm state transition messages from the Threshold Engine that are consumed by other components, such as the Persister and Notification Engine. The Message Queue is also used to publish and consume other events in the system. Currently, a Kafka based MessageQ is supported. Kafka is a high performance, distributed, fault-tolerant, and scalable message queue with durability built-in. We will look at other alternatives, such as RabbitMQ and in-fact in our previous implementation RabbitMQ was supported, but due to performance, scale, durability and high-availability limitiations with RabbitMQ we have moved to Kafka.
* Metrics and Alarms Database: A third-party component that primarily stores metrics and the alarm state history. Currently, Vertica and InfluxDB are supported. Support for Cassandra is in progress.
* Config Database: A third-party component that stores a lot of the configuration and other information in the system. Currently, MySQL is supported. Support for PostgreSQL is in progress.
* Monitoring Client (python-monascaclient): A Python command line client and library that communicates and controls the Monitoring API. The Monitoring Client was written using the OpenStack Heat Python client as a framework. The Monitoring Client also has a Python library, "monascaclient" similar to the other OpenStack clients, that can be used to quickly build additional capabilities. The Monitoring Client library is used by the Monitoring UI, Ceilometer publisher, and other components.
* Monitoring UI: A Horizon dashboard for visualizing the overall health and status of an OpenStack cloud.
* Ceilometer publisher: A multi-publisher plugin for Ceilometer, not shown, that converts and publishes samples to the Monitoring API.
Most of the components are described in their respective repositories. However, there aren't any repositories for the third-party components used, so we describe some of the relevant details here.

Further historical context for the architecture can be found at [[Monasca/Architecture Details]]

=== Message Schema ===
[[Monasca/Message Schema]]

=== Message Queue ===
A distributed, performant, scalable, HA message queue for distributing metrics, alarms and events in the monitoring system. Currently, based on Kafka.

==== Messages ====
There are several messages that are published and consumed by various components in Monasca via the MessageQ. See [[Monasca/Message Schema|Message Schema]].

=== Metrics and Alarms Database ===

A high-performance analytics database that can store massive amounts of metrics and alarms in real-time and also support interactive queries. Currently Vertica and InfluxDB are supported.

The SQL schema that is used by Vertica is as follows:

* MonMetrics.Measurements: Stores the actual measurements that are sent.
** id: An integer ID for the measurement.
** definition_dimensions_id: A reference to DefinitionDimensions.
** time_stamp
** value
* MonMetrics.DefinitionDimensions
** id: A sha1 hash of (defintion_id, dimension_set_id)
** definition_id: A reference to the Definitions.id
** dimension_set_id: A reference to the Dimensions.dimension_set_id
* MonMetrics.Definitions
** id: A sha1 hash of the (name, tenant_id, region)
** name: Name of the metric.
** tenant_id: The tenant_id that submitted the metric.
** region: The region the metric was submitted under.
* MonMetric.Dimensions
** dimension_set_id: A sha1 hash of the set of dimenions for a metric.
** name: Name of dimension.
** value: Value of dimension.

=== Config Database ===
The config database store all the configuration information. Currently based on MySQL.

The SQL schema is as follows:

* alarm
** id
** tenant_id
** name
** description
** expression
** state
** actions_enabled
** created_at
** updated_at
** deleted_at
* alarm_action
** alarm_id
** alarm_state
** action_id
* notification_method
** id
** tenant_id
** name
** type
** address
** created_at
** updated_at
* sub_alarm
** id
** alarm_id
** function
** metric_name
** operator
** threshold
** period
** periods
** state
** created_at
** updated_at
* sub_alarm_dimension
** sub_alarm_id
** dimension_name
** value

== Events ==
Support for real-time event stream processing in Monasca is in progress. For more details see the link at, [[Monasca/Events]].

== Logging ==
Support for logging in Monasca is under discussion. For more details see the link at, [[Monasca/Logging]].

== Transform and Aggregation Engine ==
For more details see the link at, [[Monasca/Transform]].

== Analytics ==
Support for anomaly detection and alarm clustering/correlation is in progress. For more details see the link at, [[Monasca/Analytics]].

== Monitoring ==
Enablement and usage for monitoring the status of Monasca is under discussion. For more details see the link at, [[Monasca/Monitoring_Of_Monasca]]

== UI/UX Support ==
Adding more support for common UI/UX queries is under discussion. For more details see the link at, [[Monasca/UI_UX_Support]]

== Keystone Requirements ==
Monasca relies on keystone for running and there are requirements about which keystone configuration must exist.

* The endpoint for the api must be registered in keystone as the 'monasca' service.
* The api must have an admin token to use in verifying the keystone tokens it receives.
* For each project which uses Monasca two users must exist, one will be in the 'monasca-agent' role and be used by the monasca-agent's running on machines. The other should not be in that role and can be used logging into the UI, using the CLI or for direct queries against the API.

== Post Metric Sequence ==

This section describes the sequence of operations involved in posting a metric to the Monasca API.

[[File:monasca-arch-post-metric-diagram.png|Monasca Architecture Post Metric Diagram]]

# A metric is posted to the Monasca API.
# The Monasca API authenticates and validates the request and publishes the metric to the the Message Queue.
# The Persister consumes the metric from the Message Queue and stores in the Metrics Store.
# The Transform Engine consumes the metrics from the Message Queue, performs transform and aggregation operations on metrics, and publishes metrics that it creates back to Message Queue.
# The Threshold Engine consumes metrics from the Message Queue and evaluates alarms. If a state change occurs in an alarm, an "alarm-state-transitioned-event" is published to the Message Queue.
# The Notification Engine consumes "alarm-state-transitioned-events" from the Message Queue, evaluates whether they have a Notification Method associated with it, and sends the appropriate notification, such as email.
# The Persister consumes the "alarm-state-transitioned-event" from the Message Queue and stores it in the Alarm State History Store.

== Alarm Managers ==
This section describes the new features including alarm grouping, inhibition and silencing.

Examples:

* Silence
SilenceRule1 = '{"alarm-silencing-definition-created": {"name": "silence_rule_1", "matchers": {"severity": "LOW"}, "start_time": "2017-02-21 20:00:00", "end_time": "2017-02-21 22:00:00"}}'
Two alarm transitions: AT1 and AT2
AT1_severity = HIGH
AT2_severity = LOW

Output: AT2 get silenced and AT1 send a notification.
SilenceRule1 expires after "end_time"="2017-02-21 22:00:00".

* Inhibit
InhibitionRule1 = '{"alarm-inhibition-definition-created": {"name": "inhibit_rule_1","source":{"severity":"HIGH"},"target":{"severity":"LOW"}, "equals":["tenantId"], "exclusions": {"alarm_name": "vm is dead"}}}'
Three alarm transitions: AT1, AT2 and AT3
AT1_tenantId = "d42bc"
AT2_tenantId = "d42bc"
AT3_tenantId = "d42bc"
AT1_severity = HIGH
AT2_severity = LOW
AT3_severity = LOW
AT1_alarm_name = "cpu high"
AT2_alarm_name = "memory high"
AT3_alarm_name = "vm is dead"
AT1_state = ALARM
AT2_state = ALARM
AT3_state = ALARM

Output: AT1 is the source alarm which will send a notification. AT2 is the target alarm and will get inhibited. AT3 matches the exclusions and will send a notification immediately.

* Grouping Scenario 1

GroupingRule1 = '{"alarm-grouping-definition-created":
{"name": "group_rule_1", "matchers": ["hostname"], "id": "b7163","repeat_interval": "2h", "group_wait": "30s", "exclusions": {"alarm_name": "cpu_percent_high"}, "tenantId": " d42bc", "alarm_actions": ["cd892"], "ok_actions": ["ad892"], "undetermined_actions": ["cf892"]}}'

Three alarm transitions: AT1, AT2 and AT3

AT1_hostname = host1
AT2_hostname = host1
AT3_hostname = host2
AT1_alarm_name = cpu_percent_high
AT2_alarm_name = cpu_system_perc_high
AT3_alarm_name = cpu_percent_high
AT1_state = ALARM
AT2_state = ALARM
AT3_state = ALARM

Output: AT1 and AT3 match exclusions and send notifications immediately. Generate a grouped notification “group_notification_rule_1_host1_alarm[1]” and send out using alarm_actions ["cd892"]. There are no alarm_actions, ok_actions or undermined_actions associated with the AT1, AT2, AT3 alarm definitions.

* Grouping Scenario 2

GroupingRule2 = '{"alarm-grouping-definition-created":
{"name": "group_rule_2", "matchers": ["hostname"], "id": "b7163","repeat_interval": "2h", "group_wait": "30s", "tenantId": " d42bc", "alarm_actions": ["cd892"], "ok_actions": ["ad892"], "undetermined_actions": ["cf892"]}}'

Three alarm transitions: AT1 with "alarm_actions": ["123ab"], AT2 with "alarm_actions": ["cd839"] and AT3

AT1_hostname = host1
AT2_hostname = host1
AT3_hostname = host2
AT1_state = ALARM
AT2_state = ALARM
AT3_state = ALARM

Output: Generate two grouped notifications “group_notification_rule_2_host1_alarm[2]” and “group_notification_rule_2_host2_alarm[1]”. Both using alarm_actions ["cd892"]. Also since AT1 and AT2 has their own alarm actions associated with them, there will be two more notifications sent out.

* Silenced and Grouped

SilenceRule1 = '{"alarm-silencing-definition-created": {"name": "silence_rule_1", "matchers": {"severity": "LOW"}, "start_time": "1487269470498", "end_time": "1587269470498"}}'

GroupingRule2 = '{"alarm-grouping-definition-created":
{"name": "group_rule_2", "matchers": ["hostname"], "id": "b7163","repeat_interval": "2h", "group_wait": "30s", "tenantId": " d42bc", "alarm_actions": ["cd892"], "ok_actions": ["ad892"], "undetermined_actions": ["cf892"]}}'

Four alarm transitions: AT1, AT2, AT3 and AT4

Silencing rule:
AT1_severity = HIGH
AT2_severity = LOW
AT3_severity = HIGH
AT4_severity = HIGH

Grouping rule:
AT1_hostname = host1
AT2_hostname = host1
AT3_hostname = host2
AT4_hostname = host1
AT1_state = ALARM
AT2_state = ALARM
AT3_state = OK
AT4_state = ALARM

Output: Generate two grouped notifications “group_notification_rule_2_host1_alarm[2]” using alarm action "cd892" and “group_notification_rule_2_host2_ok[1]” using ok action "ad892". AT2 is silenced so it isn't include in group_notification_rule_2_host1 count. There are no alarm_actions, ok_actions or undermined_actions associated with the AT1, AT2, AT3, AT4 alarm definitions.

* Silenced and inhibited (source alarm get silenced)

SilenceRule2 = '{"alarm-silencing-definition-created": {"name": "silence_rule_2", "matchers": {"severity": "HIGH, "hostname": "host1"}, "start_time": "2017-02-21 15:00:00", "end_time": "2017-02-21 21:00:00"}}'

InhibitionRule1 = '{"alarm-inhibition-definition-created": {"name": "inhibit_rule_1","source":{"severity":"HIGH"},"target":{"severity":"LOW"}, "equals":["tenantId"]}}'

Alarm transitions: AT1, AT2

Inhibition rule:
AT1_state = ALARM
AT2_state = ALARM
AT1_tenantId = "d42bc"
AT2_tenantId = "d42bc"
AT1_severity = HIGH
AT2_severity = LOW

Silencing rule:
AT1_hostname = host1
AT2_hostname = host2
AT1_severity = HIGH
AT2_severity = LOW

Output: no notification sent out. For inhibition, AT1 is the source alarm, AT2 is the target alarm. But at the same time, AT1 get silenced because it matches the silence rule.

* Inhibited and grouped

InhibitionRule1 = '{"alarm-inhibition-definition-created": {"name": "inhibit_rule_1","source":{"severity":"HIGH"},"target":{"severity":"LOW"}, "equals":["tenantId"]}}'

GroupingRule1 = '{"alarm-grouping-definition-created": {"name": "group_rule_1", "matchers": ["hostname"], "id": "b7163", "repeat_interval": "2h", "group_wait": "30s", "tenantId": " d42bc", "alarm_actions": ["cd892"], "ok_actions": ["ad892"], "undetermined_actions": ["cf892"]}}'

Alarm transitions: AT1, AT2, AT3

Inhibition rule:
AT1_tenantId = "d42bc"
AT2_tenantId = "d42bc"
AT3_tenantId = "d42bc"
AT1_severity = HIGH
AT2_severity = LOW
AT3_severity = HIGH
AT1_state = OK
AT2_state = ALARM
AT3_state = ALARM

Grouping rule:
AT1_hostname = host1
AT2_hostname = host2
AT3_hostname = host1
AT1_state = OK
AT2_state = ALARM
AT3_state = ALARM

Output: AT2 gets inhibited because its severity is low. AT3 is the source alarm. Since AT1 is in OK state, it is not a source alarm. For grouping, AT1 and AT3 has the same host name but different state. So there will be two grouped notifications sent out: “group_notification_rule_1_host1_ok[1]” and “group_notification_rule_1_host1_alarm[1]”.

* Silenced, inhibited and grouped

SilenceRule2 = '{"alarm-silencing-definition-created": {"name": "silence_rule_2", "matchers": {"severity": "HIGH, "hostname": "host1"}, "start_time": "2017-02-21 15:00:00", "end_time": "2017-02-21 21:00:00"}}'

InhibitionRule1 = '{"alarm-inhibition-definition-created": {"name": "inhibit_rule_1","source":{"severity":"HIGH"},"target":{"severity":"LOW"}, "equals":["tenantId"]}}'

GroupingRule1 = '{"alarm-grouping-definition-created": {"name": "group_rule_1", "matchers": ["hostname"], "id": "b7163", "repeat_interval": "2h", "group_wait": "30s", "tenantId": " d42bc", "alarm_actions": ["cd892"], "ok_actions": ["ad892"], "undetermined_actions": ["cf892"]}}'

Alarm transitions: AT1, AT2, AT3 and AT5

Silence rule:
AT1_severity = HIGH AT1_hostname = host1 (silenced)
AT2_severity = LOW AT2_hostname = host2
AT3_severity = HIGH AT3_hostname = host1(silenced)
AT5_severity = HIGH AT5_hostname = host3

Inhibition rule:
AT1_tenantId = "d42bc"
AT2_tenantId = "d42bc"
AT3_tenantId = "d42bc"
AT5_tenantId = "d42bc"
AT1_state = ALARM
AT2_state = ALARM
AT3_state = OK
AT5_state = UNDETERMINED
AT1_severity = HIGH (source)
AT2_severity = LOW (target)
AT3_severity = HIGH
AT5_severity = HIGH

Grouping rule:
AT1_hostname = host1
AT2_hostname = host2
AT3_hostname = host1
AT5_hostname = host3

Output: AT1 is in "group_notification_rule_1_host1_alarm" group and silenced. AT3 is in "group_notification_rule_1_host1_ok" group and silenced. AT2 is in "group_notification_rule_1_host2_alarm" group and inhibited. AT5 is in "group_notification_rule_1_host3_undetermined" group and will send notification “group_notification_rule_1_host3_undetermined[1]” using undetermined action "cf892".

= Development Environment =

* Monasca DevStack Plugin:
** DevStack is the primary developmement environment for OpenStack.
***See http://docs.openstack.org/developer/devstack/
** The Monasca DevStack plugin installs the Monasca Service, Agent, Horizon Monitoring Panel, and Grafana
** README at, https://github.com/openstack/monasca-api/tree/master/devstack
** Best way to get started is to use Vagrant with the Vagrantfile at, https://github.com/openstack/monasca-api/blob/master/devstack/Vagrantfile.

* Monasca on LaunchPad. https://launchpad.net/monasca

* Monasca projects source code. https://git.openstack.org/cgit/?q=monasca

= Coding Standards =

* Python: All Python code conforms to the OpenStack standards at, http://docs.openstack.org/developer/hacking/.
** Note, all components in Monasca, except for the Threshold Engine, have been ported to Python.

* Java: Several of the Monasca components are available as Java. OpenStack does not have any Java coding standards. We've adopted the Google Java Style at, https://google.github.io/styleguide/javaguide.html.
** The standard says either 80 or 100 length lines. We've adopted 100.

= Technologies =

Monasca uses a number of third-party technologies:

* Apache Kafka (http://kafka.apache.org): Apache Kafka is publish-subscribe messaging rethought as a distributed commit log. Kafka is a highly performant, distributed, fault-tolerant, and scalable message queue with durability built-in.

* Apache Storm (http://storm.incubator.apache.org/): Apache Storm is a free and open source distributed realtime computation system. Storm makes it easy to reliably process unbounded streams of data, doing for realtime processing what Hadoop did for batch processing.

* ZooKeeper (http://zookeeper.apache.org/): Used by Kafka and Storm.

* MySQL: MySQL is supported as a Config Database.

* POSTgres: Support for POSTgres, via Hibernate and SQLAlchemy, for the Config Database.

* Vagrant (http://www.vagrantup.com/): Vagrant provides easy to configure, reproducible, and portable work environments built on top of industry-standard technology and controlled by a single consistent workflow to help maximize the productivity and flexibility of you and your team.

* Dropwizard (https://dropwizard.github.io/dropwizard/): Dropwizard pulls together stable, mature libraries from the Java ecosystem into a simple, light-weight package that lets you focus on getting things done. Dropwizard has out-of-the-box support for sophisticated configuration, application metrics, logging, operational tools, and much more, allowing you and your team to ship a production-quality web service in the shortest time possible.

* InfluxDB (http://influxdb.com/): An open-source distributed time series database with no external dependencies. InfluxDB is supported for the Metrics Database.

* Vertica (http://www.vertica.com): A commercial Enterprise class SQL analytics database that is highly scalable. It offers built-in automatic high-availability and excels at in-database analytics and compressing and storing massive amounts of data. A free community version of Vertica is available that can store up to 1 TB of data with no time-limit is available at, https://my.vertica.com/community/. Vertica is supported for the Metrics Database.

* Cassandra: Support for Cassandra for the Metrics Database is in progress.

= License =

Copyright (c) 2014, 2015 Hewlett-Packard Development Company, L.P.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied.
See the License for the specific language governing permissions and
limitations under the License.

[[File:yklogo.png|200px|thumb|left|Monasca uses YourKit Profiler for Java development]]

 
 
 
 

[https://www.yourkit.com/ Visit YourKit website for more information]

Monasca

2019-05-08T16:34:04Z

Joseph Davis: add train board, logo

== Overview ==

Monasca is a open-source multi-tenant, highly scalable, performant, fault-tolerant monitoring-as-a-service solution that integrates with OpenStack. It uses a REST API for high-speed metrics processing and querying and has a streaming alarm engine and notification engine.

[[File:OpenStack_Project_Monasca_vertical.png|thumb|right]]

==== Contribution ====
* [https://storyboard.openstack.org/#!/project_group/59 Project Group on StoryBoard]
* [https://storyboard.openstack.org/#!/board/111 Important activities in Stein release]
* {https://storyboard.openstack.org/#!/board/141 Important activities in Train release]
* [https://docs.openstack.org/monasca-api/latest/contributor/index.html Contribution Guidelines]

==== Communication and Meetings ====
* IRC: #openstack-monasca on freenode.net
* Weekly Meetings:
** http://eavesdrop.openstack.org/#Monasca_Team_Meeting
** [[Meetings/Monasca]]
* Use [Monasca] tag on [http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-discuss openstack-discuss mailing list].

===== PTG meetings =====
* [[MonascaSteinPTG|Stein PTG Summary (Denver)]]
* (coming soon - Train summary)

==== Documentation ====
: Monasca API Specification: https://github.com/openstack/monasca-api/blob/master/docs/monasca-api-spec.md
: Agent Documentation: https://github.com/openstack/monasca-agent

==== Logo ====
: https://www.openstack.org/themes/openstack/images/project-mascots/Monasca/OpenStack_Project_Monasca_horizontal_1color.eps
: https://www.openstack.org/themes/openstack/images/project-mascots/Monasca/OpenStack_Project_Monasca_horizontal.eps
: https://www.openstack.org/themes/openstack/images/project-mascots/Monasca/OpenStack_Project_Monasca_horizontal.jpg
: https://www.openstack.org/themes/openstack/images/project-mascots/Monasca/OpenStack_Project_Monasca_horizontal.png
: https://www.openstack.org/themes/openstack/images/project-mascots/Monasca/OpenStack_Project_Monasca_mascot.jpg
: https://www.openstack.org/themes/openstack/images/project-mascots/Monasca/OpenStack_Project_Monasca_mascot.png
: https://www.openstack.org/themes/openstack/images/project-mascots/Monasca/OpenStack_Project_Monasca_vertical_1color.eps
: https://www.openstack.org/themes/openstack/images/project-mascots/Monasca/OpenStack_Project_Monasca_vertical.eps
: https://www.openstack.org/themes/openstack/images/project-mascots/Monasca/OpenStack_Project_Monasca_vertical.jpg
: https://www.openstack.org/themes/openstack/images/project-mascots/Monasca/OpenStack_Project_Monasca_vertical.png

==== Presentations ====
There have been many interesting and excellent presentations on Monasca, many given at the OpenStack or OpenInfra Summits.
: [[Monasca/Presentations]] (updated with Denver 2019 links)

==== Repositories ====

: '''Core'''
:: Core: https://git.openstack.org/cgit/?q=monasca

: '''Deployment'''
The following repositories are available for deploying Monasca:
* Docker: https://github.com/monasca/monasca-docker
* Kubernetes: https://github.com/monasca/monasca-helm
* Ansible: https://github.com/search?utf8=%E2%9C%93&q=ansible-monasca
* Puppet: https://git.openstack.org/openstack/puppet-monasca

==== Bugs ====

===== All open bugs =====
: https://storyboard.openstack.org/#!/worklist/213

===== In Stein triaged bugs =====
: https://storyboard.openstack.org/#!/worklist/467

===== Bug-fixing progress =====
: https://storyboard.openstack.org/#!/board/114

== Requirements ==
[[Monasca/Requirements]].

== Features ==

This section describes the overall features.

* A highly performant, scalable, reliable and fault-tolerant Monitoring as a Service (MONaaS) solution that scales to service provider metrics levels of metrics throughput. Performance, scalability and high-availability have been designed in from the start. Can process 100s of thousands of metrics/sec as well as offer data retention periods of greater than a year with no data loss while still processing interactive queries.

* Rest API for storing and querying metrics and historical information. Most monitoring solution use special transports and protocols, such as CollectD or NSCA (Nagios). In our solution, http is the only protocol used. This simplifies the overall design and also allows for a much richer way of describing the data via dimensions.

* Multi-tenant and authenticated. Metrics are submitted and authenticated using Keystone and stored associated with a tenant ID.

* Metrics defined using a set of (key, value) pairs called dimensions.

* Real-time thresholding and alarming on metrics.

* Compound alarms described using a simple expressive grammar composed of alarm sub-expressions and logical operators.

* Monitoring agent that supports a number of built-in system and service checks and also supports Nagios checks and statsd.

* Open-source monitoring solution built on open-source technologies.

== Comparisons to alternatives ==
[[Monasca/Comparison]]

== Architecture ==
[[File:monasca-arch-component-diagram.png|Monasca Architecture Component Diagram]]
* Monitoring Agent (monasca-agent): A modern Python based monitoring agent that consists of several sub-components and supports system metrics, such as cpu utilization and available memory, Nagios plugins, statsd and many built-in checks for services such as MySQL, RabbitMQ, and many others.
* Monitoring API (monasca-api): A well-defined and documented RESTful API for monitoring that is primarily focused on the following concepts and areas:
** Metrics: Store and query massive amounts of metrics in real-time.
** Statistics: Query statistics for metrics.
** Alarm Definitions: Create, update, query and delete alarm definitions.
** Alarms: Query and delete the alarm history.
***Simple expressive grammar for creating compound alarms composed of alarm subexpressions and logical operators.
***Alarm severities can be associated with alarms.
***The complete alarm state transition history is stored and queryable which allows for subsequent root cause analysis (RCA) or advanced analytics.
**Notification Methods: Create and delete notification methods and associate them with alarms, such as email. Supports the ability to notify users directly via email when an alarm state transitions occur.
** The Monasca API has both Java and Python implementations avaialble.
* Persister (monasca-persister): Consumes metrics and alarm state transitions from the MessageQ and stores them in the Metrics and Alarms database.
** The Persister has both Java and Python implementations.
* Transform and Aggregation Engine (monasca-transform): Transform metric names and values, such as delta or time-based derivative calculations, and creates new metrics that are published to the Message Queue. The Transform Engine is not available yet.
* Anomaly and Prediction Engine: Evaluates prediction and anomalies and generates predicted metrics as well as anomaly likelihood and anomaly scores. The Anomaly and Prediction Engine is currently in a prototype status.
* Threshold Engine (monasca-thresh): Computes thresholds on metrics and publishes alarms to the MessageQ when exceeded. Based on Apache Storm a free and open distributed real-time computation system.
* Notification Engine (monasca-notification): Consumes alarm state transition messages from the MessageQ and sends notifications, such as emails for alarms. The Notification Engine is Python based.
* Analytics Engine (monasca-analytics): Consumes alarm state transisitions and metrics from the MessageQ and does anomaly detection and alarm clustering/correlation.
* Message Queue: A third-party component that primarily receives published metrics from the Monitoring API and alarm state transition messages from the Threshold Engine that are consumed by other components, such as the Persister and Notification Engine. The Message Queue is also used to publish and consume other events in the system. Currently, a Kafka based MessageQ is supported. Kafka is a high performance, distributed, fault-tolerant, and scalable message queue with durability built-in. We will look at other alternatives, such as RabbitMQ and in-fact in our previous implementation RabbitMQ was supported, but due to performance, scale, durability and high-availability limitiations with RabbitMQ we have moved to Kafka.
* Metrics and Alarms Database: A third-party component that primarily stores metrics and the alarm state history. Currently, Vertica and InfluxDB are supported. Support for Cassandra is in progress.
* Config Database: A third-party component that stores a lot of the configuration and other information in the system. Currently, MySQL is supported. Support for PostgreSQL is in progress.
* Monitoring Client (python-monascaclient): A Python command line client and library that communicates and controls the Monitoring API. The Monitoring Client was written using the OpenStack Heat Python client as a framework. The Monitoring Client also has a Python library, "monascaclient" similar to the other OpenStack clients, that can be used to quickly build additional capabilities. The Monitoring Client library is used by the Monitoring UI, Ceilometer publisher, and other components.
* Monitoring UI: A Horizon dashboard for visualizing the overall health and status of an OpenStack cloud.
* Ceilometer publisher: A multi-publisher plugin for Ceilometer, not shown, that converts and publishes samples to the Monitoring API.
Most of the components are described in their respective repositories. However, there aren't any repositories for the third-party components used, so we describe some of the relevant details here.

=== Message Schema ===
[[Monasca/Message Schema]]

=== Message Queue ===
A distributed, performant, scalable, HA message queue for distributing metrics, alarms and events in the monitoring system. Currently, based on Kafka.

==== Messages ====
There are several messages that are published and consumed by various components in Monasca via the MessageQ. See [[Monasca/Message Schema|Message Schema]].

=== Metrics and Alarms Database ===

A high-performance analytics database that can store massive amounts of metrics and alarms in real-time and also support interactive queries. Currently Vertica and InfluxDB are supported.

The SQL schema that is used by Vertica is as follows:

* MonMetrics.Measurements: Stores the actual measurements that are sent.
** id: An integer ID for the measurement.
** definition_dimensions_id: A reference to DefinitionDimensions.
** time_stamp
** value
* MonMetrics.DefinitionDimensions
** id: A sha1 hash of (defintion_id, dimension_set_id)
** definition_id: A reference to the Definitions.id
** dimension_set_id: A reference to the Dimensions.dimension_set_id
* MonMetrics.Definitions
** id: A sha1 hash of the (name, tenant_id, region)
** name: Name of the metric.
** tenant_id: The tenant_id that submitted the metric.
** region: The region the metric was submitted under.
* MonMetric.Dimensions
** dimension_set_id: A sha1 hash of the set of dimenions for a metric.
** name: Name of dimension.
** value: Value of dimension.

=== Config Database ===
The config database store all the configuration information. Currently based on MySQL.

The SQL schema is as follows:

* alarm
** id
** tenant_id
** name
** description
** expression
** state
** actions_enabled
** created_at
** updated_at
** deleted_at
* alarm_action
** alarm_id
** alarm_state
** action_id
* notification_method
** id
** tenant_id
** name
** type
** address
** created_at
** updated_at
* sub_alarm
** id
** alarm_id
** function
** metric_name
** operator
** threshold
** period
** periods
** state
** created_at
** updated_at
* sub_alarm_dimension
** sub_alarm_id
** dimension_name
** value

== Events ==
Support for real-time event stream processing in Monasca is in progress. For more details see the link at, [[Monasca/Events]].

== Logging ==
Support for logging in Monasca is under discussion. For more details see the link at, [[Monasca/Logging]].

== Transform and Aggregation Engine ==
For more details see the link at, [[Monasca/Transform]].

== Analytics ==
Support for anomaly detection and alarm clustering/correlation is in progress. For more details see the link at, [[Monasca/Analytics]].

== Monitoring ==
Enablement and usage for monitoring the status of Monasca is under discussion. For more details see the link at, [[Monasca/Monitoring_Of_Monasca]]

== UI/UX Support ==
Adding more support for common UI/UX queries is under discussion. For more details see the link at, [[Monasca/UI_UX_Support]]

== Keystone Requirements ==
Monasca relies on keystone for running and there are requirements about which keystone configuration must exist.

* The endpoint for the api must be registered in keystone as the 'monasca' service.
* The api must have an admin token to use in verifying the keystone tokens it receives.
* For each project which uses Monasca two users must exist, one will be in the 'monasca-agent' role and be used by the monasca-agent's running on machines. The other should not be in that role and can be used logging into the UI, using the CLI or for direct queries against the API.

== Post Metric Sequence ==

This section describes the sequence of operations involved in posting a metric to the Monasca API.

[[File:monasca-arch-post-metric-diagram.png|Monasca Architecture Post Metric Diagram]]

# A metric is posted to the Monasca API.
# The Monasca API authenticates and validates the request and publishes the metric to the the Message Queue.
# The Persister consumes the metric from the Message Queue and stores in the Metrics Store.
# The Transform Engine consumes the metrics from the Message Queue, performs transform and aggregation operations on metrics, and publishes metrics that it creates back to Message Queue.
# The Threshold Engine consumes metrics from the Message Queue and evaluates alarms. If a state change occurs in an alarm, an "alarm-state-transitioned-event" is published to the Message Queue.
# The Notification Engine consumes "alarm-state-transitioned-events" from the Message Queue, evaluates whether they have a Notification Method associated with it, and sends the appropriate notification, such as email.
# The Persister consumes the "alarm-state-transitioned-event" from the Message Queue and stores it in the Alarm State History Store.

== Alarm Managers ==
This section describes the new features including alarm grouping, inhibition and silencing.

Examples:

* Silence
SilenceRule1 = '{"alarm-silencing-definition-created": {"name": "silence_rule_1", "matchers": {"severity": "LOW"}, "start_time": "2017-02-21 20:00:00", "end_time": "2017-02-21 22:00:00"}}'
Two alarm transitions: AT1 and AT2
AT1_severity = HIGH
AT2_severity = LOW

Output: AT2 get silenced and AT1 send a notification.
SilenceRule1 expires after "end_time"="2017-02-21 22:00:00".

* Inhibit
InhibitionRule1 = '{"alarm-inhibition-definition-created": {"name": "inhibit_rule_1","source":{"severity":"HIGH"},"target":{"severity":"LOW"}, "equals":["tenantId"], "exclusions": {"alarm_name": "vm is dead"}}}'
Three alarm transitions: AT1, AT2 and AT3
AT1_tenantId = "d42bc"
AT2_tenantId = "d42bc"
AT3_tenantId = "d42bc"
AT1_severity = HIGH
AT2_severity = LOW
AT3_severity = LOW
AT1_alarm_name = "cpu high"
AT2_alarm_name = "memory high"
AT3_alarm_name = "vm is dead"
AT1_state = ALARM
AT2_state = ALARM
AT3_state = ALARM

Output: AT1 is the source alarm which will send a notification. AT2 is the target alarm and will get inhibited. AT3 matches the exclusions and will send a notification immediately.

* Grouping Scenario 1

GroupingRule1 = '{"alarm-grouping-definition-created":
{"name": "group_rule_1", "matchers": ["hostname"], "id": "b7163","repeat_interval": "2h", "group_wait": "30s", "exclusions": {"alarm_name": "cpu_percent_high"}, "tenantId": " d42bc", "alarm_actions": ["cd892"], "ok_actions": ["ad892"], "undetermined_actions": ["cf892"]}}'

Three alarm transitions: AT1, AT2 and AT3

AT1_hostname = host1
AT2_hostname = host1
AT3_hostname = host2
AT1_alarm_name = cpu_percent_high
AT2_alarm_name = cpu_system_perc_high
AT3_alarm_name = cpu_percent_high
AT1_state = ALARM
AT2_state = ALARM
AT3_state = ALARM

Output: AT1 and AT3 match exclusions and send notifications immediately. Generate a grouped notification “group_notification_rule_1_host1_alarm[1]” and send out using alarm_actions ["cd892"]. There are no alarm_actions, ok_actions or undermined_actions associated with the AT1, AT2, AT3 alarm definitions.

* Grouping Scenario 2

GroupingRule2 = '{"alarm-grouping-definition-created":
{"name": "group_rule_2", "matchers": ["hostname"], "id": "b7163","repeat_interval": "2h", "group_wait": "30s", "tenantId": " d42bc", "alarm_actions": ["cd892"], "ok_actions": ["ad892"], "undetermined_actions": ["cf892"]}}'

Three alarm transitions: AT1 with "alarm_actions": ["123ab"], AT2 with "alarm_actions": ["cd839"] and AT3

AT1_hostname = host1
AT2_hostname = host1
AT3_hostname = host2
AT1_state = ALARM
AT2_state = ALARM
AT3_state = ALARM

Output: Generate two grouped notifications “group_notification_rule_2_host1_alarm[2]” and “group_notification_rule_2_host2_alarm[1]”. Both using alarm_actions ["cd892"]. Also since AT1 and AT2 has their own alarm actions associated with them, there will be two more notifications sent out.

* Silenced and Grouped

SilenceRule1 = '{"alarm-silencing-definition-created": {"name": "silence_rule_1", "matchers": {"severity": "LOW"}, "start_time": "1487269470498", "end_time": "1587269470498"}}'

GroupingRule2 = '{"alarm-grouping-definition-created":
{"name": "group_rule_2", "matchers": ["hostname"], "id": "b7163","repeat_interval": "2h", "group_wait": "30s", "tenantId": " d42bc", "alarm_actions": ["cd892"], "ok_actions": ["ad892"], "undetermined_actions": ["cf892"]}}'

Four alarm transitions: AT1, AT2, AT3 and AT4

Silencing rule:
AT1_severity = HIGH
AT2_severity = LOW
AT3_severity = HIGH
AT4_severity = HIGH

Grouping rule:
AT1_hostname = host1
AT2_hostname = host1
AT3_hostname = host2
AT4_hostname = host1
AT1_state = ALARM
AT2_state = ALARM
AT3_state = OK
AT4_state = ALARM

Output: Generate two grouped notifications “group_notification_rule_2_host1_alarm[2]” using alarm action "cd892" and “group_notification_rule_2_host2_ok[1]” using ok action "ad892". AT2 is silenced so it isn't include in group_notification_rule_2_host1 count. There are no alarm_actions, ok_actions or undermined_actions associated with the AT1, AT2, AT3, AT4 alarm definitions.

* Silenced and inhibited (source alarm get silenced)

SilenceRule2 = '{"alarm-silencing-definition-created": {"name": "silence_rule_2", "matchers": {"severity": "HIGH, "hostname": "host1"}, "start_time": "2017-02-21 15:00:00", "end_time": "2017-02-21 21:00:00"}}'

InhibitionRule1 = '{"alarm-inhibition-definition-created": {"name": "inhibit_rule_1","source":{"severity":"HIGH"},"target":{"severity":"LOW"}, "equals":["tenantId"]}}'

Alarm transitions: AT1, AT2

Inhibition rule:
AT1_state = ALARM
AT2_state = ALARM
AT1_tenantId = "d42bc"
AT2_tenantId = "d42bc"
AT1_severity = HIGH
AT2_severity = LOW

Silencing rule:
AT1_hostname = host1
AT2_hostname = host2
AT1_severity = HIGH
AT2_severity = LOW

Output: no notification sent out. For inhibition, AT1 is the source alarm, AT2 is the target alarm. But at the same time, AT1 get silenced because it matches the silence rule.

* Inhibited and grouped

InhibitionRule1 = '{"alarm-inhibition-definition-created": {"name": "inhibit_rule_1","source":{"severity":"HIGH"},"target":{"severity":"LOW"}, "equals":["tenantId"]}}'

GroupingRule1 = '{"alarm-grouping-definition-created": {"name": "group_rule_1", "matchers": ["hostname"], "id": "b7163", "repeat_interval": "2h", "group_wait": "30s", "tenantId": " d42bc", "alarm_actions": ["cd892"], "ok_actions": ["ad892"], "undetermined_actions": ["cf892"]}}'

Alarm transitions: AT1, AT2, AT3

Inhibition rule:
AT1_tenantId = "d42bc"
AT2_tenantId = "d42bc"
AT3_tenantId = "d42bc"
AT1_severity = HIGH
AT2_severity = LOW
AT3_severity = HIGH
AT1_state = OK
AT2_state = ALARM
AT3_state = ALARM

Grouping rule:
AT1_hostname = host1
AT2_hostname = host2
AT3_hostname = host1
AT1_state = OK
AT2_state = ALARM
AT3_state = ALARM

Output: AT2 gets inhibited because its severity is low. AT3 is the source alarm. Since AT1 is in OK state, it is not a source alarm. For grouping, AT1 and AT3 has the same host name but different state. So there will be two grouped notifications sent out: “group_notification_rule_1_host1_ok[1]” and “group_notification_rule_1_host1_alarm[1]”.

* Silenced, inhibited and grouped

SilenceRule2 = '{"alarm-silencing-definition-created": {"name": "silence_rule_2", "matchers": {"severity": "HIGH, "hostname": "host1"}, "start_time": "2017-02-21 15:00:00", "end_time": "2017-02-21 21:00:00"}}'

InhibitionRule1 = '{"alarm-inhibition-definition-created": {"name": "inhibit_rule_1","source":{"severity":"HIGH"},"target":{"severity":"LOW"}, "equals":["tenantId"]}}'

GroupingRule1 = '{"alarm-grouping-definition-created": {"name": "group_rule_1", "matchers": ["hostname"], "id": "b7163", "repeat_interval": "2h", "group_wait": "30s", "tenantId": " d42bc", "alarm_actions": ["cd892"], "ok_actions": ["ad892"], "undetermined_actions": ["cf892"]}}'

Alarm transitions: AT1, AT2, AT3 and AT5

Silence rule:
AT1_severity = HIGH AT1_hostname = host1 (silenced)
AT2_severity = LOW AT2_hostname = host2
AT3_severity = HIGH AT3_hostname = host1(silenced)
AT5_severity = HIGH AT5_hostname = host3

Inhibition rule:
AT1_tenantId = "d42bc"
AT2_tenantId = "d42bc"
AT3_tenantId = "d42bc"
AT5_tenantId = "d42bc"
AT1_state = ALARM
AT2_state = ALARM
AT3_state = OK
AT5_state = UNDETERMINED
AT1_severity = HIGH (source)
AT2_severity = LOW (target)
AT3_severity = HIGH
AT5_severity = HIGH

Grouping rule:
AT1_hostname = host1
AT2_hostname = host2
AT3_hostname = host1
AT5_hostname = host3

Output: AT1 is in "group_notification_rule_1_host1_alarm" group and silenced. AT3 is in "group_notification_rule_1_host1_ok" group and silenced. AT2 is in "group_notification_rule_1_host2_alarm" group and inhibited. AT5 is in "group_notification_rule_1_host3_undetermined" group and will send notification “group_notification_rule_1_host3_undetermined[1]” using undetermined action "cf892".

= Development Environment =

* Monasca DevStack Plugin:
** DevStack is the primary developmement environment for OpenStack.
***See http://docs.openstack.org/developer/devstack/
** The Monasca DevStack plugin installs the Monasca Service, Agent, Horizon Monitoring Panel, and Grafana
** README at, https://github.com/openstack/monasca-api/tree/master/devstack
** Best way to get started is to use Vagrant with the Vagrantfile at, https://github.com/openstack/monasca-api/blob/master/devstack/Vagrantfile.

* Monasca on LaunchPad. https://launchpad.net/monasca

* Monasca projects source code. https://git.openstack.org/cgit/?q=monasca

= Coding Standards =

* Python: All Python code conforms to the OpenStack standards at, http://docs.openstack.org/developer/hacking/.
** Note, all components in Monasca, except for the Threshold Engine, have been ported to Python.

* Java: Several of the Monasca components are available as Java. OpenStack does not have any Java coding standards. We've adopted the Google Java Style at, https://google.github.io/styleguide/javaguide.html.
** The standard says either 80 or 100 length lines. We've adopted 100.

= Technologies =

Monasca uses a number of third-party technologies:

* Apache Kafka (http://kafka.apache.org): Apache Kafka is publish-subscribe messaging rethought as a distributed commit log. Kafka is a highly performant, distributed, fault-tolerant, and scalable message queue with durability built-in.

* Apache Storm (http://storm.incubator.apache.org/): Apache Storm is a free and open source distributed realtime computation system. Storm makes it easy to reliably process unbounded streams of data, doing for realtime processing what Hadoop did for batch processing.

* ZooKeeper (http://zookeeper.apache.org/): Used by Kafka and Storm.

* MySQL: MySQL is supported as a Config Database.

* POSTgres: Support for POSTgres, via Hibernate and SQLAlchemy, for the Config Database.

* Vagrant (http://www.vagrantup.com/): Vagrant provides easy to configure, reproducible, and portable work environments built on top of industry-standard technology and controlled by a single consistent workflow to help maximize the productivity and flexibility of you and your team.

* Dropwizard (https://dropwizard.github.io/dropwizard/): Dropwizard pulls together stable, mature libraries from the Java ecosystem into a simple, light-weight package that lets you focus on getting things done. Dropwizard has out-of-the-box support for sophisticated configuration, application metrics, logging, operational tools, and much more, allowing you and your team to ship a production-quality web service in the shortest time possible.

* InfluxDB (http://influxdb.com/): An open-source distributed time series database with no external dependencies. InfluxDB is supported for the Metrics Database.

* Vertica (http://www.vertica.com): A commercial Enterprise class SQL analytics database that is highly scalable. It offers built-in automatic high-availability and excels at in-database analytics and compressing and storing massive amounts of data. A free community version of Vertica is available that can store up to 1 TB of data with no time-limit is available at, https://my.vertica.com/community/. Vertica is supported for the Metrics Database.

* Cassandra: Support for Cassandra for the Metrics Database is in progress.

= License =

Copyright (c) 2014, 2015 Hewlett-Packard Development Company, L.P.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied.
See the License for the specific language governing permissions and
limitations under the License.

[[File:yklogo.png|200px|thumb|left|Monasca uses YourKit Profiler for Java development]]

 
 
 
 

[https://www.yourkit.com/ Visit YourKit website for more information]

File:OpenStack Project Monasca vertical.png

2019-05-08T16:33:02Z

Joseph Davis: vertical Logo

vertical Logo

Monasca/Events

2019-05-08T16:21:18Z

Joseph Davis: /* News */

== News ==
<big>July 2018</big> - Work is continuing on Monasca Events with a revision to the architecture and API. Below information is from before Jan 2017 and back to 2015, but look for upcoming updates as work progresses.

------

== Introduction ==
Real-time event stream processing in Monasca is work in progress that will allow events from external data sources to be sent to the Monasca API where they can be transformed, stored, queried, filtered, grouped and associated with handlers. Streams can be defined by filtering and grouping events using fields in the event. Handlers can be associated with stream definitions that fire when the stream trigger occurs.

Example of OpenStack events are: instance creation and instance delete. Specific action can follow these events. For example if an instance is deleted then all the alarms related to this instance should be deleted too.

An example use case is to send OpenStack "compute.instance.create.*" events (see https://wiki.openstack.org/wiki/NotificationEventExamples) to the API. A transform on the events could be defined that reduces the number of supplied fields in the source event to a more manageable number of fields as well as normalizing the data. An event stream can be creating by defining a filter to select all "compute.instance.create.*" events and group them by a set of fields in the event, such as "instance_id". When the "compute.instance.create.end" event occurs a fire criteria can be invoked that processes the stream.

== Events API ==
=== Events ===
* POST /v2.0/events: Publish an event.
* GET /v2.0/events/{event_id}: Get an event with the specific event ID.
* GET /v2.0/events: List events.
=== Transforms ===
* POST /v2.0/events/transforms - Create a transform
* GET /v2.0/events/transforms - List transforms
* GET /v2.0/events/transforms/{transform_id} - Get the specified transform
* DELETE /v2.0/events/transforms/{transform_id}

=== Stream Definition ===
* POST /v2.0/events/stream-definitions: Creates a stream definition with the following parameters in the JSON body
** name (string(255), required) - A unique name for the stream.
** description (string(255), optional) - A description of the stream.
** select () - Fields of events to filter/match on. For example select all events where the field "event_type" matches "compute.instance.create.*".
** group_by () - Fields of events to group on. For all events that match the select criteria group them by the specified criteria. For example, group events by the field "instance_id" or "user_id".
** expires (int, required) - Elapsed time in milliseconds from the start of a stream to when the expire actions are invoked if the fire actions haven't occurred yet.
** fire_actions ([string(50)], optional) - Array of notification method IDs that are invoked when the pipeline fires.
** expire_actions ([string(50)], optional) - Array of notification method IDs that are invoked when the pipeline expires.
* GET /v2.0/events/stream-definitions
* GET /v2.0/events/stream-definitions/{stream-definition-id}
* DELETE /v2.0/events/stream-definitions/{stream-definition-id}

== Transformation Engine ==
Consumes events from Kafka, transforms them, and publishes to Kafka.

== Event Engine ==
Consumes transformed events from Kafka, and uses the Winchester pipeline to process them.

== Distiller ==
No changes required.

== Winchester ==
* Add Support for multi-tenancy
* Dynamically update pipelines.
* Add and delete pipeline definitions at run-time. Currently, the Winchester pipelines needs to be created at start-up time.
** Supply pipeline definitions in methods, not yaml files. Winchester currently reads the pipeline configuration information from yaml files at start-up time.
* Create pipeline handler that publishes notification events such that the Notification Engine can consume them.

== Threshold Engine ==
Update to generate more general alarm state transition events.

== MySQL ==
* Initialize Winchester schemas
* Initialize Monasca transforms and pipeline schemas.

Monasca/Presentations

2019-05-08T16:17:01Z

Joseph Davis: Denver 2019

: '''Denver Summit 2019'''
:: Scaling should be easy and automatic: Heat and Monasca
::: Workshop: https://www.openstack.org/summit/denver-2019/summit-schedule/events/23291/scaling-should-be-easy-and-automatic-heat-and-monasca
:: Monasca Project Update
::: https://www.openstack.org/summit/denver-2019/summit-schedule/events/23646/monasca-openstack-monitoring
::Monitoring and Analyzing your OpenStack Cloud
::: Workshop: https://www.openstack.org/summit/denver-2019/summit-schedule/events/23316/monitoring-and-analyzing-your-openstack-cloud
:: Monasca and 25,000+ Instances in Production - User Story
::: https://www.openstack.org/summit/denver-2019/summit-schedule/events/23332/monasca-and-25000-instances-in-production-user-story
: '''Berlin Summit 2018'''
:: Monasca Project Update
::: Video: https://www.openstack.org/videos/summits/berlin-2018/monasca-openstack-monitoring
:: Monitoring as a Service in the HPC Cloud
::: Video: https://www.openstack.org/videos/summits/berlin-2018/monitoring-as-a-service-in-the-hpc-cloud
: '''Sydney Summit 2017'''
:: Monasca Project Update
::: Video: https://www.openstack.org/videos/sydney-2017/monasca-project-update
:: Monitoring Hands-On Workshop
::: Material: https://github.com/witekest/monasca-bootcamp/
:: Use case: Help me from tons of alarms
::: https://www.openstack.org/videos/sydney-2017/use-case-help-me-from-tons-of-alarms
:: Proactive Monitoring for OpenStack
::: Video: https://www.openstack.org/videos/sydney-2017/proactive-monitoring-for-openstack
: '''Boston Summit 2017'''
:: Making Monasca Monitor More: Extending Monasca's Data Gathering and Reporting Capabilities
::: Video: https://www.openstack.org/videos/boston-2017/making-monasca-monitor-more-extending-monascas-data-gathering-and-reporting-capabilities
:: Monitoring Hands-On Workshop
::: Material: https://github.com/witekest/monasca-bootcamp/
:: Show Me My Packet Log: Neutron Packet Logging with Monasca
::: Video: https://www.openstack.org/videos/boston-2017/show-me-my-packet-log-neutron-packet-logging-with-monasca
:: A Monitoring Case Study for Monasca: Smart City Infrastructure
::: Video: https://www.openstack.org/videos/boston-2017/a-monitoring-case-study-for-monasca-smart-city-infrastructure
: '''Barcelona Summit 2016'''
:: Monasca: one year later
::: Video: https://www.openstack.org/videos/barcelona-2016/monasca-one-year-later
:: Fujitsu: Monasca Monitoring for Kubernetes – Delivered on OpenStack
::: Video: https://www.openstack.org/videos/barcelona-2016/fujitsu-monasca-monitoring-for-kubernetes-delivered-on-openstack
: '''Austin Summit 2016'''
:: Monasca Bootcamp:Hands-on Workshops
::: Video: https://www.openstack.org/videos/video/monasca-bootcamp
:: Log Management with Monasca
::: Video: https://www.openstack.org/videos/video/log-management-with-monasca
::: Slides: https://github.com/witekest/presentations/blob/master/OpenStack_Summit/2016_04_Austin/Monasca_Logging_OpenStack_Summit_Austin.pdf
:: Enforcing Application SLAs with Congress and Monasca
::: Video: https://www.openstack.org/videos/video/enforcing-application-slas-with-congress-and-monasca
:: Only You Can Prevent Forest Fires - A Proactive Approach to Monitoring Your OpenStack Cloud
::: https://www.openstack.org/videos/video/only-you-can-prevent-forest-fires-a-proactive-approach-to-monitoring-your-openstack-cloud
:: Monitoring a Multi-Region Cloud Based on OpenStack: The FIWARE Lab Case Study
::: Video: https://www.openstack.org/videos/video/monitoring-a-multi-region-cloud-based-on-openstack-the-fiware-lab-case-study
: '''Tokyo Summit 2015'''
:: Congrats You Stood up an OpenStack Environment, But Now They Want You to Monitor It. Introducing Using Monasca for Production OpenStack Monitoring
::: Video: https://www.openstack.org/summit/tokyo-2015/videos/presentation/congrats-you-stood-up-an-openstack-environment-but-now-they-want-you-to-monitor-it-introducing-using-monasca-for-production-openstack-monitoring
:: Ceilometer+Monasca=Ceilosca
::: Video: https://www.openstack.org/summit/tokyo-2015/videos/presentation/ceilometer-monascaceilosca
:: ELK and Monasca Crossing: Logging as an OpenStack Service
::: Video: https://www.openstack.org/summit/tokyo-2015/videos/presentation/elk-and-monasca-crossing-logging-as-an-openstack-service
::: Slides: http://www.slideshare.net/mroderus/elk-and-monasca-crossing-logging-as-an-openstack-service
:: Auto Scaling Cloud Infrastructure and Applications
::: Video: https://www.openstack.org/summit/tokyo-2015/videos/presentation/auto-scaling-cloud-infrastructure-and-applications

: '''Monasca Deep Dive (Paris Summit)'''
:: Video: https://www.openstack.org/summit/openstack-paris-summit-2014/session-videos/presentation/monasca-deep-dive-monitoring-at-scale
:: Slides: https://www.openstack.org/assets/presentation-media/Monasca-Deep-Dive-Paris-Summit.pdf

: '''Colorado OpenStack 5th Birthday Monasca Operations'''
:: Slides: http://www.slideshare.net/dlfryar/colorado-openstack-5th-birthday-monasca-operations
:: MeetUp: http://www.meetup.com/OpenStack-Colorado/events/223495998/?a=ea1_grp&rv=ea1

: '''Austin OpenStack OpenStack Monasca Architecture'''
:: Github: https://github.com/dlfryar/monasca-meetups/tree/master/2015/07/austin/openstack
:: Slides: https://github.com/dlfryar/monasca-meetups/blob/master/2015/07/austin/openstack/meetup-openstack-austin-2015.pdf
:: MeetUp: http://www.meetup.com/OpenStack-Austin/events/223068769/

: '''SFBay OpenStack Advanced Track #OSSFO Topic: Monasca and Heat'''
:: Github: https://github.com/dlfryar/monasca-meetups/tree/master/2015/08/sunnyvale/openstack
:: Slides: https://github.com/dlfryar/monasca-meetups/blob/master/2015/08/sunnyvale/openstack/meetup-openstack-sfbay-2015.pdf
:: MeetUp: http://www.meetup.com/openstack/events/215648142/?a=cr2_grp&rv=cr2&_af=event&_af_eid=215648142

Monasca

2019-05-08T16:11:42Z

Joseph Davis: /* Presentations */

== Overview ==

Monasca is a open-source multi-tenant, highly scalable, performant, fault-tolerant monitoring-as-a-service solution that integrates with OpenStack. It uses a REST API for high-speed metrics processing and querying and has a streaming alarm engine and notification engine.

==== Contribution ====
* [https://storyboard.openstack.org/#!/project_group/59 Project Group on StoryBoard]
* [https://storyboard.openstack.org/#!/board/111 Important activities in Stein release]
* [https://docs.openstack.org/monasca-api/latest/contributor/index.html Contribution Guidelines]

==== Communication and Meetings ====
* IRC: #openstack-monasca on freenode.net
* Weekly Meetings:
** http://eavesdrop.openstack.org/#Monasca_Team_Meeting
** [[Meetings/Monasca]]
* Use [Monasca] tag on [http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-discuss openstack-discuss mailing list].
===== PTG meetings =====
* [[MonascaSteinPTG|Stein PTG Summary (Denver)]]

==== Documentation ====
: Monasca API Specification: https://github.com/openstack/monasca-api/blob/master/docs/monasca-api-spec.md
: Agent Documentation: https://github.com/openstack/monasca-agent

==== Logo ====
: https://www.openstack.org/themes/openstack/images/project-mascots/Monasca/OpenStack_Project_Monasca_horizontal_1color.eps
: https://www.openstack.org/themes/openstack/images/project-mascots/Monasca/OpenStack_Project_Monasca_horizontal.eps
: https://www.openstack.org/themes/openstack/images/project-mascots/Monasca/OpenStack_Project_Monasca_horizontal.jpg
: https://www.openstack.org/themes/openstack/images/project-mascots/Monasca/OpenStack_Project_Monasca_horizontal.png
: https://www.openstack.org/themes/openstack/images/project-mascots/Monasca/OpenStack_Project_Monasca_mascot.jpg
: https://www.openstack.org/themes/openstack/images/project-mascots/Monasca/OpenStack_Project_Monasca_mascot.png
: https://www.openstack.org/themes/openstack/images/project-mascots/Monasca/OpenStack_Project_Monasca_vertical_1color.eps
: https://www.openstack.org/themes/openstack/images/project-mascots/Monasca/OpenStack_Project_Monasca_vertical.eps
: https://www.openstack.org/themes/openstack/images/project-mascots/Monasca/OpenStack_Project_Monasca_vertical.jpg
: https://www.openstack.org/themes/openstack/images/project-mascots/Monasca/OpenStack_Project_Monasca_vertical.png

==== Presentations ====
There have been many interesting and excellent presentations on Monasca, many given at the OpenStack or OpenInfra Summits.
: [[Monasca/Presentations]] (updated with Denver 2019 links)

==== Repositories ====

: '''Core'''
:: Core: https://git.openstack.org/cgit/?q=monasca

: '''Deployment'''
The following repositories are available for deploying Monasca:
* Docker: https://github.com/monasca/monasca-docker
* Kubernetes: https://github.com/monasca/monasca-helm
* Ansible: https://github.com/search?utf8=%E2%9C%93&q=ansible-monasca
* Puppet: https://git.openstack.org/openstack/puppet-monasca

==== Bugs ====

===== All open bugs =====
: https://storyboard.openstack.org/#!/worklist/213

===== In Stein triaged bugs =====
: https://storyboard.openstack.org/#!/worklist/467

===== Bug-fixing progress =====
: https://storyboard.openstack.org/#!/board/114

== Requirements ==
[[Monasca/Requirements]].

== Features ==

This section describes the overall features.

* A highly performant, scalable, reliable and fault-tolerant Monitoring as a Service (MONaaS) solution that scales to service provider metrics levels of metrics throughput. Performance, scalability and high-availability have been designed in from the start. Can process 100s of thousands of metrics/sec as well as offer data retention periods of greater than a year with no data loss while still processing interactive queries.

* Rest API for storing and querying metrics and historical information. Most monitoring solution use special transports and protocols, such as CollectD or NSCA (Nagios). In our solution, http is the only protocol used. This simplifies the overall design and also allows for a much richer way of describing the data via dimensions.

* Multi-tenant and authenticated. Metrics are submitted and authenticated using Keystone and stored associated with a tenant ID.

* Metrics defined using a set of (key, value) pairs called dimensions.

* Real-time thresholding and alarming on metrics.

* Compound alarms described using a simple expressive grammar composed of alarm sub-expressions and logical operators.

* Monitoring agent that supports a number of built-in system and service checks and also supports Nagios checks and statsd.

* Open-source monitoring solution built on open-source technologies.

== Comparisons to alternatives ==
[[Monasca/Comparison]]

== Architecture ==
[[File:monasca-arch-component-diagram.png|Monasca Architecture Component Diagram]]
* Monitoring Agent (monasca-agent): A modern Python based monitoring agent that consists of several sub-components and supports system metrics, such as cpu utilization and available memory, Nagios plugins, statsd and many built-in checks for services such as MySQL, RabbitMQ, and many others.
* Monitoring API (monasca-api): A well-defined and documented RESTful API for monitoring that is primarily focused on the following concepts and areas:
** Metrics: Store and query massive amounts of metrics in real-time.
** Statistics: Query statistics for metrics.
** Alarm Definitions: Create, update, query and delete alarm definitions.
** Alarms: Query and delete the alarm history.
***Simple expressive grammar for creating compound alarms composed of alarm subexpressions and logical operators.
***Alarm severities can be associated with alarms.
***The complete alarm state transition history is stored and queryable which allows for subsequent root cause analysis (RCA) or advanced analytics.
**Notification Methods: Create and delete notification methods and associate them with alarms, such as email. Supports the ability to notify users directly via email when an alarm state transitions occur.
** The Monasca API has both Java and Python implementations avaialble.
* Persister (monasca-persister): Consumes metrics and alarm state transitions from the MessageQ and stores them in the Metrics and Alarms database.
** The Persister has both Java and Python implementations.
* Transform and Aggregation Engine (monasca-transform): Transform metric names and values, such as delta or time-based derivative calculations, and creates new metrics that are published to the Message Queue. The Transform Engine is not available yet.
* Anomaly and Prediction Engine: Evaluates prediction and anomalies and generates predicted metrics as well as anomaly likelihood and anomaly scores. The Anomaly and Prediction Engine is currently in a prototype status.
* Threshold Engine (monasca-thresh): Computes thresholds on metrics and publishes alarms to the MessageQ when exceeded. Based on Apache Storm a free and open distributed real-time computation system.
* Notification Engine (monasca-notification): Consumes alarm state transition messages from the MessageQ and sends notifications, such as emails for alarms. The Notification Engine is Python based.
* Analytics Engine (monasca-analytics): Consumes alarm state transisitions and metrics from the MessageQ and does anomaly detection and alarm clustering/correlation.
* Message Queue: A third-party component that primarily receives published metrics from the Monitoring API and alarm state transition messages from the Threshold Engine that are consumed by other components, such as the Persister and Notification Engine. The Message Queue is also used to publish and consume other events in the system. Currently, a Kafka based MessageQ is supported. Kafka is a high performance, distributed, fault-tolerant, and scalable message queue with durability built-in. We will look at other alternatives, such as RabbitMQ and in-fact in our previous implementation RabbitMQ was supported, but due to performance, scale, durability and high-availability limitiations with RabbitMQ we have moved to Kafka.
* Metrics and Alarms Database: A third-party component that primarily stores metrics and the alarm state history. Currently, Vertica and InfluxDB are supported. Support for Cassandra is in progress.
* Config Database: A third-party component that stores a lot of the configuration and other information in the system. Currently, MySQL is supported. Support for PostgreSQL is in progress.
* Monitoring Client (python-monascaclient): A Python command line client and library that communicates and controls the Monitoring API. The Monitoring Client was written using the OpenStack Heat Python client as a framework. The Monitoring Client also has a Python library, "monascaclient" similar to the other OpenStack clients, that can be used to quickly build additional capabilities. The Monitoring Client library is used by the Monitoring UI, Ceilometer publisher, and other components.
* Monitoring UI: A Horizon dashboard for visualizing the overall health and status of an OpenStack cloud.
* Ceilometer publisher: A multi-publisher plugin for Ceilometer, not shown, that converts and publishes samples to the Monitoring API.
Most of the components are described in their respective repositories. However, there aren't any repositories for the third-party components used, so we describe some of the relevant details here.

=== Message Schema ===
[[Monasca/Message Schema]]

=== Message Queue ===
A distributed, performant, scalable, HA message queue for distributing metrics, alarms and events in the monitoring system. Currently, based on Kafka.

==== Messages ====
There are several messages that are published and consumed by various components in Monasca via the MessageQ. See [[Monasca/Message Schema|Message Schema]].

=== Metrics and Alarms Database ===

A high-performance analytics database that can store massive amounts of metrics and alarms in real-time and also support interactive queries. Currently Vertica and InfluxDB are supported.

The SQL schema that is used by Vertica is as follows:

* MonMetrics.Measurements: Stores the actual measurements that are sent.
** id: An integer ID for the measurement.
** definition_dimensions_id: A reference to DefinitionDimensions.
** time_stamp
** value
* MonMetrics.DefinitionDimensions
** id: A sha1 hash of (defintion_id, dimension_set_id)
** definition_id: A reference to the Definitions.id
** dimension_set_id: A reference to the Dimensions.dimension_set_id
* MonMetrics.Definitions
** id: A sha1 hash of the (name, tenant_id, region)
** name: Name of the metric.
** tenant_id: The tenant_id that submitted the metric.
** region: The region the metric was submitted under.
* MonMetric.Dimensions
** dimension_set_id: A sha1 hash of the set of dimenions for a metric.
** name: Name of dimension.
** value: Value of dimension.

=== Config Database ===
The config database store all the configuration information. Currently based on MySQL.

The SQL schema is as follows:

* alarm
** id
** tenant_id
** name
** description
** expression
** state
** actions_enabled
** created_at
** updated_at
** deleted_at
* alarm_action
** alarm_id
** alarm_state
** action_id
* notification_method
** id
** tenant_id
** name
** type
** address
** created_at
** updated_at
* sub_alarm
** id
** alarm_id
** function
** metric_name
** operator
** threshold
** period
** periods
** state
** created_at
** updated_at
* sub_alarm_dimension
** sub_alarm_id
** dimension_name
** value

== Events ==
Support for real-time event stream processing in Monasca is in progress. For more details see the link at, [[Monasca/Events]].

== Logging ==
Support for logging in Monasca is under discussion. For more details see the link at, [[Monasca/Logging]].

== Transform and Aggregation Engine ==
For more details see the link at, [[Monasca/Transform]].

== Analytics ==
Support for anomaly detection and alarm clustering/correlation is in progress. For more details see the link at, [[Monasca/Analytics]].

== Monitoring ==
Enablement and usage for monitoring the status of Monasca is under discussion. For more details see the link at, [[Monasca/Monitoring_Of_Monasca]]

== UI/UX Support ==
Adding more support for common UI/UX queries is under discussion. For more details see the link at, [[Monasca/UI_UX_Support]]

== Keystone Requirements ==
Monasca relies on keystone for running and there are requirements about which keystone configuration must exist.

* The endpoint for the api must be registered in keystone as the 'monasca' service.
* The api must have an admin token to use in verifying the keystone tokens it receives.
* For each project which uses Monasca two users must exist, one will be in the 'monasca-agent' role and be used by the monasca-agent's running on machines. The other should not be in that role and can be used logging into the UI, using the CLI or for direct queries against the API.

== Post Metric Sequence ==

This section describes the sequence of operations involved in posting a metric to the Monasca API.

[[File:monasca-arch-post-metric-diagram.png|Monasca Architecture Post Metric Diagram]]

# A metric is posted to the Monasca API.
# The Monasca API authenticates and validates the request and publishes the metric to the the Message Queue.
# The Persister consumes the metric from the Message Queue and stores in the Metrics Store.
# The Transform Engine consumes the metrics from the Message Queue, performs transform and aggregation operations on metrics, and publishes metrics that it creates back to Message Queue.
# The Threshold Engine consumes metrics from the Message Queue and evaluates alarms. If a state change occurs in an alarm, an "alarm-state-transitioned-event" is published to the Message Queue.
# The Notification Engine consumes "alarm-state-transitioned-events" from the Message Queue, evaluates whether they have a Notification Method associated with it, and sends the appropriate notification, such as email.
# The Persister consumes the "alarm-state-transitioned-event" from the Message Queue and stores it in the Alarm State History Store.

== Alarm Managers ==
This section describes the new features including alarm grouping, inhibition and silencing.

Examples:

* Silence
SilenceRule1 = '{"alarm-silencing-definition-created": {"name": "silence_rule_1", "matchers": {"severity": "LOW"}, "start_time": "2017-02-21 20:00:00", "end_time": "2017-02-21 22:00:00"}}'
Two alarm transitions: AT1 and AT2
AT1_severity = HIGH
AT2_severity = LOW

Output: AT2 get silenced and AT1 send a notification.
SilenceRule1 expires after "end_time"="2017-02-21 22:00:00".

* Inhibit
InhibitionRule1 = '{"alarm-inhibition-definition-created": {"name": "inhibit_rule_1","source":{"severity":"HIGH"},"target":{"severity":"LOW"}, "equals":["tenantId"], "exclusions": {"alarm_name": "vm is dead"}}}'
Three alarm transitions: AT1, AT2 and AT3
AT1_tenantId = "d42bc"
AT2_tenantId = "d42bc"
AT3_tenantId = "d42bc"
AT1_severity = HIGH
AT2_severity = LOW
AT3_severity = LOW
AT1_alarm_name = "cpu high"
AT2_alarm_name = "memory high"
AT3_alarm_name = "vm is dead"
AT1_state = ALARM
AT2_state = ALARM
AT3_state = ALARM

Output: AT1 is the source alarm which will send a notification. AT2 is the target alarm and will get inhibited. AT3 matches the exclusions and will send a notification immediately.

* Grouping Scenario 1

GroupingRule1 = '{"alarm-grouping-definition-created":
{"name": "group_rule_1", "matchers": ["hostname"], "id": "b7163","repeat_interval": "2h", "group_wait": "30s", "exclusions": {"alarm_name": "cpu_percent_high"}, "tenantId": " d42bc", "alarm_actions": ["cd892"], "ok_actions": ["ad892"], "undetermined_actions": ["cf892"]}}'

Three alarm transitions: AT1, AT2 and AT3

AT1_hostname = host1
AT2_hostname = host1
AT3_hostname = host2
AT1_alarm_name = cpu_percent_high
AT2_alarm_name = cpu_system_perc_high
AT3_alarm_name = cpu_percent_high
AT1_state = ALARM
AT2_state = ALARM
AT3_state = ALARM

Output: AT1 and AT3 match exclusions and send notifications immediately. Generate a grouped notification “group_notification_rule_1_host1_alarm[1]” and send out using alarm_actions ["cd892"]. There are no alarm_actions, ok_actions or undermined_actions associated with the AT1, AT2, AT3 alarm definitions.

* Grouping Scenario 2

GroupingRule2 = '{"alarm-grouping-definition-created":
{"name": "group_rule_2", "matchers": ["hostname"], "id": "b7163","repeat_interval": "2h", "group_wait": "30s", "tenantId": " d42bc", "alarm_actions": ["cd892"], "ok_actions": ["ad892"], "undetermined_actions": ["cf892"]}}'

Three alarm transitions: AT1 with "alarm_actions": ["123ab"], AT2 with "alarm_actions": ["cd839"] and AT3

AT1_hostname = host1
AT2_hostname = host1
AT3_hostname = host2
AT1_state = ALARM
AT2_state = ALARM
AT3_state = ALARM

Output: Generate two grouped notifications “group_notification_rule_2_host1_alarm[2]” and “group_notification_rule_2_host2_alarm[1]”. Both using alarm_actions ["cd892"]. Also since AT1 and AT2 has their own alarm actions associated with them, there will be two more notifications sent out.

* Silenced and Grouped

SilenceRule1 = '{"alarm-silencing-definition-created": {"name": "silence_rule_1", "matchers": {"severity": "LOW"}, "start_time": "1487269470498", "end_time": "1587269470498"}}'

GroupingRule2 = '{"alarm-grouping-definition-created":
{"name": "group_rule_2", "matchers": ["hostname"], "id": "b7163","repeat_interval": "2h", "group_wait": "30s", "tenantId": " d42bc", "alarm_actions": ["cd892"], "ok_actions": ["ad892"], "undetermined_actions": ["cf892"]}}'

Four alarm transitions: AT1, AT2, AT3 and AT4

Silencing rule:
AT1_severity = HIGH
AT2_severity = LOW
AT3_severity = HIGH
AT4_severity = HIGH

Grouping rule:
AT1_hostname = host1
AT2_hostname = host1
AT3_hostname = host2
AT4_hostname = host1
AT1_state = ALARM
AT2_state = ALARM
AT3_state = OK
AT4_state = ALARM

Output: Generate two grouped notifications “group_notification_rule_2_host1_alarm[2]” using alarm action "cd892" and “group_notification_rule_2_host2_ok[1]” using ok action "ad892". AT2 is silenced so it isn't include in group_notification_rule_2_host1 count. There are no alarm_actions, ok_actions or undermined_actions associated with the AT1, AT2, AT3, AT4 alarm definitions.

* Silenced and inhibited (source alarm get silenced)

SilenceRule2 = '{"alarm-silencing-definition-created": {"name": "silence_rule_2", "matchers": {"severity": "HIGH, "hostname": "host1"}, "start_time": "2017-02-21 15:00:00", "end_time": "2017-02-21 21:00:00"}}'

InhibitionRule1 = '{"alarm-inhibition-definition-created": {"name": "inhibit_rule_1","source":{"severity":"HIGH"},"target":{"severity":"LOW"}, "equals":["tenantId"]}}'

Alarm transitions: AT1, AT2

Inhibition rule:
AT1_state = ALARM
AT2_state = ALARM
AT1_tenantId = "d42bc"
AT2_tenantId = "d42bc"
AT1_severity = HIGH
AT2_severity = LOW

Silencing rule:
AT1_hostname = host1
AT2_hostname = host2
AT1_severity = HIGH
AT2_severity = LOW

Output: no notification sent out. For inhibition, AT1 is the source alarm, AT2 is the target alarm. But at the same time, AT1 get silenced because it matches the silence rule.

* Inhibited and grouped

InhibitionRule1 = '{"alarm-inhibition-definition-created": {"name": "inhibit_rule_1","source":{"severity":"HIGH"},"target":{"severity":"LOW"}, "equals":["tenantId"]}}'

GroupingRule1 = '{"alarm-grouping-definition-created": {"name": "group_rule_1", "matchers": ["hostname"], "id": "b7163", "repeat_interval": "2h", "group_wait": "30s", "tenantId": " d42bc", "alarm_actions": ["cd892"], "ok_actions": ["ad892"], "undetermined_actions": ["cf892"]}}'

Alarm transitions: AT1, AT2, AT3

Inhibition rule:
AT1_tenantId = "d42bc"
AT2_tenantId = "d42bc"
AT3_tenantId = "d42bc"
AT1_severity = HIGH
AT2_severity = LOW
AT3_severity = HIGH
AT1_state = OK
AT2_state = ALARM
AT3_state = ALARM

Grouping rule:
AT1_hostname = host1
AT2_hostname = host2
AT3_hostname = host1
AT1_state = OK
AT2_state = ALARM
AT3_state = ALARM

Output: AT2 gets inhibited because its severity is low. AT3 is the source alarm. Since AT1 is in OK state, it is not a source alarm. For grouping, AT1 and AT3 has the same host name but different state. So there will be two grouped notifications sent out: “group_notification_rule_1_host1_ok[1]” and “group_notification_rule_1_host1_alarm[1]”.

* Silenced, inhibited and grouped

SilenceRule2 = '{"alarm-silencing-definition-created": {"name": "silence_rule_2", "matchers": {"severity": "HIGH, "hostname": "host1"}, "start_time": "2017-02-21 15:00:00", "end_time": "2017-02-21 21:00:00"}}'

InhibitionRule1 = '{"alarm-inhibition-definition-created": {"name": "inhibit_rule_1","source":{"severity":"HIGH"},"target":{"severity":"LOW"}, "equals":["tenantId"]}}'

GroupingRule1 = '{"alarm-grouping-definition-created": {"name": "group_rule_1", "matchers": ["hostname"], "id": "b7163", "repeat_interval": "2h", "group_wait": "30s", "tenantId": " d42bc", "alarm_actions": ["cd892"], "ok_actions": ["ad892"], "undetermined_actions": ["cf892"]}}'

Alarm transitions: AT1, AT2, AT3 and AT5

Silence rule:
AT1_severity = HIGH AT1_hostname = host1 (silenced)
AT2_severity = LOW AT2_hostname = host2
AT3_severity = HIGH AT3_hostname = host1(silenced)
AT5_severity = HIGH AT5_hostname = host3

Inhibition rule:
AT1_tenantId = "d42bc"
AT2_tenantId = "d42bc"
AT3_tenantId = "d42bc"
AT5_tenantId = "d42bc"
AT1_state = ALARM
AT2_state = ALARM
AT3_state = OK
AT5_state = UNDETERMINED
AT1_severity = HIGH (source)
AT2_severity = LOW (target)
AT3_severity = HIGH
AT5_severity = HIGH

Grouping rule:
AT1_hostname = host1
AT2_hostname = host2
AT3_hostname = host1
AT5_hostname = host3

Output: AT1 is in "group_notification_rule_1_host1_alarm" group and silenced. AT3 is in "group_notification_rule_1_host1_ok" group and silenced. AT2 is in "group_notification_rule_1_host2_alarm" group and inhibited. AT5 is in "group_notification_rule_1_host3_undetermined" group and will send notification “group_notification_rule_1_host3_undetermined[1]” using undetermined action "cf892".

= Development Environment =

* Monasca DevStack Plugin:
** DevStack is the primary developmement environment for OpenStack.
***See http://docs.openstack.org/developer/devstack/
** The Monasca DevStack plugin installs the Monasca Service, Agent, Horizon Monitoring Panel, and Grafana
** README at, https://github.com/openstack/monasca-api/tree/master/devstack
** Best way to get started is to use Vagrant with the Vagrantfile at, https://github.com/openstack/monasca-api/blob/master/devstack/Vagrantfile.

* Monasca on LaunchPad. https://launchpad.net/monasca

* Monasca projects source code. https://git.openstack.org/cgit/?q=monasca

= Coding Standards =

* Python: All Python code conforms to the OpenStack standards at, http://docs.openstack.org/developer/hacking/.
** Note, all components in Monasca, except for the Threshold Engine, have been ported to Python.

* Java: Several of the Monasca components are available as Java. OpenStack does not have any Java coding standards. We've adopted the Google Java Style at, https://google.github.io/styleguide/javaguide.html.
** The standard says either 80 or 100 length lines. We've adopted 100.

= Technologies =

Monasca uses a number of third-party technologies:

* Apache Kafka (http://kafka.apache.org): Apache Kafka is publish-subscribe messaging rethought as a distributed commit log. Kafka is a highly performant, distributed, fault-tolerant, and scalable message queue with durability built-in.

* Apache Storm (http://storm.incubator.apache.org/): Apache Storm is a free and open source distributed realtime computation system. Storm makes it easy to reliably process unbounded streams of data, doing for realtime processing what Hadoop did for batch processing.

* ZooKeeper (http://zookeeper.apache.org/): Used by Kafka and Storm.

* MySQL: MySQL is supported as a Config Database.

* POSTgres: Support for POSTgres, via Hibernate and SQLAlchemy, for the Config Database.

* Vagrant (http://www.vagrantup.com/): Vagrant provides easy to configure, reproducible, and portable work environments built on top of industry-standard technology and controlled by a single consistent workflow to help maximize the productivity and flexibility of you and your team.

* Dropwizard (https://dropwizard.github.io/dropwizard/): Dropwizard pulls together stable, mature libraries from the Java ecosystem into a simple, light-weight package that lets you focus on getting things done. Dropwizard has out-of-the-box support for sophisticated configuration, application metrics, logging, operational tools, and much more, allowing you and your team to ship a production-quality web service in the shortest time possible.

* InfluxDB (http://influxdb.com/): An open-source distributed time series database with no external dependencies. InfluxDB is supported for the Metrics Database.

* Vertica (http://www.vertica.com): A commercial Enterprise class SQL analytics database that is highly scalable. It offers built-in automatic high-availability and excels at in-database analytics and compressing and storing massive amounts of data. A free community version of Vertica is available that can store up to 1 TB of data with no time-limit is available at, https://my.vertica.com/community/. Vertica is supported for the Metrics Database.

* Cassandra: Support for Cassandra for the Metrics Database is in progress.

= License =

Copyright (c) 2014, 2015 Hewlett-Packard Development Company, L.P.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied.
See the License for the specific language governing permissions and
limitations under the License.

[[File:yklogo.png|200px|thumb|left|Monasca uses YourKit Profiler for Java development]]

 
 
 
 

[https://www.yourkit.com/ Visit YourKit website for more information]

Auto-scaling SIG

2019-05-06T19:00:39Z

Joseph Davis: /* Community Infrastructure / Resources */

== Auto-scaling SIG ==

'''Status''': In review

'''Original proposal''': http://lists.openstack.org/pipermail/openstack-discuss/2019-January/001656.html

=== Mission ===

This [[OpenStack_SIGs|SIG]] aims to improve the experience of developing, operating, and using auto-scaling and its related features (like metering, cluster schedule, life cycle management), and to coordinate efforts across projects and communities (like k8s cluster auto-scaling on OpenStack). The SIG also provides a central place to put tests, documentations, and even common libraries for auto-scaling features.

The SIG is expected to focus more on auto-scaling user workloads; however work on auto-scaling infrastructure is also welcome, especially considering that user workloads in an undercloud are actually infrastructure in the corresponding overcloud.

=== Background ===
OpenStack provides multiple methods to auto-scale your cluster (Like using Heat AutoScalingGroup, Senlin Cluster, etc.). However without general coordination across projects, it may not be easy for users and ops to achieve auto-scaling on OpenStack. Developers tend to be focused on individual projects rather than cross-project integration. Most of the components required by auto-scaling already exist within OpenStack, but we need to provide a more simple way for users and ops to adopt auto-scaling. And allowing developers to coordinate together instead of implement something all over again.

=== Goals ===
* Set up SIG, init doc, meeting, ml
* Create Forum and Auto-scaling SIG PTG sessions

=== Getting Involved ===

We're currently working on a structure to make everyone more easier to getting involved. Our initial git repository, documentation, and other information will be up soon.

=== Community Infrastructure / Resources ===

* Wiki: this page
* [https://storyboard.openstack.org/#!/project/openstack/auto-scaling-sig SIG StoryBoard] (for an authoritative list of all ongoing work within the SIG)
* [http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-discuss openstack-discuss mailing list]; use the <code>[auto-scaling]</code> tag
* IRC channel: #openstack-auto-scaling on [http://freenode.net/ Freenode] IRC
* Regular IRC meetings 0900 and 2300 UTC on even weeks on IRC #openstack-auto-scaling [https://review.opendev.org/#/c/656810/]
* [https://review.openstack.org/#/q/project:openstack/auto-scaling-sig patch reviews] (gerrit)
* [https://opendev.org/openstack/auto-scaling-sig Git repository for SIG] (more content coming soon)

=== SIG Chairs ===

* Rico Lin
* Duc Truong
* Zane Bitter

We welcome anyone who willing to join and help. And we surely are willing to re-elect chairs as soon as we got our init setup done.

=== Upcoming events ===

TBD

=== Past events ===

* Superuser article, Sept 2018
** [https://superuser.openstack.org/articles/how-to-auto-scale-a-self-healing-cluster-with-heat/ How to auto scale a self healing cluster with Heat]
* Events at Berlin Forum, November 2018
** [https://etherpad.openstack.org/p/autoscaling-integration-and-feedback Autoscaling Integration, improvement, and feedback]
* Events at Denver OpenInfra Summit and PTG, April 2019
** [https://www.openstack.org/summit/denver-2019/summit-schedule/events/23379/how-blizzard-entertainment-uses-autoscaling-with-overwatch (presentation) How Blizzard uses Auto-scaling with Overwatch]
** [https://www.openstack.org/summit/denver-2019/summit-schedule/events/23291/scaling-should-be-easy-and-automatic-heat-and-monasca (workshop) Scaling should be Easy and Automatic]
** [https://etherpad.openstack.org/p/DEN-auto-scaling-SIG Auto-Scaling SIG PTG etherpad]

=== Project liasons ===
This is a good idea, but we need some more discussion on who and how we can bridge this system up.

Auto-scaling SIG

2019-05-06T18:52:42Z

Joseph Davis: /* Past events */

== Auto-scaling SIG ==

'''Status''': In review

'''Original proposal''': http://lists.openstack.org/pipermail/openstack-discuss/2019-January/001656.html

=== Mission ===

This [[OpenStack_SIGs|SIG]] aims to improve the experience of developing, operating, and using auto-scaling and its related features (like metering, cluster schedule, life cycle management), and to coordinate efforts across projects and communities (like k8s cluster auto-scaling on OpenStack). The SIG also provides a central place to put tests, documentations, and even common libraries for auto-scaling features.

The SIG is expected to focus more on auto-scaling user workloads; however work on auto-scaling infrastructure is also welcome, especially considering that user workloads in an undercloud are actually infrastructure in the corresponding overcloud.

=== Background ===
OpenStack provides multiple methods to auto-scale your cluster (Like using Heat AutoScalingGroup, Senlin Cluster, etc.). However without general coordination across projects, it may not be easy for users and ops to achieve auto-scaling on OpenStack. Developers tend to be focused on individual projects rather than cross-project integration. Most of the components required by auto-scaling already exist within OpenStack, but we need to provide a more simple way for users and ops to adopt auto-scaling. And allowing developers to coordinate together instead of implement something all over again.

=== Goals ===
* Set up SIG, init doc, meeting, ml
* Create Forum and Auto-scaling SIG PTG sessions

=== Getting Involved ===

We're currently working on a structure to make everyone more easier to getting involved. Our initial git repository, documentation, and other information will be up soon.

=== Community Infrastructure / Resources ===

* Wiki: this page
* [https://storyboard.openstack.org/#!/project/openstack/auto-scaling-sig SIG StoryBoard] (for an authoritative list of all ongoing work within the SIG)
* [http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-discuss openstack-discuss mailing list]; use the <code>[auto-scaling]</code> tag
* IRC channel: #openstack-auto-scaling on [http://freenode.net/ Freenode] IRC
* Regular IRC meetings have not been arranged yet
* [https://review.openstack.org/#/q/project:openstack/auto-scaling-sig patch reviews] (gerrit)

=== SIG Chairs ===

* Rico Lin
* Duc Truong
* Zane Bitter

We welcome anyone who willing to join and help. And we surely are willing to re-elect chairs as soon as we got our init setup done.

=== Upcoming events ===

TBD

=== Past events ===

* Superuser article, Sept 2018
** [https://superuser.openstack.org/articles/how-to-auto-scale-a-self-healing-cluster-with-heat/ How to auto scale a self healing cluster with Heat]
* Events at Berlin Forum, November 2018
** [https://etherpad.openstack.org/p/autoscaling-integration-and-feedback Autoscaling Integration, improvement, and feedback]
* Events at Denver OpenInfra Summit and PTG, April 2019
** [https://www.openstack.org/summit/denver-2019/summit-schedule/events/23379/how-blizzard-entertainment-uses-autoscaling-with-overwatch (presentation) How Blizzard uses Auto-scaling with Overwatch]
** [https://www.openstack.org/summit/denver-2019/summit-schedule/events/23291/scaling-should-be-easy-and-automatic-heat-and-monasca (workshop) Scaling should be Easy and Automatic]
** [https://etherpad.openstack.org/p/DEN-auto-scaling-SIG Auto-Scaling SIG PTG etherpad]

=== Project liasons ===
This is a good idea, but we need some more discussion on who and how we can bridge this system up.

Auto-scaling SIG

2019-05-06T18:49:43Z

Joseph Davis: /* Past events */

== Auto-scaling SIG ==

'''Status''': In review

'''Original proposal''': http://lists.openstack.org/pipermail/openstack-discuss/2019-January/001656.html

=== Mission ===

This [[OpenStack_SIGs|SIG]] aims to improve the experience of developing, operating, and using auto-scaling and its related features (like metering, cluster schedule, life cycle management), and to coordinate efforts across projects and communities (like k8s cluster auto-scaling on OpenStack). The SIG also provides a central place to put tests, documentations, and even common libraries for auto-scaling features.

The SIG is expected to focus more on auto-scaling user workloads; however work on auto-scaling infrastructure is also welcome, especially considering that user workloads in an undercloud are actually infrastructure in the corresponding overcloud.

=== Background ===
OpenStack provides multiple methods to auto-scale your cluster (Like using Heat AutoScalingGroup, Senlin Cluster, etc.). However without general coordination across projects, it may not be easy for users and ops to achieve auto-scaling on OpenStack. Developers tend to be focused on individual projects rather than cross-project integration. Most of the components required by auto-scaling already exist within OpenStack, but we need to provide a more simple way for users and ops to adopt auto-scaling. And allowing developers to coordinate together instead of implement something all over again.

=== Goals ===
* Set up SIG, init doc, meeting, ml
* Create Forum and Auto-scaling SIG PTG sessions

=== Getting Involved ===

We're currently working on a structure to make everyone more easier to getting involved. Our initial git repository, documentation, and other information will be up soon.

=== Community Infrastructure / Resources ===

* Wiki: this page
* [https://storyboard.openstack.org/#!/project/openstack/auto-scaling-sig SIG StoryBoard] (for an authoritative list of all ongoing work within the SIG)
* [http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-discuss openstack-discuss mailing list]; use the <code>[auto-scaling]</code> tag
* IRC channel: #openstack-auto-scaling on [http://freenode.net/ Freenode] IRC
* Regular IRC meetings have not been arranged yet
* [https://review.openstack.org/#/q/project:openstack/auto-scaling-sig patch reviews] (gerrit)

=== SIG Chairs ===

* Rico Lin
* Duc Truong
* Zane Bitter

We welcome anyone who willing to join and help. And we surely are willing to re-elect chairs as soon as we got our init setup done.

=== Upcoming events ===

TBD

=== Past events ===

* Events at Berlin Forum, November 2018
** [https://etherpad.openstack.org/p/autoscaling-integration-and-feedback Autoscaling Integration, improvement, and feedback]
* Events at Denver OpenInfra Summit and PTG, April 2019
** [https://www.openstack.org/summit/denver-2019/summit-schedule/events/23379/how-blizzard-entertainment-uses-autoscaling-with-overwatch (presentation) How Blizzard uses Auto-scaling with Overwatch]
** [https://www.openstack.org/summit/denver-2019/summit-schedule/events/23291/scaling-should-be-easy-and-automatic-heat-and-monasca (workshop) Scaling should be Easy and Automatic]
** [https://etherpad.openstack.org/p/DEN-auto-scaling-SIG Auto-Scaling SIG PTG etherpad]

=== Project liasons ===
This is a good idea, but we need some more discussion on who and how we can bridge this system up.

PTG/Train/Etherpads

2019-05-02T19:42:29Z

Joseph Davis: /* SIG/Theme/Other */ alphabetize SIGs

__NOTOC__

This is the list of etherpads for the Projects Team Gathering for the Train release in Denver, 2019. Each team can organize the content on their allocated day(s) in the way that seems to most appropriate to them. We suspect most teams will avoid strict timeboxed slots and will use etherpads to list topics to cover. This page lists those etherpads for easy reference.

For more details on the event, see the [https://www.openstack.org/ptg/ event website].

For what's happening '''right now''' (during the event), see the [http://ptg.openstack.org/ptg.html ptgbot page].

=== Projects ===
* Blazar - https://etherpad.openstack.org/p/blazar-ptg-train
* Charms - https://etherpad.openstack.org/p/charms-train-ptg
* Cinder - https://etherpad.openstack.org/p/cinder-train-ptg-planning
* Cyborg - https://etherpad.openstack.org/p/cyborg-ptg-train
* Docs/I18n - https://etherpad.openstack.org/p/docs-i18n-ptg-train
* Fenix
** Fenix general - https://etherpad.openstack.org/p/DEN2019-fenix-PTG
** Fenix - ETSI NFV - https://etherpad.openstack.org/p/DEN2019-fenix-ETSI-NFV-PTG
* Glance - https://etherpad.openstack.org/p/Glance-Train-PTG-planning
* Heat - https://etherpad.openstack.org/p/DEN-Train-Heat
* Horizon - https://etherpad.openstack.org/p/horizon-train-ptg
* Infra - https://etherpad.openstack.org/p/2019-denver-ptg-infra-planning
* Ironic - https://etherpad.openstack.org/p/DEN-train-ironic-ptg
* Keystone - https://etherpad.openstack.org/p/keystone-train-ptg
* Magnum - https://etherpad.openstack.org/p/magnum-train-ptg
* Manila
** Planning: https://etherpad.openstack.org/p/manila-denver-train-ptg-planning
** Minutes/Proceedings: https://etherpad.openstack.org/p/manila-ptg-train
* Monasca - https://etherpad.openstack.org/p/monasca-ptg-train
* Neutron - https://etherpad.openstack.org/p/openstack-networking-train-ptg
* Nova - https://etherpad.openstack.org/p/nova-ptg-train
* Octavia - https://etherpad.openstack.org/p/octavia-train-ptg
* OpenStackAnsible - https://etherpad.openstack.org/p/osa-train-ptg
* OpenStack Client - https://etherpad.openstack.org/p/train-ptg-osc
* OpenStack Helm - https://etherpad.openstack.org/p/osh-ptg-train
* Oslo - https://etherpad.openstack.org/p/oslo-train-topics
* Placement - https://etherpad.openstack.org/p/placement-ptg-train
* QA
** Schedule: https://ethercalc.openstack.org/Train-PTG-QA-Schedule
** Etherpad: https://etherpad.openstack.org/p/qa-train-ptg
* Release Team - https://etherpad.openstack.org/p/relmgmt-train-ptg
* StoryBoard - https://etherpad.openstack.org/p/sb-train-ptg
* Swift - https://etherpad.openstack.org/p/swift-ptg-train
* Tripleo
** Schedule: https://etherpad.openstack.org/p/tripleo-ptg-train
** Topic Planning: https://etherpad.openstack.org/p/tripleo-train-topics
* Vitrage - https://etherpad.openstack.org/p/vitrage-train-ptg

=== Cross-Project ===
* Nova/Cinder - https://etherpad.openstack.org/p/ptg-train-xproj-nova-cinder
* Nova/Cyborg - https://etherpad.openstack.org/p/ptg-train-xproj-nova-cyborg
* Nova/Ironic - https://etherpad.openstack.org/p/ptg-train-xproj-nova-ironic
* Nova/Keystone - https://etherpad.openstack.org/p/ptg-train-xproj-nova-keystone
* Nova/Neutron - https://etherpad.openstack.org/p/ptg-train-xproj-nova-neutron
* Nova/Placement - https://etherpad.openstack.org/p/ptg-train-xproj-nova-placement
* Cyborg/Ironic - https://etherpad.openstack.org/p/ptg-train-xproj-ironic-cyborg

=== SIG/Theme/Other ===
* API SIG - https://etherpad.openstack.org/p/api-sig-ptg-train
* Auto-scaling SIG - https://etherpad.openstack.org/p/DEN-auto-scaling-SIG
* Edge WG - https://etherpad.openstack.org/p/edge-wg-ptg-preparation-denver-2019
* K8s SIG - https://etherpad.openstack.org/p/k8s-sig-ptg-train
* Public Cloud SIG - https://etherpad.openstack.org/p/DEN-public-cloud-SIG
* Scientific SIG - https://etherpad.openstack.org/p/scientific-sig-ptg-train
* Self-healing SIG - https://etherpad.openstack.org/p/DEN-self-healing-SIG
* Security SIG - https://etherpad.openstack.org/p/security-sig-ptg-train
* Technical Committee - https://etherpad.openstack.org/p/tc-train-ptg
* Upgrades SIG - https://etherpad.openstack.org/p/upgrade-sig-ptg-train

=== Pilot Projects ===
* Airship - https://etherpad.openstack.org/p/airship-ptg-train
* StarlingX - https://etherpad.openstack.org/p/stx-ptg-agenda

=== Other ===