Jump to: navigation, search

Difference between revisions of "Monasca"

(Overview)
(Messages)
Line 124: Line 124:
 
! Message !! Produced By !! Consumed By !! Kafka Topic !! Description
 
! Message !! Produced By !! Consumed By !! Kafka Topic !! Description
 
|-
 
|-
| Metric || API, Transform and Aggregation Engine || Persister, Threshold Engine || metrics || A metric sent to the Monitoring API or created by the Transform and Aggregation Engine is published to the MessageQ.
+
| Metric || API, Transform and Aggregation Engine || Persister, Threshold Engine || metrics || A metric sent to the API or created by the Transform Engine, Event Engine or Anomaly Engine is published to the MessageQ.
 
|-
 
|-
| Alarm Definition Event || API || Threshold Engine || events || When an alarm is created, updated, or deleted by the monitoring API an Alarm Definition Event is published to the MessageQ.
+
| AlarmDefinitionCreatedEvent || API || Threshold Engine || events || When an alarm definition is created by the API an AlarmDefinitionCreatedEvent is published to the MessageQ.
 
|-
 
|-
| Alarm State Transitioned || API || Threshold Engine || events || When an alarm is created, updated, or deleted by the monitoring API an Alarm Definition Event is published to the MessageQ.
+
| AlarmDefinitionDeletedEvent || API || Threshold Engine || events || When an alarm definition is deleted by the API an AlarmDefinitionDeletedEvent is published to the MessageQ.
 
|-
 
|-
| Alarm State Transitioned || Threshold Engine || Notification Engine, Persister || alarm-state-transitions || When an alarm transitions from the OK to Alarmed, Alarmed to OK, ..., this event is published to the MessageQ and persisted by the persister and processed by the Notification Engine. The Monitoring API can query the history of alarm state transition events.
+
| AlarmDefinitionUpdatedEvent || API || Threshold Engine || events || When an alarm definition is updated by the API an AlarmDefinitionUpdatedEvent is published to the MessageQ.
 
|-
 
|-
| Alarm Notification || Notification Engine || Persister || alarm-notifications || This event is published to the MessageQ when the notification engine processes an alarm and sends a notification. The alarm notification is persisted by the Persister and can be queried by the Monitoring API. The database maintains a history of all events.
+
| AlarmCreatedEvent || Threshold Engine ||  || events || When an alarm is created by the Threshold Engine an AlarmCreatedEvent is published to the MessageQ.
 +
|-
 +
| AlarmDeletedEvent || API || Threshold Engine || events || When an alarm is deleted by the API an AlarmDeletedEvent is published to the MessageQ.
 +
|-
 +
| AlarmUpdatedEvent || API || Threshold Engine || events || When an alarm is updated by the API an AlarmUpdatedEvent is published to the MessageQ.
 +
|-
 +
| AlarmStateTransitionedEvent || Threshold Engine || Notification Engine, Persister || alarm-state-transitions || When an alarm transitions from the OK to Alarmed, Alarmed to OK, ..., this event is published to the MessageQ and persisted by the persister and processed by the Notification Engine. The Monitoring API can query the history of alarm state transition events.
 +
|-
 +
| AlarmNotification || Notification Engine || Persister || alarm-notifications || This event is published to the MessageQ when the notification engine processes an alarm and sends a notification. The alarm notification is persisted by the Persister and can be queried by the Monitoring API. The database maintains a history of all events.
 
|}
 
|}
  

Revision as of 01:31, 14 January 2015

Overview

Monasca is a open-source multi-tenant, highly scalable, performant, fault-tolerant monitoring-as-a-service solution that integrates with OpenStack. It uses a REST API for high-speed metrics processing and querying and has a streaming alarm engine and notification engine.

  • IRC
    • #openstack-monasca on freenode.net

Requirements

  • Monitors OpenStack infrastructure and resources
    • monitors both OpenStack infrastructure and its resources (e.g., VMs) so that another monitoring tool may not be needed.
    • monitors OpenStack CRUD operations using StackTach
    • (notion of host, applications, grouping)
  • Can Scale
    • to hundreds of thousands of servers, metrics, alarms
  • Allows user-defined metric and alarm creation
    • Where a user is system administrator of the cloud, or a cloud consumer
    • Allows metrics and alarms to be defined on a group of resources (e.g., host aggregates, availability zones)
  • Multi-tenant, multi-cloud, multiple deployment models
    • can process metrics from multiple clouds, tenants, and users
    • works in multiple deployment models
      • single private cloud (e.g., 10 nodes)
      • multiple private clouds
      • private and public cloud (hybrid)
  • Has an HTTP based extensible API
    • for metrics, alarm creation, metric collection.
  • Can monitor OpenStack projects
  • Authentication
    • Authenticates with keystone. May also authenticate with other backends (e.g., CloudFoundry, Open LDAP, AWS)
  • Provides a pluggable messaging system
  • Provides a pluggable backend store
    • e.g., vertica, influx DB, Cassandra, MySQL
  • A monitoring dashboard
    • using Grafana
  • Built using python and open source tools

Features

This section describes the overall features.

  • A highly performant, scalable, reliable and fault-tolerant Monitoring as a Service (MONaaS) solution that scales to service provider metrics levels of metrics throughput. Performance, scalability and high-availability have been designed in from the start. Can process 100s of thousands of metrics/sec as well as offer data retention periods of greater than a year with no data loss while still processing interactive queries.
  • Rest API for storing and querying metrics and historical information. Most monitoring solution use special transports and protocols, such as CollectD or NSCA (Nagios). In our solution, http is the only protocol used. This simplifies the overall design and also allows for a much richer way of describing the data via dimensions.
  • Multi-tenant and authenticated. Metrics are submitted and authenticated using Keystone and stored associated with a tenant ID.
  • Metrics defined using a set of (key, value) pairs called dimensions.
  • Real-time thresholding and alarming on metrics.
  • Compound alarms described using a simple expressive grammar composed of alarm sub-expressions and logical operators.
  • Monitoring agent that supports a number of built-in system and service checks and also supports Nagios checks and statsd.
  • Open-source monitoring solution built on open-source technologies.

Architecture

Monasca Architecture Component Diagram

  • Monitoring Agent (monasca-agent): A modern Python based monitoring agent that consists of several sub-components and supports system metrics, such as cpu utilization and available memory, Nagios plugins, statsd and many built-in checks for services such as MySQL, RabbitMQ, and many others.
  • Monitoring API (monasca-api): A well-defined and documented RESTful API for monitoring that is primarily focused on the following concepts and areas:
    • Metrics: Store and query massive amounts of metrics in real-time.
    • Statistics: Query statistics for metrics.
    • Alarm Definitions: Create, update, query and delete alarm definitions.
    • Alarms: Query and delete the alarm history.
      • Simple expressive grammar for creating compound alarms composed of alarm subexpressions and logical operators.
      • Alarm severities can be associated with alarms.
      • The complete alarm state transition history is stored and queryable which allows for subsequent root cause analysis (RCA) or advanced analytics.
    • Notification Methods: Create and delete notification methods and associate them with alarms, such as email. Supports the ability to notify users directly via email when an alarm state transitions occur.
  • Persister (monasca-persister): Consumes metrics and alarm state transitions from the MessageQ and stores them in the Metrics and Alarms database. We will look into converting the Persister to a Python component in the future.
  • Transform and Aggregation Engine: Transform metric names and values, such as delta or time-based derivative calculations, and creates new metrics that are published to the Message Queue. The Transform Engine is not available yet.
  • Anomaly and Prediction Engine: Evaluates prediction and anomalies and generates predicted metrics as well as anomaly likelihood and anomaly scores.
  • Threshold Engine (monasca-thresh): Computes thresholds on metrics and publishes alarms to the MessageQ when exceeded. Based on Apache Storm a free and open distributed real-time computation system.
  • Notification Engine (monasca-notification): Consumes alarm state transition messages from the MessageQ and sends notifications, such as emails for alarms. The Notification Engine is Python based.
  • Message Queue: A third-party component that primarily receives published metrics from the Monitoring API and alarm state transition messages from the Threshold Engine that are consumed by other components, such as the Persister and Notification Engine. The Message Queue is also used to publish and consume other events in the system. Currently, a Kafka based MessageQ is supported. Kafka is a high performance, distributed, fault-tolerant, and scalable message queue with durability built-in. We will look at other alternatives, such as RabbitMQ and in-fact in our previous implementation RabbitMQ was supported, but due to performance, scale, durability and high-availability limitiations with RabbitMQ we have moved to Kafka.
  • Metrics and Alarms Database: A third-party component that primarily stores metrics and the alarm state history. Currently, Vertica and InfluxDB are supported.
  • Config Database: A third-party component that stores a lot of the configuration and other information in the system. Currently, MySQL is supported.
  • Monitoring Client (python-monascaclient): A Python command line client and library that communicates and controls the Monitoring API. The Monitoring Client was written using the OpenStack Heat Python client as a framework. The Monitoring Client also has a Python library, "monascaclient" similar to the other OpenStack clients, that can be used to quickly build additional capabilities. The Monitoring Client library is used by the Monitoring UI, Ceilometer publisher, and other components.
  • Alarm Configuration Manager: A Python process that will detect new metrics and configure alarms based on the configuration. It uses the monitoring client library that communicates with the Monitoring API. The Alarm Configuration Manager is a Python Daemon that runs on a configurable interval and detects new metrics that need to be alarmed and creates the alarms.
  • Monitoring UI: A Horizon dashboard for visualizing the overall health and status of an OpenStack cloud.
  • Ceilometer publisher: A multi-publisher plugin for Ceilometer, not shown, that converts and publishes samples to the Monitoring API.

Most of the components are described in their respective repositories. However, there aren't any repositories for the third-party components used, so we describe some of the relevant details here.

Message Queue

A distributed, performant, scalable, HA message queue for distributing metrics, alarms and events in the monitoring system. Currently, based on Kafka.

Messages

There are several messages that are published and consumed by various components in the monitoring system via the MessageQ.

Message Produced By Consumed By Kafka Topic Description
Metric API, Transform and Aggregation Engine Persister, Threshold Engine metrics A metric sent to the API or created by the Transform Engine, Event Engine or Anomaly Engine is published to the MessageQ.
AlarmDefinitionCreatedEvent API Threshold Engine events When an alarm definition is created by the API an AlarmDefinitionCreatedEvent is published to the MessageQ.
AlarmDefinitionDeletedEvent API Threshold Engine events When an alarm definition is deleted by the API an AlarmDefinitionDeletedEvent is published to the MessageQ.
AlarmDefinitionUpdatedEvent API Threshold Engine events When an alarm definition is updated by the API an AlarmDefinitionUpdatedEvent is published to the MessageQ.
AlarmCreatedEvent Threshold Engine events When an alarm is created by the Threshold Engine an AlarmCreatedEvent is published to the MessageQ.
AlarmDeletedEvent API Threshold Engine events When an alarm is deleted by the API an AlarmDeletedEvent is published to the MessageQ.
AlarmUpdatedEvent API Threshold Engine events When an alarm is updated by the API an AlarmUpdatedEvent is published to the MessageQ.
AlarmStateTransitionedEvent Threshold Engine Notification Engine, Persister alarm-state-transitions When an alarm transitions from the OK to Alarmed, Alarmed to OK, ..., this event is published to the MessageQ and persisted by the persister and processed by the Notification Engine. The Monitoring API can query the history of alarm state transition events.
AlarmNotification Notification Engine Persister alarm-notifications This event is published to the MessageQ when the notification engine processes an alarm and sends a notification. The alarm notification is persisted by the Persister and can be queried by the Monitoring API. The database maintains a history of all events.

Metrics and Alarms Database

A high-performance analytics database that can store massive amounts of metrics and alarms in real-time and also support interactive queries. Currently Vertica and InfluxDB are supported.

The SQL schema that is used by Vertica is as follows:

  • MonMetrics.Measurements: Stores the actual measurements that are sent.
    • id: An integer ID for the measurement.
    • definition_dimensions_id: A reference to DefinitionDimensions.
    • time_stamp
    • value
  • MonMetrics.DefinitionDimensions
    • id: A sha1 hash of (defintion_id, dimension_set_id)
    • definition_id: A reference to the Definitions.id
    • dimension_set_id: A reference to the Dimensions.dimension_set_id
  • MonMetrics.Definitions
    • id: A sha1 hash of the (name, tenant_id, region)
    • name: Name of the metric.
    • tenant_id: The tenant_id that submitted the metric.
    • region: The region the metric was submitted under.
  • MonMetric.Dimensions
    • dimension_set_id: A sha1 hash of the set of dimenions for a metric.
    • name: Name of dimension.
    • value: Value of dimension.

Config Database

The config database store all the configuration information. Currently based on MySQL.

The SQL schema is as follows:

  • alarm
    • id
    • tenant_id
    • name
    • description
    • expression
    • state
    • actions_enabled
    • created_at
    • updated_at
    • deleted_at
  • alarm_action
    • alarm_id
    • alarm_state
    • action_id
  • notification_method
    • id
    • tenant_id
    • name
    • type
    • address
    • created_at
    • updated_at
  • sub_alarm
    • id
    • alarm_id
    • function
    • metric_name
    • operator
    • threshold
    • period
    • periods
    • state
    • created_at
    • updated_at
  • sub_alarm_dimension
    • sub_alarm_id
    • dimension_name
    • value

Keystone Requirements

Monasca relies on keystone for running and there are requirements about which keystone configuration must exist.

  • The endpoint for the api must be registered in keystone as the 'monasca' service.
  • The api must have an admin token to use in verifying the keystone tokens it receives.
  • For each project which uses Monasca two users must exist, one will be in the 'monasca-agent' role and be used by the monasca-agent's running on machines. The other should not be in that role and can be used logging into the UI, using the CLI or for direct queries against the API.

Post Metric Sequence

This section describes the sequence of operations involved in posting a metric to the Monasca API.

Monasca Architecture Post Metric Diagram

  1. A metric is posted to the Monasca API.
  2. The Monasca API authenticates and validates the request and publishes the metric to the the Message Queue.
  3. The Persister consumes the metric from the Message Queue and stores in the Metrics Store.
  4. The Transform Engine consumes the metrics from the Message Queue, performs transform and aggregation operations on metrics, and publishes metrics that it creates back to Message Queue.
  5. The Threshold Engine consumes metrics from the Message Queue and evaluates alarms. If a state change occurs in an alarm, an "alarm-state-transitioned-event" is published to the Message Queue.
  6. The Notification Engine consumes "alarm-state-transitioned-events" from the Message Queue, evaluates whether they have a Notification Method associated with it, and sends the appropriate notification, such as email.
  7. The Persister consumes the "alarm-state-transitioned-event" from the Message Queue and stores it in the Alarm State History Store.

Development Environment

Coding Standards


We are in the process of porting to Python and evaluating the performance for the API layer.

Technologies

Uses a number of underlying technologies:

  • Apache Kafka (http://kafka.apache.org): Apache Kafka is publish-subscribe messaging rethought as a distributed commit log. Kafka is a highly performant, distributed, fault-tolerant, and scalable message queue with durability built-in.
  • Apache Storm (http://storm.incubator.apache.org/): Apache Storm is a free and open source distributed realtime computation system. Storm makes it easy to reliably process unbounded streams of data, doing for realtime processing what Hadoop did for batch processing.
  • MySQL:
  • Vagrant (http://www.vagrantup.com/): Vagrant provides easy to configure, reproducible, and portable work environments built on top of industry-standard technology and controlled by a single consistent workflow to help maximize the productivity and flexibility of you and your team.
  • Dropwizard (https://dropwizard.github.io/dropwizard/): Dropwizard pulls together stable, mature libraries from the Java ecosystem into a simple, light-weight package that lets you focus on getting things done. Dropwizard has out-of-the-box support for sophisticated configuration, application metrics, logging, operational tools, and much more, allowing you and your team to ship a production-quality web service in the shortest time possible.
  • InfluxDB (http://influxdb.com/): An open-source distributed time series database with no external dependencies.
  • Vertica (http://www.vertica.com): A commercial Enterprise class SQL analytics database that is highly scalable. It offers built-in automatic high-availability and excels at in-database analytics and compressing and storing massive amounts of data. In the HP Public Cloud we use Vertica in a number of areas such as metrics and many other data streams. Currently, we process around 25 K metrics/sec and store them for > 13 month data retention periods. A free version of Vertica that can store up to 1 TB of data with no time-limit is available at, https://my.vertica.com/community/.

License

Copyright (c) 2014 Hewlett-Packard Development Company, L.P.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

   http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.