Revision as of 11:58, 27 September 2017

Distributed Monitoring

Overview

Monitoring and its application are becoming key factor for service lifecycle management of various systems such as NFV and cloud native platform. Distributed monitoring approach is one of the framework which enables flexible and scalable monitoring that can work with current OpenStack telemetry and monitoring framework. In this documantation, you can find the architecture and how to implement this framework to your environment.

Who will be interested in ?

• For infrastracture operators who want to collect detailed data in short intarval in their compute nodes.
• For NFV operators who want to know abnormal behaivers of Virtual Network Functions.

Architecture

In this architecture includes several functions for monitoring, collector, in-memory database, analysis engine and notification. Below picture shows the architecture of distributed monitoring. Each compute node has it own monitoring function following this architecture.

Function

• Poller/Notification process collect data from guest OSs and host OS using SNMP protocol, libvirt API, OpenStack API, etc.
• Collector format data suitable for in-memory database and insert these data into database.
• Analytics Engine analyzes data on in-memory database, you can use several analytics engine like machine learning libraries. And also you can use evaluator that function is threshold monitoring directly.
• Transmitter send analytics results and alarms that are caught on threshold to Operation Support System, OpenStack API and Orchestrator.

Feature

• Short interval
  • You can collect several data in short interval like 0.1 sec. Because collect agent doesn't have to collect data from huge amount of computing nodes and VMs in this architecture. That means the load of agent become lower than centralized monitoring architecture.
• Scalable
  • This architecture has high scalability. Because each compute node has it own monitoring function, so you don't have to caluculate specs of nodes for monitoring.
• Fast detection
  • In some case, MQ is bottlneck of performances. But any MQ isn't used in this architecture, because monitoring process is closed in compute node.

Use Cases

• Micro Burst Traffic
• Memory Leak
• Abnormal behaviour of software/hardware
• ...

Implementation

For distributed monitoring, some open source software are useful. ... You can learn an example of how to implement distributed monitoring with open source software.

OSS

collectd

collectd is a daemon which collects system and application performance metrics periodically and provides mechanisms to store the values in a variety of ways. You can use collectd as Poller/Notification to gather metrics and Collector to store the metrics in Redis. You can implement Analytics Engine and Transmitter as collectd's plugins.

collectd's official page

Redis

Redis is an in-memory data structure store, used as a database, cache and message broker.

Redis's official page

scikit-learn

scikit-learn is simple and efficient tools for data mining and data analysis. You can use scikit-learn as light weight machine learning library for Analytics Engine.

scikit-learn's official page

Setup

(to be edited) In computing nodes, controller nods and every nodes that you want to monitor, you can setup following example. In this example, ubuntu16.04 is selected as each node's OS.

1. set up OpenStack environment using DevStack or manual installation
2. install collectd, redis and some other python library
3. get demo code of DMA

collectd

You can implement distributed monitoring by setting collectd and making collectd's plugins. See distributed monitoring github repository if you want to see an example .

Poller

collectd has plugins to collect standard metrics such as cpu/memory/disk/network utiliziation. See collectd's official page.

Collector

(to be edited)

<Plugin "write_redis">
  <Node "dma">
    Host "localhost"
    Port "6379"
    Timeout 1000
  </Node>
</Plugin>

Analytics Engine

Make analytics plugin with python and scikit learn. (to be edited)

<Plugin python>
  ModulePath "/opt/dma/lib"
  LogTraces true
  Interactive false
  Import "analysis"

  <Module "analysis">
  </Module>
</Plugin>

Transmitter

Make transimit plugin with python. (to be edited)

<Plugin "threshold">
  <Host "localhost">
    <Plugin "dma">
      <Type "gauge">
        WarningMax 0
        Hits 3
      </Type>
    </Plugin>
  </Host>
</Plugin>

<Plugin python>
   ModulePath "/opt/dma/lib"
     LogTraces true
     Interactive false
     Import "write_openstack"

     <Module "write_openstack">
     </Module>
</Plugin>

Redis

We recommend that you disable saving data of Redis DB on disk.

redis.conf

#save 900 1
#save 300 10
#save 60 10000
save ""

References

• OpenStack Summit Balacerona
  • Video
    • https://www.openstack.org/videos/barcelona-2016/nec-the-telecom-requirements-for-openstack-how-to-reduce-operation-cost-after-day-2
• OpenStack Summit Boston
  • Video
    • https://www.openstack.org/videos/boston-2017/distributed-monitoring-and-analysis-for-telecom-requirements
  • Slide
    • https://www.openstack.org/assets/presentation-media/DMA-OpenStackSummit-Boston.pdf
• OpenStack Summit Sydney
  • https://www.openstack.org/summit/sydney-2017/summit-schedule/events/19178/dmadistributed-monitoring-and-analysis-monitoring-practice-and-lifecycle-management-for-telecom

Who is contributing to this guide?