Difference between revisions of "Monasca/Logging"

Latest revision as of 11:25, 1 August 2017

This page documents the Monasca Logging solution that is in progress.

Log Management - Client Side

Monasca Log Agent - Logstash

Monitors one or more log files, adds meta information (e.g. dimensions), authenticates with KeyStone and sends the logs (in a bulk) to the Monasca Log API.
Base technology: Logstash
Plugin: https://github.com/logstash-plugins/logstash-output-monasca_log_api

Monasca Log Agent - Beaver

Monitors one file, adds meta information (e.g. dimensions), authenticates with KeyStone and sends the logs (in a bulk) to the Monasca Log API.
https://github.com/python-beaver/python-beaver/pull/406
https://github.com/python-beaver/python-beaver

Log Management - Server Side - Consuming Logs

Monasca Log API

Consumes logs from the agents, authorizes them and publishes them to Kafka.
https://github.com/openstack/monasca-log-api
https://github.com/openstack/monasca-log-api/tree/master/docs

Monasca Log Transformer

Consumes logs from Kafka, transforms them, and publishes them to Kafka.

Monasca Log Persister

Consumes logs from Kafka, prepares them for bulk storage, and stores them into Elasticsearch.

Monasca Log Metrics

Consumes logs from Kafka, creates metrics for logs with severity CRITICAL, ERROR, WARNING, and publishes them to Kafka.

Monasca Log Storage

All logs are stored in Elasticsearch.

Log Management - Server Side - Visualizing Logs

Monasca Kibana Server

Authorization with KeyStone and visualization of logs (stored in elasticsearch).
Base technology: Kibana
Plugins:
https://github.com/FujitsuEnablingSoftwareTechnologyGmbH/fts-keystone
https://github.com/FujitsuEnablingSoftwareTechnologyGmbH/keystone-v3-client

Log Data Flow

The following diagram visualizes the integration of logs in the processing pipeline of Monasca. We indicated some short cuts we want to take as a first step. Also, we indicated some advanced functionality (multi-tenancy) that we plan for the future.

@@ Line 1: / Line 1: @@
 This page documents the Monasca Logging solution that is in progress.
+== Log Management - Client Side ==
+=== Monasca Log Agent - Logstash ===
+Monitors one or more log files, adds meta information (e.g. dimensions), authenticates with KeyStone and sends the logs (in a bulk) to the Monasca Log API.<br />
+Base technology: Logstash<br />
+Plugin: https://github.com/logstash-plugins/logstash-output-monasca_log_api<br />
-== Monasca Agent ==
+=== Monasca Log Agent - Beaver ===
-Needed functionality.
+Monitors one file, adds meta information (e.g. dimensions), authenticates with KeyStone and sends the logs (in a bulk) to the Monasca Log API.<br />
+https://github.com/python-beaver/python-beaver/pull/406<br />
+https://github.com/python-beaver/python-beaver<br />
+<br />
+<br />
-=== OpenStack Hypervisor Monitoring ===
+== Log Management - Server Side - Consuming Logs ==
-==== The agent should monitor each created VM (in detail) ====
+=== Monasca Log API ===
-* Install agent automatically on each created VM.
+Consumes logs from the agents, authorizes them and publishes them to Kafka. <br />
-==== The agent should monitor each created VM (basic metrics like CPU, RAM) ====
+https://github.com/openstack/monasca-log-api<br />
-Sources:
+https://github.com/openstack/monasca-log-api/tree/master/docs<br />
-* Nova
-  Supported by Agent?
-* KVM
-Supported by Agent?
-* VMware vSphere
-vSphere is not currently supported by Agent. A new plugin must be developed (blueprint...).
-* VMware ESXi
-Probably not supported by Agent?
-User:
-* OpenStack administrator
-* OpenStack user (multi tenancy! Can agent add the tenant ID per VM?)
-==== Hypervisor --> Nova --> Ceilometer --> Monasca-ceilometer-plugin --> Monasca ====
+=== Monasca Log Transformer ===
-* KVM
+Consumes logs from Kafka, transforms them, and publishes them to Kafka. <br />
-KVM sends data to Nova, but Ceilometer doesn't (currently?) poll that data. Monasca-ceilometer-plugin is not in "product status".
-* VMware vSphere
+=== Monasca Log Persister ===
-Although Nova provides support for VMware vCenter Server, Ceilometer does not. It does not collect metrics of virtual machines deployed on VMware virtualization platform. It will be necessary to extend this support in Ceilometer. The aim is to implement a polling agent that queries samples from VMware vCenter Server
+Consumes logs from Kafka, prepares them for bulk storage, and stores them into Elasticsearch.<br />
-[https://wiki.openstack.org/wiki/Ceilometer/blueprints/vmware-vcenter-server Ceilometer/blueprints/vmware-vcenter-server]
+=== Monasca Log Metrics ===
+Consumes logs from Kafka, creates metrics for logs with severity CRITICAL, ERROR, WARNING, and publishes them to Kafka.<br />
-=== Collect and forward Log Data ===
+=== Monasca Log Storage===
-The agent should be extended to collect log data and forward them to the Monasca Log API.
+All logs are stored in Elasticsearch.<br />
+<br />
+<br />
+== Log Management - Server Side - Visualizing Logs ==
+=== Monasca  Kibana Server ===
+Authorization with KeyStone and visualization of logs (stored in elasticsearch).<br />
+Base technology: Kibana <br />
+Plugins:<br />
+https://github.com/FujitsuEnablingSoftwareTechnologyGmbH/fts-keystone<br />
+https://github.com/FujitsuEnablingSoftwareTechnologyGmbH/keystone-v3-client<br />
+<br />
+<br />
-== Log Management Backend/Server ==
+== Log Data Flow ==
-Integrate ELK stack into the existing Monasca Architecture.
+The following diagram visualizes the integration of logs in the processing pipeline of Monasca. We indicated some short cuts we want to take as a first step. Also, we indicated some advanced functionality (multi-tenancy) that we plan for the future.<br />
-=== Log Data Flow ===
 [[File:DrawIO LogManagement.png|bigpx|center]]
-=== Monasca Log API ===