Difference between revisions of "Monasca/Logging"

Latest revision as of 11:25, 1 August 2017

This page documents the Monasca Logging solution that is in progress.

Log Management - Client Side

Monasca Log Agent - Logstash

Monitors one or more log files, adds meta information (e.g. dimensions), authenticates with KeyStone and sends the logs (in a bulk) to the Monasca Log API.
Base technology: Logstash
Plugin: https://github.com/logstash-plugins/logstash-output-monasca_log_api

Monasca Log Agent - Beaver

Monitors one file, adds meta information (e.g. dimensions), authenticates with KeyStone and sends the logs (in a bulk) to the Monasca Log API.
https://github.com/python-beaver/python-beaver/pull/406
https://github.com/python-beaver/python-beaver

Log Management - Server Side - Consuming Logs

Monasca Log API

Consumes logs from the agents, authorizes them and publishes them to Kafka.
https://github.com/openstack/monasca-log-api
https://github.com/openstack/monasca-log-api/tree/master/docs

Monasca Log Transformer

Consumes logs from Kafka, transforms them, and publishes them to Kafka.

Monasca Log Persister

Consumes logs from Kafka, prepares them for bulk storage, and stores them into Elasticsearch.

Monasca Log Metrics

Consumes logs from Kafka, creates metrics for logs with severity CRITICAL, ERROR, WARNING, and publishes them to Kafka.

Monasca Log Storage

All logs are stored in Elasticsearch.

Log Management - Server Side - Visualizing Logs

Monasca Kibana Server

Authorization with KeyStone and visualization of logs (stored in elasticsearch).
Base technology: Kibana
Plugins:
https://github.com/FujitsuEnablingSoftwareTechnologyGmbH/fts-keystone
https://github.com/FujitsuEnablingSoftwareTechnologyGmbH/keystone-v3-client

Log Data Flow

The following diagram visualizes the integration of logs in the processing pipeline of Monasca. We indicated some short cuts we want to take as a first step. Also, we indicated some advanced functionality (multi-tenancy) that we plan for the future.

@@ Line 1: / Line 1: @@
 This page documents the Monasca Logging solution that is in progress.
+== Log Management - Client Side ==
+=== Monasca Log Agent - Logstash ===
+Monitors one or more log files, adds meta information (e.g. dimensions), authenticates with KeyStone and sends the logs (in a bulk) to the Monasca Log API.<br />
+Base technology: Logstash<br />
+Plugin: https://github.com/logstash-plugins/logstash-output-monasca_log_api<br />
+=== Monasca Log Agent - Beaver ===
+Monitors one file, adds meta information (e.g. dimensions), authenticates with KeyStone and sends the logs (in a bulk) to the Monasca Log API.<br />
+https://github.com/python-beaver/python-beaver/pull/406<br />
+https://github.com/python-beaver/python-beaver<br />
+<br />
 <br />
-'''Note:'''<br />
-'''Currently we only have this page to discuss our proposals. For that reason some "metric related stuff" can be found here too. We will move or remove it whenever the discussion is finished.'''<br /><br />
-== Monasca Agent ==
+== Log Management - Server Side - Consuming Logs ==
-Additionally needed functionality.
+=== Monasca Log API ===
+Consumes logs from the agents, authorizes them and publishes them to Kafka. <br />
-=== Collect and forward Log Data ===
+https://github.com/openstack/monasca-log-api<br />
-The agent should be extended to collect log data and forward them to the Monasca Log API.
+https://github.com/openstack/monasca-log-api/tree/master/docs<br />
-==== Victors Approach ====
-* The (external) statsd-client send logs (as metric) with the metric type ‘l’ to the Monasca-agent.
-* The Monasca-agent send these logs as “GAUGE” metric to the forwarder and from there to the Monasca (Metric) API.
-==== DataDog ====
-Note: The Monasca-agent is based on DataDog.<br />
-The DataDog agent can parse metrics and events from logs, so the data within can be send as metric...
-==== Log Collector for the Monasca-Agent (Proposed Solution) ====
-Implement or integrate a Logstash-like collector (e.g. Beaver) in python. Beaver is a lightweight python log file shipper that is used to send logs to an intermediate broker for further processing by Logstash.
-=== OpenStack Hypervisor/VM Monitoring ===
-==== The Monasca Agent should monitor each created VM (logs and metrics) ====
-* Install agent (automatically) on each created VM.
-Metrics as well as logs can be forwarded to Monasca.
-=== Windows Support ===
-The Monasca agent should run on Windows also.<br />
-Note: The Monasca-agent is based on DataDog.<br />
-DataDog supports Windows since November 9th, 2012, but currently the Monasca agent runs on Ubuntu only.
-<br /><br />
-== Log Management Backend ==
-Integrate ELK stack into the existing Monasca architecture.
-Receiving logs, authentication, processing and storing of logs.
-=== Log Data Flow ===
-[[File:DrawIO LogManagement.png|bigpx|center]]
-=== Monasca Log API ===
+=== Monasca Log Transformer ===
-We currently consider two possible approaches: <br />
+Consumes logs from Kafka, transforms them, and publishes them to Kafka. <br />
-) We could mimic the Loggly API in order to let end users benefit from the simplicity of the API usage. <br />
-) Or we could aim to achieve a high consistency with existing metric API. In the later approach we would simplified speaking copy the existing Monasca API and tweak it a bit. We would do this intentionally for consistency reasons. As a result, this would be kind of similar to Victors approach, however we would prefer to have a dedicated API in order to let them evolve separately.<br /><br />
+=== Monasca Log Persister ===
+Consumes logs from Kafka, prepares them for bulk storage, and stores them into Elasticsearch.<br />
-To summarize the two API approaches again:<br />
+=== Monasca Log Metrics ===
-) Mimicking Loggly API:<br />
+Consumes logs from Kafka, creates metrics for logs with severity CRITICAL, ERROR, WARNING, and publishes them to Kafka.<br />
- '''Plaintext'''
- curl -H "content-type:text/plain" -d ''''Hello World'''' http://logs-01.loggly.com/inputs/TOKEN/tag/http/
- '''JSON'''
- curl -H "content-type:application/x-www-form-urlencoded" -d ''''{"message":"hello world!", "from":"hoover"}'''' http://logs-01.loggly.com/inputs/TOKEN/tag/http/
- '''Multiline'''
- curl -H "content-type:text/plain" -d $''''Hello\nWorld'''' http://logs-01.loggly.com/inputs/TOKEN/tag/http/
-* simpler
+=== Monasca Log Storage===
-* authentication via a customer token in the URL. Maybe the X-Auth-Token can be used for that purpose.
+All logs are stored in Elasticsearch.<br />
-* no required structure or fields like host, name, dimensions, timestamp (user is responsible and can organize his data for his needs)
-* the request body contains only the payload (log entry) and no meta-data (simpler for parsing the payload (e.g. special characters)).
-* tags could be used to organize/filter logs easily (probably also possible with the Monasca “dimensions”)<br />
 <br />
-) Consistency with Monasca metric API:<br />
- POST /v2.0/metrics HTTP/1.1
- Host: 192.168.10.4:8080
- Content-Type: application/json
- X-Auth-Token: 27feed73a0ce4138934e30d619b415b0
- Cache-Control: no-cache
- {
-    "name":"name1",
-    "dimensions":{
-       "key1":"value1",
-       "key2":"value2"
-    },
-    "timestamp":1405630174,
-    "value":'''1.0'''
- }
-* Simple speaking: copy Monasca Metric API
-* replace “metric” with “log”
-* instead of a “double value” (in the JSON structure of the request body) we will expect a “string value” in JSON format (must be valid JSON! escape special characters...)<br />
 <br />
-'''Currently, we are thinking about which approach would be better for customers and acceptable for Monasca.'''<br /><br />
+== Log Management - Server Side - Visualizing Logs ==
+=== Monasca  Kibana Server ===
-== Log Management Frontend ==
+Authorization with KeyStone and visualization of logs (stored in elasticsearch).<br />
-=== Monasca Log API ===
+Base technology: Kibana <br />
-Visualization of logs.<br />
+Plugins:<br />
-'''TODO'''<br />
+https://github.com/FujitsuEnablingSoftwareTechnologyGmbH/fts-keystone<br />
-<br />
+https://github.com/FujitsuEnablingSoftwareTechnologyGmbH/keystone-v3-client<br />
-<br />
 <br />
 <br />
-== Metric related Stuff ==
+== Log Data Flow ==
-=== Monasca Agent ===
+The following diagram visualizes the integration of logs in the processing pipeline of Monasca. We indicated some short cuts we want to take as a first step. Also, we indicated some advanced functionality (multi-tenancy) that we plan for the future.<br />
-Additionally needed functionality.
-==== The Monasca Agent should monitor the OpenStack Hypervisor (basic VM metrics like CPU, RAM only) ====
+[[File:DrawIO LogManagement.png|bigpx|center]]
-Sources:
-* Nova
-Supported by agent?
-* KVM
-Supported by agent?
-* VMware vSphere
-vSphere is not supported by agent. A new plugin must be developed (blueprint...).
-* VMware ESXi
-Probably not supported by agent?
-User:
-* OpenStack administrator
-* OpenStack user (multi tenancy! Can agent add the tenant ID per VM?)
-==== Monasca Ceilometer Plugin should forward Metrics ====
-Data Flow:
-Hypervisor --> Nova --> Ceilometer --> Monasca-Ceilometer-Plugin --> Monasca
-* KVM
-KVM sends metrics to Nova, but Ceilometer (currently?) doesn't poll that data. Monasca-Ceilometer-Plugin is not in "product status", that means no metrics are forwarded to Monasca.
-* VMware vSphere
-Although Nova provides support for VMware vCenter Server, Ceilometer does not. It does not collect metrics of virtual machines deployed on VMware virtualization platform. It will be necessary to extend this support in Ceilometer. The aim is to implement a polling agent that queries samples from VMware vCenter Server.<br />
-[https://wiki.openstack.org/wiki/Ceilometer/blueprints/vmware-vcenter-server Ceilometer/blueprints/vmware-vcenter-server]
-<br />