Difference between revisions of "Monasca/Logging"
m |
(→Monasca Log Agent - Logstash: Fix Logstash Output Plugin link) |
||
| (74 intermediate revisions by 4 users not shown) | |||
| Line 1: | Line 1: | ||
This page documents the Monasca Logging solution that is in progress. | This page documents the Monasca Logging solution that is in progress. | ||
| + | == Log Management - Client Side == | ||
| + | === Monasca Log Agent - Logstash === | ||
| + | Monitors one or more log files, adds meta information (e.g. dimensions), authenticates with KeyStone and sends the logs (in a bulk) to the Monasca Log API.<br /> | ||
| + | Base technology: Logstash<br /> | ||
| + | Plugin: https://github.com/logstash-plugins/logstash-output-monasca_log_api<br /> | ||
| − | == Monasca Agent == | + | === Monasca Log Agent - Beaver === |
| − | + | Monitors one file, adds meta information (e.g. dimensions), authenticates with KeyStone and sends the logs (in a bulk) to the Monasca Log API.<br /> | |
| + | https://github.com/python-beaver/python-beaver/pull/406<br /> | ||
| + | https://github.com/python-beaver/python-beaver<br /> | ||
| + | <br /> | ||
| + | <br /> | ||
| − | == | + | == Log Management - Server Side - Consuming Logs == |
| − | === | + | === Monasca Log API === |
| − | + | Consumes logs from the agents, authorizes them and publishes them to Kafka. <br /> | |
| − | + | https://github.com/openstack/monasca-log-api<br /> | |
| − | + | https://github.com/openstack/monasca-log-api/tree/master/docs<br /> | |
| − | |||
| − | |||
| − | |||
| − | |||
| − | + | === Monasca Log Transformer === | |
| − | + | Consumes logs from Kafka, transforms them, and publishes them to Kafka. <br /> | |
| − | |||
| − | === | + | === Monasca Log Persister === |
| − | + | Consumes logs from Kafka, prepares them for bulk storage, and stores them into Elasticsearch.<br /> | |
| + | === Monasca Log Metrics === | ||
| + | Consumes logs from Kafka, creates metrics for logs with severity CRITICAL, ERROR, WARNING, and publishes them to Kafka.<br /> | ||
| + | === Monasca Log Storage=== | ||
| + | All logs are stored in Elasticsearch.<br /> | ||
| + | <br /> | ||
| + | <br /> | ||
| + | == Log Management - Server Side - Visualizing Logs == | ||
| + | === Monasca Kibana Server === | ||
| + | Authorization with KeyStone and visualization of logs (stored in elasticsearch).<br /> | ||
| + | Base technology: Kibana <br /> | ||
| + | Plugins:<br /> | ||
| + | https://github.com/FujitsuEnablingSoftwareTechnologyGmbH/fts-keystone<br /> | ||
| + | https://github.com/FujitsuEnablingSoftwareTechnologyGmbH/keystone-v3-client<br /> | ||
| + | <br /> | ||
| + | <br /> | ||
| + | == Log Data Flow == | ||
| + | The following diagram visualizes the integration of logs in the processing pipeline of Monasca. We indicated some short cuts we want to take as a first step. Also, we indicated some advanced functionality (multi-tenancy) that we plan for the future.<br /> | ||
[[File:DrawIO LogManagement.png|bigpx|center]] | [[File:DrawIO LogManagement.png|bigpx|center]] | ||
Latest revision as of 11:25, 1 August 2017
This page documents the Monasca Logging solution that is in progress.
Contents
Log Management - Client Side
Monasca Log Agent - Logstash
Monitors one or more log files, adds meta information (e.g. dimensions), authenticates with KeyStone and sends the logs (in a bulk) to the Monasca Log API.
Base technology: Logstash
Plugin: https://github.com/logstash-plugins/logstash-output-monasca_log_api
Monasca Log Agent - Beaver
Monitors one file, adds meta information (e.g. dimensions), authenticates with KeyStone and sends the logs (in a bulk) to the Monasca Log API.
https://github.com/python-beaver/python-beaver/pull/406
https://github.com/python-beaver/python-beaver
Log Management - Server Side - Consuming Logs
Monasca Log API
Consumes logs from the agents, authorizes them and publishes them to Kafka.
https://github.com/openstack/monasca-log-api
https://github.com/openstack/monasca-log-api/tree/master/docs
Monasca Log Transformer
Consumes logs from Kafka, transforms them, and publishes them to Kafka.
Monasca Log Persister
Consumes logs from Kafka, prepares them for bulk storage, and stores them into Elasticsearch.
Monasca Log Metrics
Consumes logs from Kafka, creates metrics for logs with severity CRITICAL, ERROR, WARNING, and publishes them to Kafka.
Monasca Log Storage
All logs are stored in Elasticsearch.
Log Management - Server Side - Visualizing Logs
Monasca Kibana Server
Authorization with KeyStone and visualization of logs (stored in elasticsearch).
Base technology: Kibana
Plugins:
https://github.com/FujitsuEnablingSoftwareTechnologyGmbH/fts-keystone
https://github.com/FujitsuEnablingSoftwareTechnologyGmbH/keystone-v3-client
Log Data Flow
The following diagram visualizes the integration of logs in the processing pipeline of Monasca. We indicated some short cuts we want to take as a first step. Also, we indicated some advanced functionality (multi-tenancy) that we plan for the future.
