Difference between revisions of "Monasca/Logging"
(→Metric related and other Stuff) |
m |
||
Line 88: | Line 88: | ||
==== Proposal ==== | ==== Proposal ==== | ||
===== Request Line ===== | ===== Request Line ===== | ||
− | * POST /v2.0/log/single - Endpoint for single and multiline logs; maximum | + | * POST /v2.0/log/single - Endpoint for single and multiline logs; maximum log size: 1MB<br /> |
− | * POST /v2.0/log/bulk - Endpoint for bulk logs (the logs must be line-separated); maximum | + | * POST /v2.0/log/bulk - Endpoint for bulk logs (the logs must be line-separated); maximum log size: 5MB <br /><br /> |
===== Request Headers ===== | ===== Request Headers ===== | ||
Line 99: | Line 99: | ||
'''Tags:'''<br /> | '''Tags:'''<br /> | ||
− | Tags can be added on client side to your | + | Tags can be added on client side to your logs, to structure your logs and helps you later on with filtering (dashboard). |
'''Timestamps:'''<br /> | '''Timestamps:'''<br /> | ||
Line 119: | Line 119: | ||
===== Request Examples ===== | ===== Request Examples ===== | ||
− | '''Single | + | '''Single Log - Plain Text:''' |
POST /v2.0/log/single HTTP/1.1 | POST /v2.0/log/single HTTP/1.1 | ||
Content-Type: text/plain | Content-Type: text/plain | ||
Line 130: | Line 130: | ||
− | '''Single | + | '''Single Log - JSON''' |
POST /v2.0/log/single HTTP/1.1 | POST /v2.0/log/single HTTP/1.1 | ||
Content-Type: application/json | Content-Type: application/json | ||
Line 141: | Line 141: | ||
− | '''Single | + | '''Single Log - Multiline''' |
POST /v2.0/log/single HTTP/1.1 | POST /v2.0/log/single HTTP/1.1 | ||
Content-Type: text/plain | Content-Type: text/plain | ||
Line 152: | Line 152: | ||
− | '''Bulk of | + | '''Bulk of Logs - Plain Text''' |
POST /v2.0/log/bulk HTTP/1.1 | POST /v2.0/log/bulk HTTP/1.1 | ||
Content-Type: text/plain | Content-Type: text/plain | ||
Line 169: | Line 169: | ||
=== Log Transformer === | === Log Transformer === | ||
− | Consumes | + | Consumes logs from Kafka, transforms them, and publishes to Kafka. |
=== Log Persister === | === Log Persister === | ||
− | Consumes | + | Consumes logs from Kafka, prepares them for bulk storage, and stores them into Elasticsearch. |
== Log Management Frontend == | == Log Management Frontend == |
Revision as of 10:35, 11 March 2015
This page documents the Monasca Logging solution that is in progress.
Note:
Currently we only have this page to discuss our proposals. For that reason some "metric related stuff" can be found here too. We will move or remove it whenever the discussion is finished.
Contents
- 1 Log Client
- 2 Log Management Backend
- 3 Log Management Frontend
- 4 Off-Topic. Please ignore
Log Client
Monasca Agent
Collect and forward Log Data
The agent should be extended to collect log data and forward them to the Monasca Log API.
Victors Approach
- The (external) statsd-client send logs (as metric) with the metric type ‘l’ to the Monasca-agent.
- The Monasca-agent send these logs as “GAUGE” metric to the forwarder and from there to the Monasca (Metric) API.
DataDog
Note: The Monasca-agent is based on DataDog.
The DataDog agent can parse metrics and events from logs, so the data within can be send as metric...
Log Collector for the Monasca-Agent (Proposed Solution)
Implement or integrate a Logstash-like collector (e.g. Beaver) in python. Beaver is a lightweight python log file shipper that is used to send logs to an intermediate broker for further processing by Logstash.
Plugins
Logstash Output Plugin
Create a Monasca output plugin for logstash.
Other Output Plugins
TODO e.g. for fluentd...
Log Management Backend
Integrate ELK stack into the existing Monasca architecture. Receiving logs, authentication, processing and storing of logs.
Log Data Flow
The following diagram visualizes the integration of logs in the processing pipeline of Monasca. We indicated some short cuts we want to take as a first step. Also, we indicated some advanced functionality (multi-tennancy) that we plan for the future
Monasca Log API
Background Loggly API
Loggly is a popular SaaS for managing log files. We investigated in Loggly, because we aim for logging-as-a-service functionality (midterm).
Examples:
Plain Text curl -H "content-type:text/plain" -d 'Hello World' http://logs-01.loggly.com/inputs/TOKEN/tag/http/ JSON curl -H "content-type:application/x-www-form-urlencoded" -d '{"message":"Hello World!", "from":"hoover"}' http://logs-01.loggly.com/inputs/TOKEN/tag/http/ Multiline curl -H "content-type:text/plain" -d $'Hello\nWorld' http://logs-01.loggly.com/inputs/TOKEN/tag/http/
Key takeaways:
- simpler
- authentication via a customer token in the URL. Use the X-Auth-Token for that purpose.
- no required structure or fields like host, name, dimensions, timestamp (user is responsible and can organize his data for his needs)
- the request body contains only the payload (log entry) and no meta-data (simpler for parsing the payload (e.g. special characters)).
- tags could be used to organize/filter logs easily (probably also possible with the Monasca “dimensions”)
Background Current Monasca API
Examples:
POST /v2.0/metrics HTTP/1.1 Host: 192.168.10.4:8080 Content-Type: application/json X-Auth-Token: 27feed73a0ce4138934e30d619b415b0 Cache-Control: no-cache { "name":"name1", "dimensions":{ "key1":"value1", "key2":"value2" }, "timestamp":1405630174, "value":1.0 }
Background Event API
There is also an Event API in progress:
Event API
Proposal
Request Line
- POST /v2.0/log/single - Endpoint for single and multiline logs; maximum log size: 1MB
- POST /v2.0/log/bulk - Endpoint for bulk logs (the logs must be line-separated); maximum log size: 5MB
Request Headers
- Content-Type (string, required) - application/json; text/plain
- X-Auth-Token (string, required) - Keystone auth token (for authorization)
- X-Tags: (string, optional) - A comma separated list of tags (for filtering on the dashboard)
- X-Application-Type: (string, optional) - Type of application (to have a hint how to parse)
- X-Application-Name: (string, optional) - Name of the application (to uniquely identify the application log)
Tags:
Tags can be added on client side to your logs, to structure your logs and helps you later on with filtering (dashboard).
Timestamps:
The priority how to determine the timestamp:
1) Try to parse timestamp from original log data
2) If that doesn't work:
- RESTful API: Use receiving time as timestamp
- future version: Syslog API: Take timestamp from syslog
Monasca operates with UTC timezone, that means the timestamp is convert to the concurrent time in UTC.
Will be added by the API:
- X-Request-Received: (long) - The timestamp in UTC when the request was received at the Monasca API
- X-Tenant-ID: (string) - The ID of the tenant after the authentication (via token)
Request Body
The original (potentially unstructured) log data.
Request Examples
Single Log - Plain Text:
POST /v2.0/log/single HTTP/1.1 Content-Type: text/plain X-Auth-Token: 27feed73a0ce4138934e30d619b415b0 X-Tags: production X-Application-Type: apache X-Application-Name: WebServer01 Hello World
Single Log - JSON
POST /v2.0/log/single HTTP/1.1 Content-Type: application/json X-Auth-Token: 27feed73a0ce4138934e30d619b415b0 X-Tags: production X-Application-Type: apache X-Application-Name: WebServer01 {"message":"Hello World!", "from":"hoover"}
Single Log - Multiline
POST /v2.0/log/single HTTP/1.1 Content-Type: text/plain X-Auth-Token: 27feed73a0ce4138934e30d619b415b0 X-Tags: production X-Application-Type: apache X-Application-Name: WebServer01 Hello\nWorld
Bulk of Logs - Plain Text
POST /v2.0/log/bulk HTTP/1.1 Content-Type: text/plain X-Auth-Token: 27feed73a0ce4138934e30d619b415b0 X-Tags: production X-Application-Type: apache X-Application-Name: WebServer01 Hello\nWorld
Response Status Code
204 - No Content
Response Body
This request does not return a response body.
Log Transformer
Consumes logs from Kafka, transforms them, and publishes to Kafka.
Log Persister
Consumes logs from Kafka, prepares them for bulk storage, and stores them into Elasticsearch.
Log Management Frontend
Monasca Log API
Visualization of logs.
TODO
Off-Topic. Please ignore
Tags
Introduce "tags" like in the proposal of the Log API. The tags will be needed for the container monitoring where the workload/applications are very dynamic and are not related to a static host anymore (e.g. kubernetes/docker cluster).
Monasca Agent
Additionally needed functionality.
The Monasca Agent should monitor the OpenStack Hypervisor (basic VM metrics like CPU, RAM only)
Sources:
- Nova
Supported by agent?
- KVM
Supported by agent?
- VMware vSphere
vSphere is not supported by agent. A new plugin must be developed (blueprint...).
- VMware ESXi
Probably not supported by agent?
User:
- OpenStack administrator
- OpenStack user (multi tenancy! Can agent add the tenant ID per VM?)
Monasca Ceilometer Plugin should forward Metrics
Data Flow: Hypervisor --> Nova --> Ceilometer --> Monasca-Ceilometer-Plugin --> Monasca
- KVM
KVM sends metrics to Nova, but Ceilometer (currently?) doesn't poll that data. Monasca-Ceilometer-Plugin is not in "product status", that means no metrics are forwarded to Monasca.
- VMware vSphere
Although Nova provides support for VMware vCenter Server, Ceilometer does not. It does not collect metrics of virtual machines deployed on VMware virtualization platform. It will be necessary to extend this support in Ceilometer. The aim is to implement a polling agent that queries samples from VMware vCenter Server.
Ceilometer/blueprints/vmware-vcenter-server
OpenStack Hypervisor/VM Monitoring
The Monasca Agent should monitor each created VM (logs and metrics)
- Install agent (automatically) on each created VM.
Metrics as well as logs can be forwarded to Monasca.
Windows Support
The Monasca agent should run on Windows also.
Note: The Monasca-agent is based on DataDog. DataDog supports Windows since November 9th, 2012, but currently the Monasca agent runs on Ubuntu only.