Difference between revisions of "Monasca/Logging"
m (→Monasca Ceilometer Plugin should forward Metrics) |
m (→Log Management Backend/Server) |
||
| Line 70: | Line 70: | ||
<br /> | <br /> | ||
| − | + | ==== Sending Data to the Log API ==== | |
We currently consider two possible approaches: | We currently consider two possible approaches: | ||
# We could mimic the Loggly API in order to let end users benefit from the simplicity of the API usage. | # We could mimic the Loggly API in order to let end users benefit from the simplicity of the API usage. | ||
| − | |||
# Or we could aim achieve a high consistency with existing metric API. In the later approach we would simplified speaking copy the existing Monasca | # Or we could aim achieve a high consistency with existing metric API. In the later approach we would simplified speaking copy the existing Monasca | ||
API and tweak it a bit. We would do this intentionally for consistency reasons. As a result, this would be kind of similar to Victors approach, | API and tweak it a bit. We would do this intentionally for consistency reasons. As a result, this would be kind of similar to Victors approach, | ||
| Line 92: | Line 91: | ||
* replace “metric” with “log” | * replace “metric” with “log” | ||
* instead of a “double value” (in the JSON of the request body) we will expect a “string value” in JSON format (must be valid JSON! escape special characters...) | * instead of a “double value” (in the JSON of the request body) we will expect a “string value” in JSON format (must be valid JSON! escape special characters...) | ||
| + | |||
| + | ==== Retrieving Data from the Log API ==== | ||
| + | TODO | ||
Revision as of 14:55, 5 February 2015
This page documents the Monasca Logging solution that is in progress.
Note:
Currently we only have this page to discuss our proposals. For that reason some "metric related stuff" can be found here too. We will move or remove it whenever the discussion is finished.
Contents
Monasca Agent
Additionally needed functionality.
Collect and forward Log Data
The agent should be extended to collect log data and forward them to the Monasca Log API.
Victors approach:
- The (external) statsd-client send logs (as metric) with the metric type ‘l’ to the Monasca-agent.
- The Monasca-agent send these logs as “GAUGE” metric to the forwarder and from there to the Monasca (Metric) API.
DataDog
Note: the Monasca-agent is based on DataDog The DataDog agent can parse metrics and events from logs, so the data within can be send as metric...
Log Collector for the Monasca-Agent
Implement or integrate a Logstash-like collector (e.g. Beaver) in python. Beaver is a lightweight python log file shipper that is used to send logs to an intermediate broker for further processing by Logstash.
OpenStack Hypervisor/VM Monitoring
The Monasca Agent should monitor each created VM (logs and metrics)
- Install agent (automatically) on each created VM.
Metrics as well as logs can be forwarded to Monasca.
The Monasca Agent should monitor the OpenStack Hypervisor (basic VM metrics like CPU, RAM only)
Metrics only!
Sources:
- Nova
Supported by agent?
- KVM
Supported by agent?
- VMware vSphere
vSphere is not supported by agent. A new plugin must be developed (blueprint...).
- VMware ESXi
Probably not supported by agent?
User:
- OpenStack administrator
- OpenStack user (multi tenancy! Can agent add the tenant ID per VM?)
Monasca Ceilometer Plugin should forward Metrics
Metrics only!
Data Flow: Hypervisor --> Nova --> Ceilometer --> Monasca-Ceilometer-Plugin --> Monasca
- KVM
KVM sends metrics to Nova, but Ceilometer (currently?) doesn't poll that data. Monasca-Ceilometer-Plugin is not in "product status", that means no metrics are forwarded to Monasca.
- VMware vSphere
Although Nova provides support for VMware vCenter Server, Ceilometer does not. It does not collect metrics of virtual machines deployed on VMware virtualization platform. It will be necessary to extend this support in Ceilometer. The aim is to implement a polling agent that queries samples from VMware vCenter Server.
Ceilometer/blueprints/vmware-vcenter-server
Log Management Backend/Server
Integrate ELK stack into the existing Monasca architecture.
Log Data Flow
Monasca Log API
!!!! under construction !!!!
Sending Data to the Log API
We currently consider two possible approaches:
- We could mimic the Loggly API in order to let end users benefit from the simplicity of the API usage.
- Or we could aim achieve a high consistency with existing metric API. In the later approach we would simplified speaking copy the existing Monasca
API and tweak it a bit. We would do this intentionally for consistency reasons. As a result, this would be kind of similar to Victors approach, however we would prefer to have a dedicated API in order to let them evolve separately.
Currently, we are thinking about which approach would be better for customers and acceptable for Monasca.
To summarize the two API approaches again:
- Mimicking Loggly API:
- simpler
- authentication via a customer token. Maybe the X-Auth-Token can be used for that purpose.
- no required structure or fields like host, name, dimensions, timestamp (user is responsible and can organize his data for his needs)
- the request body contains only the payload (log entry) and no meta-data (simpler for parsing the payload (special characters)).
- tags could be used to organize/filter logs easily (probably also possible with the Monasca “dimensions”)
- Consistency with Monasca metric API:
- Simple speaking: copy Monasca Metric API
- replace “metric” with “log”
- instead of a “double value” (in the JSON of the request body) we will expect a “string value” in JSON format (must be valid JSON! escape special characters...)
Retrieving Data from the Log API
TODO
