Jump to: navigation, search

Difference between revisions of "Monasca/Logging"

(Request Examples)
Line 119: Line 119:
  
 
===== Request Examples =====
 
===== Request Examples =====
'''TODO'''<br />
 
  
 +
'''Single Event - Plain Text:'''
 +
POST /v2.0/log/single HTTP/1.1
 +
Content-Type: text/plain
 +
X-Auth-Token: 27feed73a0ce4138934e30d619b415b0
 +
X-Tags: production
 +
X-Application-Type: apache
 +
X-Application-Name: WebServer01
 +
 +
Hello World
 +
 +
 +
'''Single Event - JSON'''
 
  POST /v2.0/log/single HTTP/1.1
 
  POST /v2.0/log/single HTTP/1.1
 
  Content-Type: application/json
 
  Content-Type: application/json
Line 127: Line 138:
 
  X-Application-Type: apache
 
  X-Application-Type: apache
 
  X-Application-Name: WebServer01
 
  X-Application-Name: WebServer01
 +
 +
{"message":"Hello World!", "from":"hoover"}
 +
 +
 +
'''Single Event - Multiline'''
 +
POST /v2.0/log/single HTTP/1.1
 +
Content-Type: text/plain
 +
X-Auth-Token: 27feed73a0ce4138934e30d619b415b0
 +
X-Tags: production
 +
X-Application-Type: apache
 +
X-Application-Name: WebServer01
 +
 +
Hello\nWorld
 +
 +
 +
'''Bulk of Events - Plain Text'''
 +
POST /v2.0/log/bulk HTTP/1.1
 +
Content-Type: text/plain
 +
X-Auth-Token: 27feed73a0ce4138934e30d619b415b0
 +
X-Tags: production
 +
X-Application-Type: apache
 +
X-Application-Name: WebServer01
 +
 +
Hello\nWorld
  
TODO...
 
 
===== Response Status Code =====
 
===== Response Status Code =====
 
204 - No Content
 
204 - No Content

Revision as of 15:36, 10 February 2015

This page documents the Monasca Logging solution that is in progress.



Note:
Currently we only have this page to discuss our proposals. For that reason some "metric related stuff" can be found here too. We will move or remove it whenever the discussion is finished.

Log Client

Monasca Agent

Collect and forward Log Data

The agent should be extended to collect log data and forward them to the Monasca Log API.

Victors Approach
  • The (external) statsd-client send logs (as metric) with the metric type ‘l’ to the Monasca-agent.
  • The Monasca-agent send these logs as “GAUGE” metric to the forwarder and from there to the Monasca (Metric) API.
DataDog

Note: The Monasca-agent is based on DataDog.

The DataDog agent can parse metrics and events from logs, so the data within can be send as metric...

Log Collector for the Monasca-Agent (Proposed Solution)

Implement or integrate a Logstash-like collector (e.g. Beaver) in python. Beaver is a lightweight python log file shipper that is used to send logs to an intermediate broker for further processing by Logstash.

Plugins

Logstash Output Plugin

Create a Monasca output plugin for logstash.

Other Output Plugins

TODO e.g. for fluentd...

Log Management Backend

Integrate ELK stack into the existing Monasca architecture. Receiving logs, authentication, processing and storing of logs.

Log Data Flow

bigpx

Monasca Log API

Background Loggly API

We could mimic the Loggly API in order to let end users benefit from the simplicity of the API usage.

Examples:

Plaintext
curl -H "content-type:text/plain" -d 'Hello World' http://logs-01.loggly.com/inputs/TOKEN/tag/http/

JSON
curl -H "content-type:application/x-www-form-urlencoded" -d '{"message":"hello world!", "from":"hoover"}' http://logs-01.loggly.com/inputs/TOKEN/tag/http/ 

Multiline
curl -H "content-type:text/plain" -d $'Hello\nWorld' http://logs-01.loggly.com/inputs/TOKEN/tag/http/
  • simpler
  • authentication via a customer token in the URL. Use the X-Auth-Token for that purpose.
  • no required structure or fields like host, name, dimensions, timestamp (user is responsible and can organize his data for his needs)
  • the request body contains only the payload (log entry) and no meta-data (simpler for parsing the payload (e.g. special characters)).
  • tags could be used to organize/filter logs easily (probably also possible with the Monasca “dimensions”)

Background Current Monasca API

We could aim to achieve a high consistency with existing metric API. In the later approach we would simplified speaking copy the existing Monasca API and tweak it a bit. We would do this intentionally for consistency reasons. As a result, this would be kind of similar to Victors approach, however we would prefer to have a dedicated API in order to let them evolve separately.

Examples:

POST /v2.0/metrics HTTP/1.1
Host: 192.168.10.4:8080
Content-Type: application/json
X-Auth-Token: 27feed73a0ce4138934e30d619b415b0
Cache-Control: no-cache

{  
   "name":"name1",
   "dimensions":{  
      "key1":"value1",
      "key2":"value2"
   },
   "timestamp":1405630174,
   "value":1.0
}
  • Simple speaking: copy Monasca Metric API
  • replace “metric” with “log”
  • instead of a “double value” (in the JSON structure of the request body) we will expect a “string value” in JSON format (must be valid JSON! escape special characters...)

Background Event API

There is also an Event API in progress:
Event API

Proposal

Request Line
  • POST /v2.0/log/single - Endpoint for single and multiline logs; maximum event size: 1MB
  • POST /v2.0/log/bulk - Endpoint for bulk logs (the logs must be line-separated); maximum event size: 5MB

Request Headers
  • Content-Type (string, required) - application/json; text/plain
  • X-Auth-Token (string, required) - Keystone auth token (for authorization)
  • X-Tags: (string, optional) - A comma separated list of tags (for filtering on the dashboard)
  • X-Application-Type: (string, optional) - Type of application (to have a hint how to parse)
  • X-Application-Name: (string, optional) - Name of the application (to uniquely identify the application log)

Tags:
Tags can be added on client side to your log event, to structure your log events and helps you later on with filtering (dashboard).

Timestamps:
The priority how to determine the timestamp:
1) Try to parse timestamp from log event (value)
2) If that doesn't work:
RESTful API: Use receiving time as timestamp
future version: Syslog API: Take timestamp from syslog
Monasca operates with UTC timezone, that means the timestamp is convert to the concurrent time in UTC.

Will be added by the API:

  • X-Request-Received: (long) - The timestamp in UTC when the request was received at the Monasca API
  • X-Tenant-ID: (string) - The ID of the tenant after the authentication (via token)
Request Body

The original (potentially unstructured) log data.

Request Examples

Single Event - Plain Text:

POST /v2.0/log/single HTTP/1.1
Content-Type: text/plain
X-Auth-Token: 27feed73a0ce4138934e30d619b415b0
X-Tags: production
X-Application-Type: apache
X-Application-Name: WebServer01

Hello World


Single Event - JSON

POST /v2.0/log/single HTTP/1.1
Content-Type: application/json
X-Auth-Token: 27feed73a0ce4138934e30d619b415b0
X-Tags: production
X-Application-Type: apache
X-Application-Name: WebServer01

{"message":"Hello World!", "from":"hoover"}


Single Event - Multiline

POST /v2.0/log/single HTTP/1.1
Content-Type: text/plain
X-Auth-Token: 27feed73a0ce4138934e30d619b415b0
X-Tags: production
X-Application-Type: apache
X-Application-Name: WebServer01

Hello\nWorld


Bulk of Events - Plain Text

POST /v2.0/log/bulk HTTP/1.1
Content-Type: text/plain
X-Auth-Token: 27feed73a0ce4138934e30d619b415b0
X-Tags: production
X-Application-Type: apache
X-Application-Name: WebServer01

Hello\nWorld
Response Status Code

204 - No Content

Response Body

This request does not return a response body.

Log Transformer

Consumes events from Kafka, transforms them, and publishes to Kafka.

Log Persister

Consumes events from Kafka, prepares them for bulk storage, and stores them into Elasticsearch.

Log Management Frontend

Monasca Log API

Visualization of logs.

TODO




Metric related and other Stuff

Tags

Introduce "tags" like in the proposal of the Log API. The tags will be needed for the container monitoring where the workload/applications are very dynamic and are not related to a static host anymore (e.g. kubernetes/docker cluster).

Monasca Agent

Additionally needed functionality.

The Monasca Agent should monitor the OpenStack Hypervisor (basic VM metrics like CPU, RAM only)

Sources:

  • Nova

Supported by agent?

  • KVM

Supported by agent?

  • VMware vSphere

vSphere is not supported by agent. A new plugin must be developed (blueprint...).

  • VMware ESXi

Probably not supported by agent?

User:

  • OpenStack administrator
  • OpenStack user (multi tenancy! Can agent add the tenant ID per VM?)

Monasca Ceilometer Plugin should forward Metrics

Data Flow: Hypervisor --> Nova --> Ceilometer --> Monasca-Ceilometer-Plugin --> Monasca

  • KVM

KVM sends metrics to Nova, but Ceilometer (currently?) doesn't poll that data. Monasca-Ceilometer-Plugin is not in "product status", that means no metrics are forwarded to Monasca.

  • VMware vSphere

Although Nova provides support for VMware vCenter Server, Ceilometer does not. It does not collect metrics of virtual machines deployed on VMware virtualization platform. It will be necessary to extend this support in Ceilometer. The aim is to implement a polling agent that queries samples from VMware vCenter Server.
Ceilometer/blueprints/vmware-vcenter-server

OpenStack Hypervisor/VM Monitoring

The Monasca Agent should monitor each created VM (logs and metrics)
  • Install agent (automatically) on each created VM.

Metrics as well as logs can be forwarded to Monasca.

Windows Support

The Monasca agent should run on Windows also.

Note: The Monasca-agent is based on DataDog. DataDog supports Windows since November 9th, 2012, but currently the Monasca agent runs on Ubuntu only.