Jump to: navigation, search

Difference between revisions of "Trove/DBInstanceLogOperationV1"

(Why does Trove needs it ?)
(API Calls)
Line 89: Line 89:
 
# Look over assigned configuration for log configuration mentioned in datastore-specific mapping
 
# Look over assigned configuration for log configuration mentioned in datastore-specific mapping
 
# Return every logs enabled by the assigned configuration
 
# Return every logs enabled by the assigned configuration
 +
 +
If none of logs are enabled by the configuration - user will obtain list that contains nothing.
 +
  
 
==== Create and save database logging file entry. HTTP method POST====
 
==== Create and save database logging file entry. HTTP method POST====

Revision as of 18:19, 21 July 2014

Mission

Provide specific API interface to end-user which would allow to manipulate with database log files. This feature provides the ability to access log files via Swift so the can download them for Auditing/Troubleshooting purposes.

Description

Log manipulations are designed to let user perform log investigations. Since Trove is PaaS - level project, it's user cannot interact with compute instance directly, only with database through given API (database operations).

Justification/Benefits

Justification

Perfomance tuning based on log file analyze.

Database throughput is always limited by the mathimatical equations based upon available resources, after reaching defined limit database starts throwing exceptions that are logged in general log(cassandra system.log, redis server log) or special error log (mysql, percona, maria, etc.).

Database startup Issues resolving based upon error/general log analyz.

Example: Mysql. The error log contains information indicating when mysqld was started and stopped and also any critical errors that occur while the server is running. If mysqld notices a table that needs to be automatically checked or repaired, it writes a message to the error log. Cassandra. Heap size errors based on automated memory allocation while database service launching (http://stackoverflow.com/questions/16243434/getting-outofmemory-in-cassandra-java-heap-space-version-1-0-7-frequently-on-dif).
The minimum requirements set forth in the “general overview and risk assessment” section below must be completed for the audit to qualify for core audit coverage. Following completion of the general overview and risk assessment, the auditor will use professional judgment to select specific areas for additional focus and audit testing.

Benefits

From the user perspective, this feature completely covers real-worl use cases mentioned in justification section (management, perfomance tuning, audit, etc.)

Impacts

Thi feature would not affect/break current Trove API. It changes the attitute to the actual Trove instance from the simple database server with connection URL to something bigger. It affects in-accessability of the instance, that is restricted by the terms of use of the public/private cloud and affects the PaaS term.

Database

Database changes are not required because database logs are not tracked at the Trove backend as a resource that can be re-used in future.

Configuration

None

Public API

New routes are presented. Description mentioned below.

Public API Details

Three new resources, log-create, log-show will be exposed as part of the Trove API.
The log-show is used to provide an ability to list all available(availability defined by Trove) database logging filenames per instance.
The log-create is used to provide an ability to save database logging file into the Swift container, required attribute - instance.


To implement this capability, the create/modify/list instance operations will be extended in a manner that does not break the existing 1.0 contract. These operations will permit a user to create a new database logging file entry for already existed instance, list all available database logging filenames for all registered datstore versions (basically, for all datastore managers across all versions), show all available database logging filenames per certain datastore version (manager).

API Calls


Get the list of all available database logging files per given datastore version. HTTP method GET

Route: /{tenant_id}/instances/{id}/logs

Response

   {
       "instance": {
                          {
                             "log_files": [
                                                    {
                                                        "type": "general_log", 
                                                        "path": "/var/log/mysql/general.log"
                                                    },
                                                    {
                                                        "type": "log_slow_queries", 
                                                        "path": "/var/log/mysql/slow.log"
                                                    },
                                                    {
                                                        "type": "bin_log", 
                                                        "path": "/var/log/mysql/bin.log"
                                                    },
                                                    {
                                                        "type": "error_log", 
                                                        "path": "/var/log/mysql/error.log"
                                                    },
                             ],
                          }
       }
   }

How does it works

Trove will have datastore-spicif (mysql, cassandra, etc.) list that contain all possible log file entities that which can be (possible) exist at given instance defined by datastore. List looks like:

       "mapping": [
           'general_log', 'log_slow_queries',  'log_error', 'bin_log'
       ]
If assigned configuration groups has any ony them, user will see them in the response body. If assigned configuration group has none of log configuration attributes user will receive empty list.
* Some of datastore has already enabled logging process (mysql - error.log, cassandra - system.log, etc.)

Steps:

  1. Look over assigned configuration for log configuration mentioned in datastore-specific mapping
  2. Return every logs enabled by the assigned configuration

If none of logs are enabled by the configuration - user will obtain list that contains nothing.


Create and save database logging file entry. HTTP method POST

Route: /{tenant_id}/instance/{id}/logs

Request

   {
       "instance": {
         "log" : "general_log",
       }
   }


Response

   {
       "log": {
         "instance_id" : "12345678-1111-2222-3333-444444444444",
         "file" : "mysql.log",
         "locationURL" : "http://somewhere.com:PORT/dblogcontainer/{instance_id}/filename.timestamp",
         "created_at": POSIX Timestamp,
         "modified_at": POSIX Timestamp,
       }
   }

create_at and modified_at describes the time of log creation and the time last modification.

How does it works

When filename will be sent to a API service Trove will validate if the given log file is the part of the configuration group assigned to instance, if it is, API service will pick actual file system path to log file and will send request to guestagent through taskmanager. Steps:

  1. Check if log is a part of the configuration group assigned to instance;
  2. Pick the FS path to log
  3. Send appropriate request to taskmanager -> guestagent
  4. Wait until response will occure

Internal API

All required API will be covered

Guest Agent

Added new method and flow agent for saving the log files directly to Swift container

  • naming convention: dblogcontainer/{instance_id}/filename.timestamp
  • Storage container file extension: *.log, *.tar.gz
  • Storage Strategy: Swift
  • Container: dblogcontainer