Difference between revisions of "Trove/DBInstanceLogOperationV1"
(→Create and save database logging file entry. HTTP method POST) |
|||
(37 intermediate revisions by the same user not shown) | |||
Line 1: | Line 1: | ||
= Mission = | = Mission = | ||
− | Provide specific API interface to end-user which would allow to manipulate with database log files. This feature provides the ability to access log files via Swift so the can download them for Auditing/Troubleshooting purposes | + | :Provide specific API interface to end-user which would allow to manipulate with database log files. This feature provides the ability to access log files via Swift so the can download them for Auditing/Troubleshooting purposes. |
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
= Description = | = Description = | ||
Line 20: | Line 12: | ||
=== Perfomance tuning based on log file analyze. === | === Perfomance tuning based on log file analyze. === | ||
:Database throughput is always limited by the mathimatical equations based upon available resources, after reaching defined limit database starts throwing exceptions that are logged in general log(cassandra system.log, redis server log) or special error log (mysql, percona, maria, etc.). | :Database throughput is always limited by the mathimatical equations based upon available resources, after reaching defined limit database starts throwing exceptions that are logged in general log(cassandra system.log, redis server log) or special error log (mysql, percona, maria, etc.). | ||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
=== Database startup Issues resolving based upon error/general log analyz. === | === Database startup Issues resolving based upon error/general log analyz. === | ||
Line 37: | Line 19: | ||
== Benefits == | == Benefits == | ||
− | :From the user perspective, this feature completely covers real- | + | :From the user perspective, this feature completely covers real-world use cases mentioned in justification section (management, perfomance tuning, audit, etc.) |
= Impacts = | = Impacts = | ||
− | : | + | :This feature would not affect/break current Trove API. It changes the attitute to the actual Trove instance from the ''simple database server with connection URL'' to something bigger. It affects in-accessability of the instance, that is restricted by the terms of use of the public/private cloud and affects the PaaS term. |
== Database == | == Database == | ||
Line 51: | Line 33: | ||
=== Public API Details === | === Public API Details === | ||
− | : | + | :Two new resources, ''log-create'', ''log-show'' will be exposed as part of the Trove API. |
− | :The ''log-show'' is used to provide an ability to list all available(availability defined by | + | :The ''log-show'' is used to provide an ability to list all available(availability defined by applied configuration group) database logging filenames per instance. |
:The ''log-create'' is used to provide an ability to save database logging file into the Swift container, required attribute - instance. | :The ''log-create'' is used to provide an ability to save database logging file into the Swift container, required attribute - instance. | ||
Line 60: | Line 42: | ||
:To implement this capability, the create/modify/list instance operations will be extended in a manner that does not break the existing 1.0 contract. These operations will permit a user to create a new database logging file entry for already existed instance, list all available database logging filenames for all registered datstore versions (basically, for all datastore managers across all versions), show all available database logging filenames per certain datastore version (manager). | :To implement this capability, the create/modify/list instance operations will be extended in a manner that does not break the existing 1.0 contract. These operations will permit a user to create a new database logging file entry for already existed instance, list all available database logging filenames for all registered datstore versions (basically, for all datastore managers across all versions), show all available database logging filenames per certain datastore version (manager). | ||
− | == | + | == log-show == |
− | + | === Get the list of all available database logging files per given instance. HTTP method GET === | |
− | |||
'''''Route''''': | '''''Route''''': | ||
Line 72: | Line 53: | ||
{ | { | ||
"log_files": [ | "log_files": [ | ||
− | "general_log", "log_slow_queries", " | + | { |
+ | "type": "general_log", | ||
+ | "path": "/var/log/mysql/general.log" | ||
+ | }, | ||
+ | { | ||
+ | "type": "log_slow_queries", | ||
+ | "path": "/var/log/mysql/slow.log" | ||
+ | }, | ||
+ | { | ||
+ | "type": "bin_log", | ||
+ | "path": "/var/log/mysql/bin.log" | ||
+ | }, | ||
+ | { | ||
+ | "type": "error_log", | ||
+ | "path": "/var/log/mysql/error.log" | ||
+ | }, | ||
], | ], | ||
} | } | ||
Line 78: | Line 74: | ||
} | } | ||
− | ==== | + | == Configuration groups == |
+ | Logging can be turned on/ff by the configuration group through Configurations API and only. | ||
− | + | === HTTP code responses === | |
− | + | HTTP 200 Accepted. Logging was enabled by the configuration group. | |
− | + | HTTP 404 NotFound. Logging is not enabled by applied configuration group. | |
− | + | <br/> | |
− | |||
− | + | == log-create == | |
− | + | === Create and save database logging file entry. HTTP method POST=== | |
− | |||
− | |||
− | |||
− | |||
− | |||
'''''Route:''''' | '''''Route:''''' | ||
Line 102: | Line 93: | ||
{ | { | ||
"instance": { | "instance": { | ||
− | " | + | "log" : "general_log", |
} | } | ||
} | } | ||
Line 121: | Line 112: | ||
create_at and modified_at describes the time of log creation and the time last modification. | create_at and modified_at describes the time of log creation and the time last modification. | ||
− | === | + | === HTTP code responses === |
+ | HTTP 201 Created. Log was shipped to Storage | ||
+ | |||
+ | HTTP 403 Forbidden. Storage is not available. | ||
+ | |||
+ | HTTP 404 NotFound. Instance is missing. | ||
− | + | HTTP 400 BadRequest. Instance is not ACTIVE, can't execute procedure. | |
− | |||
− | |||
− | |||
− | |||
− | |||
== Internal API == | == Internal API == | ||
− | + | Message: log_create | |
+ | Type: call | ||
+ | Parameters: log_file_path | ||
== Guest Agent== | == Guest Agent== |
Latest revision as of 10:28, 30 July 2014
Contents
Mission
- Provide specific API interface to end-user which would allow to manipulate with database log files. This feature provides the ability to access log files via Swift so the can download them for Auditing/Troubleshooting purposes.
Description
- Log manipulations are designed to let user perform log investigations. Since Trove is PaaS - level project, it's user cannot interact with compute instance directly, only with database through given API (database operations).
Justification/Benefits
Justification
Perfomance tuning based on log file analyze.
- Database throughput is always limited by the mathimatical equations based upon available resources, after reaching defined limit database starts throwing exceptions that are logged in general log(cassandra system.log, redis server log) or special error log (mysql, percona, maria, etc.).
Database startup Issues resolving based upon error/general log analyz.
- Example: Mysql. The error log contains information indicating when mysqld was started and stopped and also any critical errors that occur while the server is running. If mysqld notices a table that needs to be automatically checked or repaired, it writes a message to the error log. Cassandra. Heap size errors based on automated memory allocation while database service launching (http://stackoverflow.com/questions/16243434/getting-outofmemory-in-cassandra-java-heap-space-version-1-0-7-frequently-on-dif).
- The minimum requirements set forth in the “general overview and risk assessment” section below must be completed for the audit to qualify for core audit coverage. Following completion of the general overview and risk assessment, the auditor will use professional judgment to select specific areas for additional focus and audit testing.
Benefits
- From the user perspective, this feature completely covers real-world use cases mentioned in justification section (management, perfomance tuning, audit, etc.)
Impacts
- This feature would not affect/break current Trove API. It changes the attitute to the actual Trove instance from the simple database server with connection URL to something bigger. It affects in-accessability of the instance, that is restricted by the terms of use of the public/private cloud and affects the PaaS term.
Database
- Database changes are not required because database logs are not tracked at the Trove backend as a resource that can be re-used in future.
Configuration
- None
Public API
New routes are presented. Description mentioned below.
Public API Details
- Two new resources, log-create, log-show will be exposed as part of the Trove API.
- The log-show is used to provide an ability to list all available(availability defined by applied configuration group) database logging filenames per instance.
- The log-create is used to provide an ability to save database logging file into the Swift container, required attribute - instance.
- To implement this capability, the create/modify/list instance operations will be extended in a manner that does not break the existing 1.0 contract. These operations will permit a user to create a new database logging file entry for already existed instance, list all available database logging filenames for all registered datstore versions (basically, for all datastore managers across all versions), show all available database logging filenames per certain datastore version (manager).
log-show
Get the list of all available database logging files per given instance. HTTP method GET
Route: /{tenant_id}/instances/{id}/logs
Response
{ "instance": { { "log_files": [ { "type": "general_log", "path": "/var/log/mysql/general.log" }, { "type": "log_slow_queries", "path": "/var/log/mysql/slow.log" }, { "type": "bin_log", "path": "/var/log/mysql/bin.log" }, { "type": "error_log", "path": "/var/log/mysql/error.log" }, ], } } }
Configuration groups
Logging can be turned on/ff by the configuration group through Configurations API and only.
HTTP code responses
HTTP 200 Accepted. Logging was enabled by the configuration group.
HTTP 404 NotFound. Logging is not enabled by applied configuration group.
log-create
Create and save database logging file entry. HTTP method POST
Route: /{tenant_id}/instance/{id}/logs
Request
{ "instance": { "log" : "general_log", } }
Response
{ "log": { "instance_id" : "12345678-1111-2222-3333-444444444444", "file" : "mysql.log", "locationURL" : "http://somewhere.com:PORT/dblogcontainer/{instance_id}/filename.timestamp", "created_at": POSIX Timestamp, "modified_at": POSIX Timestamp, } }
create_at and modified_at describes the time of log creation and the time last modification.
HTTP code responses
HTTP 201 Created. Log was shipped to Storage
HTTP 403 Forbidden. Storage is not available.
HTTP 404 NotFound. Instance is missing.
HTTP 400 BadRequest. Instance is not ACTIVE, can't execute procedure.
Internal API
Message: log_create Type: call Parameters: log_file_path
Guest Agent
Added new method and flow agent for saving the log files directly to Swift container
- naming convention: dblogcontainer/{instance_id}/filename.timestamp
- Storage container file extension: *.log, *.tar.gz
- Storage Strategy: Swift
- Container: dblogcontainer