https://wiki.openstack.org/w/index.php?title=AuditLogging&feed=atom&action=historyAuditLogging - Revision history2024-03-28T10:57:50ZRevision history for this page on the wikiMediaWiki 1.28.2https://wiki.openstack.org/w/index.php?title=AuditLogging&diff=17823&oldid=prevRyan Lane: Text replace - "NovaSpec" to "NovaSpec"2013-02-17T23:31:14Z<p>Text replace - "<a href="/w/index.php?title=NovaSpec&action=edit&redlink=1" class="new" title="NovaSpec (page does not exist)">NovaSpec</a>" to "NovaSpec"</p>
<table class="diff diff-contentalign-left" data-mw="interface">
<col class='diff-marker' />
<col class='diff-content' />
<col class='diff-marker' />
<col class='diff-content' />
<tr style='vertical-align: top;' lang='en'>
<td colspan='2' style="background-color: white; color:black; text-align: center;">← Older revision</td>
<td colspan='2' style="background-color: white; color:black; text-align: center;">Revision as of 23:31, 17 February 2013</td>
</tr><tr><td colspan="2" class="diff-lineno" id="mw-diff-left-l1" >Line 1:</td>
<td colspan="2" class="diff-lineno">Line 1:</td></tr>
<tr><td class='diff-marker'> </td><td style="background-color: #f9f9f9; color: #333333; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #e6e6e6; vertical-align: top; white-space: pre-wrap;"></td><td class='diff-marker'> </td><td style="background-color: #f9f9f9; color: #333333; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #e6e6e6; vertical-align: top; white-space: pre-wrap;"></td></tr>
<tr><td class='diff-marker'>−</td><td style="color:black; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #ffe49c; vertical-align: top; white-space: pre-wrap;"><div>* '''Launchpad Entry''': <del class="diffchange diffchange-inline">[[</del>NovaSpec<del class="diffchange diffchange-inline">]]</del>:audit-logging</div></td><td class='diff-marker'>+</td><td style="color:black; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div>* '''Launchpad Entry''': NovaSpec:audit-logging</div></td></tr>
<tr><td class='diff-marker'> </td><td style="background-color: #f9f9f9; color: #333333; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #e6e6e6; vertical-align: top; white-space: pre-wrap;"><div>* '''Created''': 2010-11-04</div></td><td class='diff-marker'> </td><td style="background-color: #f9f9f9; color: #333333; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #e6e6e6; vertical-align: top; white-space: pre-wrap;"><div>* '''Created''': 2010-11-04</div></td></tr>
<tr><td class='diff-marker'> </td><td style="background-color: #f9f9f9; color: #333333; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #e6e6e6; vertical-align: top; white-space: pre-wrap;"><div>* '''Contributors''': termie, toddwilley</div></td><td class='diff-marker'> </td><td style="background-color: #f9f9f9; color: #333333; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #e6e6e6; vertical-align: top; white-space: pre-wrap;"><div>* '''Contributors''': termie, toddwilley</div></td></tr>
</table>Ryan Lanehttps://wiki.openstack.org/w/index.php?title=AuditLogging&diff=16558&oldid=prevRyan Lane: Text replace - "__NOTOC__" to ""2013-02-17T23:25:46Z<p>Text replace - "__NOTOC__" to ""</p>
<table class="diff diff-contentalign-left" data-mw="interface">
<col class='diff-marker' />
<col class='diff-content' />
<col class='diff-marker' />
<col class='diff-content' />
<tr style='vertical-align: top;' lang='en'>
<td colspan='2' style="background-color: white; color:black; text-align: center;">← Older revision</td>
<td colspan='2' style="background-color: white; color:black; text-align: center;">Revision as of 23:25, 17 February 2013</td>
</tr><tr><td colspan="2" class="diff-lineno" id="mw-diff-left-l1" >Line 1:</td>
<td colspan="2" class="diff-lineno">Line 1:</td></tr>
<tr><td class='diff-marker'>−</td><td style="color:black; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #ffe49c; vertical-align: top; white-space: pre-wrap;"><div><del class="diffchange diffchange-inline">__NOTOC__</del></div></td><td class='diff-marker'>+</td><td style="color:black; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div> </div></td></tr>
<tr><td class='diff-marker'> </td><td style="background-color: #f9f9f9; color: #333333; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #e6e6e6; vertical-align: top; white-space: pre-wrap;"><div>* '''Launchpad Entry''': [[NovaSpec]]:audit-logging</div></td><td class='diff-marker'> </td><td style="background-color: #f9f9f9; color: #333333; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #e6e6e6; vertical-align: top; white-space: pre-wrap;"><div>* '''Launchpad Entry''': [[NovaSpec]]:audit-logging</div></td></tr>
<tr><td class='diff-marker'> </td><td style="background-color: #f9f9f9; color: #333333; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #e6e6e6; vertical-align: top; white-space: pre-wrap;"><div>* '''Created''': 2010-11-04</div></td><td class='diff-marker'> </td><td style="background-color: #f9f9f9; color: #333333; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #e6e6e6; vertical-align: top; white-space: pre-wrap;"><div>* '''Created''': 2010-11-04</div></td></tr>
</table>Ryan Lanehttps://wiki.openstack.org/w/index.php?title=AuditLogging&diff=8164&oldid=prevTermie at 08:52, 5 November 20102010-11-05T08:52:46Z<p></p>
<table class="diff diff-contentalign-left" data-mw="interface">
<col class='diff-marker' />
<col class='diff-content' />
<col class='diff-marker' />
<col class='diff-content' />
<tr style='vertical-align: top;' lang='en'>
<td colspan='2' style="background-color: white; color:black; text-align: center;">← Older revision</td>
<td colspan='2' style="background-color: white; color:black; text-align: center;">Revision as of 08:52, 5 November 2010</td>
</tr><tr><td colspan="2" class="diff-lineno" id="mw-diff-left-l2" >Line 2:</td>
<td colspan="2" class="diff-lineno">Line 2:</td></tr>
<tr><td class='diff-marker'> </td><td style="background-color: #f9f9f9; color: #333333; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #e6e6e6; vertical-align: top; white-space: pre-wrap;"><div>* '''Launchpad Entry''': [[NovaSpec]]:audit-logging</div></td><td class='diff-marker'> </td><td style="background-color: #f9f9f9; color: #333333; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #e6e6e6; vertical-align: top; white-space: pre-wrap;"><div>* '''Launchpad Entry''': [[NovaSpec]]:audit-logging</div></td></tr>
<tr><td class='diff-marker'> </td><td style="background-color: #f9f9f9; color: #333333; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #e6e6e6; vertical-align: top; white-space: pre-wrap;"><div>* '''Created''': 2010-11-04</div></td><td class='diff-marker'> </td><td style="background-color: #f9f9f9; color: #333333; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #e6e6e6; vertical-align: top; white-space: pre-wrap;"><div>* '''Created''': 2010-11-04</div></td></tr>
<tr><td class='diff-marker'>−</td><td style="color:black; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #ffe49c; vertical-align: top; white-space: pre-wrap;"><div>* '''Contributors''': termie, <del class="diffchange diffchange-inline">toddwiley</del></div></td><td class='diff-marker'>+</td><td style="color:black; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div>* '''Contributors''': termie, <ins class="diffchange diffchange-inline">toddwilley</ins></div></td></tr>
<tr><td class='diff-marker'> </td><td style="background-color: #f9f9f9; color: #333333; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #e6e6e6; vertical-align: top; white-space: pre-wrap;"></td><td class='diff-marker'> </td><td style="background-color: #f9f9f9; color: #333333; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #e6e6e6; vertical-align: top; white-space: pre-wrap;"></td></tr>
<tr><td class='diff-marker'> </td><td style="background-color: #f9f9f9; color: #333333; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #e6e6e6; vertical-align: top; white-space: pre-wrap;"><div>== Summary ==</div></td><td class='diff-marker'> </td><td style="background-color: #f9f9f9; color: #333333; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #e6e6e6; vertical-align: top; white-space: pre-wrap;"><div>== Summary ==</div></td></tr>
</table>Termiehttps://wiki.openstack.org/w/index.php?title=AuditLogging&diff=8163&oldid=prevTermie at 08:52, 5 November 20102010-11-05T08:52:14Z<p></p>
<p><b>New page</b></p><div>__NOTOC__<br />
* '''Launchpad Entry''': [[NovaSpec]]:audit-logging<br />
* '''Created''': 2010-11-04<br />
* '''Contributors''': termie, toddwiley<br />
<br />
== Summary ==<br />
<br />
The current state of logging within Nova is largely targetted at debugging and very much relies on a human reading and interpretting the output, lowering the effectiveness of any analysis.<br />
<br />
This proposal suggests a more formalized and verbose system of logging targetted at machine-readability with the following sub-goals:<br />
<br />
* performance profiling<br />
* error tracking<br />
* system usage statistics<br />
* security auditing<br />
<br />
Also suggested is the creation / usage of log analysis tools to provide useful information based on this new data, with the notable mention of the recently open-sourced Sawzall toolset as a good candidate.<br />
<br />
== Release Note ==<br />
<br />
As of this release logging has become much more verbose to provide information for statistical analysis of an active cluster. There are additionally a variety of new flags available to disable the new types of logging, and a couple helpers to assist in getting more human-readable output for debugging purposes.<br />
<br />
The new (self-explanatory) flags are:<br />
<br />
* --[no]rpc_performance_logging<br />
* --[no]db_performance_logging<br />
* --[no]exception_logging<br />
* --[no]request_logging<br />
<br />
The new helpers are:<br />
<br />
* bin/nova-log-format, to provide custom formatted log lines, defaults to summary output<br />
* bin/nova-log-expand, to expand any complex objects in log lines, usually used after you have grepped the log for the lines you care about.<br />
<br />
Additionally, the following log analysis scripts have been provided to crunch stats on your cluster, for usage information and output view the scripts themselves:<br />
<br />
* tools/user_demographics.sh<br />
* tools/load_graphs.sh<br />
* tools/exception_counts.sh<br />
* tools/performance_graphs.sh<br />
<br />
== Rationale ==<br />
<br />
To accurately guage performance, usage and have an audit trail both of actions taken by any given user more verbose logging needs to begin taking place.<br />
<br />
The current system relies more on intuition than hard data and for a production hosting environment intuition is not enough, we need to know our actual scaling characteristics and be able to trace security breaches.<br />
<br />
Logging is not the whole story but it is an integral part for being able to manage after-the-fact information and providing statistical data over time for sophisticated analysis.<br />
<br />
== User stories ==<br />
<br />
=== Johnny Slowqueries ===<br />
<br />
* Utilization<br />
* Load spike<br />
* Scheduling bullies<br />
<br />
...<br />
<br />
=== Megan Managerpants ===<br />
<br />
* 7-day actives<br />
* Largest users<br />
* Hourly activity graph<br />
<br />
...<br />
<br />
=== Sally Systems ===<br />
<br />
* Downtime Post-mortem<br />
* Unauthorized instance<br />
* Crashing instances<br />
<br />
...<br />
<br />
=== Danny Rollout ===<br />
<br />
* Exceptions per second before and after update<br />
* Rolling upgrade distribution lag<br />
<br />
...<br />
<br />
== Assumptions ==<br />
<br />
* Log rotation and/or centralized storage/access is handled outside of the scope of this document.<br />
* Logging overhead does not create a significant drop in performance.<br />
<br />
== Design ==<br />
<br />
At the highest level this is about adding complex data to logs, at a lower one it is a formalization of the process used by apps to log. Somewhere in between it is about wrapping methods with profiling code.<br />
<br />
=== Log Format ===<br />
<br />
For the sake of simplistic compatibility, JSON is a decent contender. Some anecdotal evidence ( http://blog.juma.me.uk/2009/03/25/json-serializationdeserialization-faster-than-protocol-buffers/ ) suggests JSON is on par with protobufs for serialization speed, and while I wouldn't trust those numbers too far it seems reasonable to say that we can probably dump to JSON roughly as quickly as any other format for complex data.<br />
<br />
Common logging keys would be:<br />
<br />
* Timestamp<br />
* Context (request context)<br />
** User who initiated action<br />
** Action user attempted to initiate<br />
* Current service<br />
* Current rpc (maybe this is part of context)<br />
* Freeform Message<br />
* sys.exc_info<br />
<br />
Taking the time to sort the keys alphabetically might make it easier for people to make quick hacky tools to get the data they want for some ad-hoc query idea.<br />
<br />
=== The Logging Process ===<br />
<br />
For most "human-generated" statements, those log lines that are just messages from developers to themselves that developers will make in day-to-day situations while working on the code the only real difference will be including 'context' in the statement, and even that could possibly be inferred by stack inspection.<br />
<br />
A custom wrapper for the default logging utility that is service-aware, outputs in the desired log format and a line of boilerplate on importing it should be enough for day-to-day use.<br />
<br />
=== Profiling ===<br />
<br />
The machine-generated logging would use the same system but would rarely need to be touched, it would wrap high-level entry points into the system to do profiling of:<br />
<br />
* overall call performance, how long from initiating an rpc until it has returned<br />
* db performance, for each db layer call how long it takes to return<br />
* uncaught exceptions at the highest level<br />
* requests handled<br />
* number of active X where X is a VM, a network config, or the like<br />
<br />
== Implementation ==<br />
<br />
=== Log Generation ===<br />
<br />
Implementation of outputting the logs is straightforward and is mostly a find and replace mission to make sure each module includes the boilerplate for the custom logging wrapper, and each call to a logging statement includes the additional context information.<br />
<br />
At this time it appears that the lp:~anso/nova/deploy branch has much of this part already handled.<br />
<br />
From there the logging output needs to be configured to use the desired logging format, and timing wrappers need to be placed at the appropriate entry points.<br />
<br />
=== Tools ===<br />
<br />
The two helpers are easy enough to write, the format string input for nova-log-sformat can be simple python formatting strings ( %(timestamp)s %(level)s %(message)s ) and the expander will basically be prettyprint.<br />
<br />
=== Analysis Tools ===<br />
<br />
This stuff is more tricky and probably will wait until the logging itself is implemented, but again the suggestion of Sawzall may be worth the learning curve.<br />
<br />
== Test/Demo Plan ==<br />
<br />
Testing and demoing will basically be running the tools against dumps of logs, possibly logs from testing though real logs would be preferrable, and documenting some example output.<br />
<br />
== Unresolved issues ==<br />
<br />
* Should we magically infer "context" from the stack?<br />
<br />
== BoF agenda and discussion ==<br />
<br />
Use this section to take notes during the BoF; if you keep it in the approved spec, use it for summarising what was discussed and note any options that were rejected.<br />
<br />
----<br />
[[Category:Spec]]</div>Termie