Ceilometer/blueprints/support-standard-audit-formats

Provide support for auditing events in standardized formats

From the onset, Ceilometer’s goal was:

”to become the infrastructure to collect measurements within OpenStack” with its “primary targets [being] are monitoring and metering”.  However it was noted that its framework “should be easily expandable to collect for other needs. To that effect, Ceilometer should be able to share collected data with a variety of consumers”.

It is clear that core project developers appreciate the strengths of Ceilometer in having a reliable, core centralized service with the ability to track usage information towards statistical usage analysis and billing. It seems that many of these same projects are seeing similar “auditing” requirements but now with the emphasis on tracking access to services (by users and other services) for the purposes of security auditing. Leveraging Ceilometer’s design to enable different types of event auditing standards (and to provide for different purposes and views on events) seems to make good sense.

Proposal Background

In the auditing and compliance space, standards are being developed for “Cloud Audit” that are designed to align with international and industry requirements yet address “clouds” unique deployment and service models. These standards recognize the challenge of auditing workloads in all types of cloud deployments while providing the data required for regulatory compliance (e.g. PCI-DSS, SoX, ISO 27017, etc.), as well as for proving adherence to localized customer security policies. One of these standards is the Distributed Mgmt. Task Force’s (DMTF) “Cloud Audit” standard which provides a standardized format (in JSON) and a RESTful query syntax.

DMTF Cloud Audit Standard Resources

The latest1.0b specification (released June 18, 2013): http://dmtf.org/sites/default/files/standards/documents/DSP0262_1.0.0b.pdf

This new version has added a few items of significance including "control" event type, explains how to express complex "targets" of actions, the ability to add "tags" for cross-domain references and most importantly a RESTful query API and syntax.

The specification is designed to address a wide range of auditing use cases as published here: http://www.dmtf.org/sites/default/files/standards/documents/DSP2028_1.0.0a.pdf

Proposal

This proposal seeks to leverage the design (agents/meters, etc.) and logging facilities already in Ceilometer and enable users to configure additional meters to allow event data to be produced in one or more (standard) formats. The configuration may also provide the location as to where the logs are created for each meter type (e.g. DMTF format at …/audit/dmtf/xxx).

We would like to add support for auditing APIs access using the DMTF format as a reference implementation as it provides normative data for the key “auditor questions” needed for any compliance regime known in compliance circles as the “7 Ws” and also provides guidance on classifying data by resource, correlation of events (across transactions and service layers) and guidance for federation/merging with other logs (e.g. use UUIDs, normalized timestamps). The format also supports optionally providing geolocation information (per ISO 27017 standards) and Metric/Measurement information that aligns with the developing NIST cloud model/standard.

Below is a sample of a representative DMTF format mappings to show it provides data similar to that in Ceilometer today, but also know that it provides for extended fields/structured data for many of the target regulatory standards mentioned standards above.

For auditing security and access control (e.g. for APIs calls) the CADF format may not contain any measurements (optional), but would include all the information needed for auditing (i.e. see the 7 “Ws” listed in the table below) such as information about the account (user). The reference/use of the DMTF standard format, which is designed for all types of cloud auditing, brings other forward- thinking benefits. For example, if OpenStack services scaled across regional boundaries, geolocation and other detailed resource information could optionally be tracked using the CADF standard. Note that this is just an example benefit the standard could offer for the future.

CADF Model is designed to answer all Audit and Compliance Questions

How CADF standard expresses the 7 “W”s of Audit and Compliance

Currently, there are two established methods for automatically sending data to Ceilometer (i.e. a cental agent or "pull" path and notification or "push" path) as shown below.

This blueprint proposes leveraging the existing Ceilometer design to establish a configurable audit filter/notification path to the Ceilometer collector that would use the CADF format (initially carried within the payload). The following diagram shows that each component that would require configurable API audit would derive a filter from a common audit filter based upon CADF, validated by an audit notifier which would use the existing common notification method and register a new “audit.api” event type which would be handled by a matching event filter for that type.

Link to PDF version of this blueprint: File:Support-standard-audit-formats-blueprint.pdf

Link to sample (WIP) code: https://review.openstack.org/#/c/30124/