SystemUsageDataEvol


 * Launchpad Entry: NovaSpec:nova-notifications
 * Created: 28 Sep 2011
 * Contributors: Juan Lebrijo, Craig Vyvial

Summary
Nova administrators want data on system usage for billing, chargeback, or monitoring purposes.

Release Note
This paper is based on "System Usage Data" http://wiki.openstack.org/SystemUsageData implemented for Essex realease.

We are planning to implement this features for Folsom

Rationale
The best way to gather this data is from a notifications system based on AMQP queues (RabbitMQ messaging System).

Then the main challenge of this BluePrint is to define the most useful events to be transmitted from OpenStack subsystems to AMQP queues. In order to be consumed by billing, chargeback or monitoring systems.

User stories

 * 1) As a systems integrator, I need to retrieve usage data so that I can properly bill my customers.
 * 2) As a systems integrator or enterprise cloud administrator, I want to monitor usage data over time so that I can optimize the utilization of my resources.
 * 3) As a systems integrator, I need to provide usage data to multiple third-party systems without building custom interfaces to each one so that I can efficiently utilize my manpower.

Design
There are two events generated for every activity .start and .end. The "start" one specifies the beginning of the activity, and the "end" one specifies the success of the activity.

The Events resources are the objects in the platform that we are interested to scan:


 * Instances
 * Memory increase/decrease
 * Disk increase/decrease
 * Image mounted
 * BW consumed
 * Volumes
 * Created/deleted
 * Attached/detached
 * Images:
 * Created/Deleted
 * Floating IPs
 * Associated/dissasociated (to an instance)
 * Allocated/released (by a user/tenant)
 * Snapshots

On other projects (keystone, quantum, ...)


 * Fixed Networks
 * Tenants
 * Users
 * Inbound traffic
 * Outbound traffic

Event Types

 * * Events from the previous version:
 * ** Seen in the reviewed code [nova/compute/manager.py|https://review.openstack.org/#patch,sidebyside,2460,10,nova/compute/manager.py]

Example of a reboot event:

#!highlight javascript { "_context_roles": ["admin"], "_context_request_id": "req-b41fca68-5fe4-4c9a-b3ba-956b72bae971", "event_type": "compute.instance.reboot.end", "timestamp": "2012-02-29 00:07:23.442571", "_context_auth_token": null, "_context_user_id": null, "payload": { "state_description": "rebooting_hard", "display_name": "instance03", "disk_gb": 0, "tenant_id": "54deadcc2c724dadbb3f59b139c97c09", "created_at": "2012-02-28 23:56:10", "instance_id": "8c12bedb-1817-4543-9223-2484b138dc66", "instance_type_id": 2, "memory_mb": 512, "instance_type": "m1.tiny", "state": "active", "user_id": "admin", "launched_at": "2012-02-28 23:56:17", "image_ref_url": "http:\/\/192.168.10.179:9292\/images\/9c5bbe68-04eb-4687-ae90-f96b876c9d50" }, "_context_read_deleted": "no", "priority": "INFO", "_context_strategy": "noauth", "_context_is_admin": true, "_context_project_id": null, "_context_timestamp": "2012-02-29T00:07:23.442607", "publisher_id": "compute.devstack", "message_id": "88d14623-fa5a-4e2e-aff6-3289b7ae72dd", "_context_remote_address": null }

Create instance event:

#!highlight javascript { "_context_roles": ["admin"], "_context_request_id": "req-67c9cc6a-c6a0-4bd6-901d-4cab5fac2704", "event_type": "compute.instance.create.end", "timestamp": "2012-02-29 00:16:51.844539", "_context_auth_token": null, "_context_user_id": null, "payload": { "state_description": "", "display_name": "instance07", "disk_gb": 0, "tenant_id": "464e0e5fac1747e1b81f482d80525ea6", "created_at": "2012-02-29 00:16:43", "instance_id": "576b3a16-b8ee-4105-b84c-623fbbda1f04", "instance_type_id": 2, "memory_mb": 512, "instance_type": "m1.tiny", "state": "active", "user_id": "admin", "fixed_ips": [{ "meta": {}, "type": "fixed", "version": 4, "address": "10.0.0.3", "floating_ips": [] }],   "launched_at": "2012-02-29 00:16:51.564723", "image_ref_url": "http:\/\/192.168.10.179:9292\/images\/9c5bbe68-04eb-4687-ae90-f96b876c9d50" }, "_context_read_deleted": "no", "priority": "INFO", "_context_strategy": "noauth", "_context_is_admin": true, "_context_project_id": null, "_context_timestamp": "2012-02-29T00:16:51.844580", "publisher_id": "compute.devstack", "message_id": "be0d96ba-73ce-4a37-b6a9-923eadd8f86a", "_context_remote_address": null }

Instance

 * tenant_id: Tenant ID that owns the this instance (string)
 * user_id: User ID that owns this instance (string)
 * instance_id: Nova instance ID of this instance (string)
 * instance_type: Name of the instance type ('flavor') of this instance. (string)
 * instance_type_id: Nova ID for instance type ('flavor') of this instance. (string)
 * display_name: User selected display name for instance.
 * created_at: Timestamp for when this instance's record was created in Nova (string, formatted "YYYY-MM-DD hh:mm:ss.ssssss")
 * launched_at: Timestamp for when this instance was last launched by hypervisor. (string, formatted "YYYY-MM-DD hh:mm:ss.ssssss")
 * image_ref_url: Image URL (from Glance) that this instance was created from. (string)
 * state: Current state of instance. (string, such as 'active' or 'deleted')
 * state_description: Additional human readable description of current state of instance.
 * fixed_ips: list of ip addresses (as strings) assigned to instance.
 * memory_mb: memory allocation for this instance (in mb)
 * disk_gb: disk allocation for this instance (in gb)

Instance Audit
All instance attributes and:
 * audit_period_begining: Timestamp of beginning of audit period. (string, formatted "YYYY-MM-DD hh:mm:ss.ssssss")
 * audit_period_ending: Timestamp of end of audit period. (string, formatted "YYYY-MM-DD hh:mm:ss.ssssss")
 * bandwidth: Hash listing bandwidth used for each network this instance is connected to. Keys will be network labels, values will be hashes containing the keys 'bw_in', and 'bw_out', listing the incoming, and outgoing bandwith, respectively, used by that instance, on that network, for that audit period. Bandwith is reported in bytes.

Volume

 * volume.create.start/.end (payload)
 * tenant_id: Tenant ID that owns the this volume (string)
 * user_id: User ID that owns this volume (string)
 * volume_id: Nova volume ID of the volume (string)
 * volume_type: Nova ID for the current volume type of this volume. (string)
 * display_name: Display name for the volume. (string)
 * created_at: Timestamp for when this volume's record was created in Nova (string, formatted "YYYY-MM-DD hh:mm:ss.ssssss")
 * launched_at: Timestamp for when this volume was last launched by hypervisor. (string, formatted "YYYY-MM-DD hh:mm:ss.ssssss")
 * status: Current status of volume. (string, such as 'attached')
 * size_gb: size of disk allocation for this volume (in gb)
 * volume.delete.start/.end
 * all the volume payload
 * volume.attach.start/.end
 * all the instance payload
 * all the volume payload
 * volume.detach.start/.end
 * all the instance payload
 * all the volume payload
 * volume.resize.start/.end
 * all the volume payload as well as
 * new_size_gb: size of disk allocation (after resize) for this volume (in gb)

Volume Audit

 * volume.exists
 * all the volume payload as well as
 * audit_period_begining: Timestamp of beginning of audit period. (string, formatted "YYYY-MM-DD hh:mm:ss.ssssss")
 * audit_period_ending: Timestamp of end of audit period. (string, formatted "YYYY-MM-DD hh:mm:ss.ssssss")