Jump to: navigation, search

Difference between revisions of "PersonalHomepage"

m (Text replace - "NovaSpec" to "NovaSpec")
 
(4 intermediate revisions by 2 users not shown)
Line 1: Line 1:
__NOTOC__
+
 
* '''Launchpad Entry''': [[NovaSpec]]:nova-notifications
+
* '''Launchpad Entry''': NovaSpec:nova-notifications
 
* '''Created''': 28 Sep 2011
 
* '''Created''': 28 Sep 2011
 
* '''Contributors''': Juan Lebrijo
 
* '''Contributors''': Juan Lebrijo
Line 22: Line 22:
 
== User stories ==
 
== User stories ==
  
# As a systems integrator, I need to retrieve usage data so that I can properly bill my customers.
+
# As a systems integrator, I need to retrieve usage data so that I can properly bill my customers.
# As a systems integrator or enterprise cloud administrator, I want to monitor usage data over time so that I can optimize the utilization of my resources.
+
# As a systems integrator or enterprise cloud administrator, I want to monitor usage data over time so that I can optimize the utilization of my resources.
# As a systems integrator, I need to provide usage data to multiple third-party systems without building custom interfaces to each one so that I can efficiently utilize my manpower.
+
# As a systems integrator, I need to provide usage data to multiple third-party systems without building custom interfaces to each one so that I can efficiently utilize my manpower.
  
 
== Design ==
 
== Design ==
Line 33: Line 33:
  
 
* Instances
 
* Instances
* * Memory increase/decrease
+
** Memory increase/decrease
* * Disk increase/decrease
+
** Disk increase/decrease
* * Image mounted
+
** Image mounted
* * BW consumed
+
** BW consumed
 
* Volumes
 
* Volumes
* * Created/deleted
+
** Created/deleted
* * Attached/detached
+
** Attached/detached
 
* Images:
 
* Images:
* * Created/Deleted
+
** Created/Deleted
 
* Floating IPs
 
* Floating IPs
* * Associated/dissasociated (to an instance)
+
** Associated/dissasociated (to an instance)
* * Allocated/released (by a user/tenant)
+
** Allocated/released (by a user/tenant)
 
* Snapshots
 
* Snapshots
 +
 +
On other projects (keystone, quantum, ...)
 +
 
* Fixed Networks
 
* Fixed Networks
 
* Tenants
 
* Tenants
Line 57: Line 60:
  
 
{| border="1" cellpadding="2" cellspacing="0"
 
{| border="1" cellpadding="2" cellspacing="0"
|  Type  
+
'''Type'''
|  Resource  
+
'''Resource'''
 +
|-
 +
|  compute.instance.create.start/.end **
 +
|  Instance
 +
|-
 +
|  compute.instance.delete.start/.end **
 +
|  Instance
 +
|-
 +
|  compute.instance.rebuild.start/.end **
 +
|  Instance
 +
|-
 +
|  compute.instance.resize.prep.start/.end **
 +
|  Instance
 +
|-
 +
|  compute.instance.resize.confirm.start/.end **
 +
|  Instance
 +
|-
 +
|  compute.instance.resize.revert.start/.end **
 +
|  Instance
 +
|-
 +
|  compute.instance.exists **
 +
|  Instance Audit
 +
|-
 +
|  compute.instance.reboot.start/.end ***
 +
|  Instance
 +
|-
 +
|  compute.volume.create.start/.end
 +
|  Volume
 +
|-
 +
|  compute.volume.delete.start/.end
 +
|  Volume
 +
|-
 +
|  compute.volume.attached.start/.end
 +
|  Volume
 +
|-
 +
|  compute.volume.detached.start/.end
 +
|  Volume
 +
|-
 +
|  compute.volume.exists
 +
|  Volume Audit
 +
|-
 +
|  compute.image.create.start/.end
 +
|  Image
 +
|-
 +
|  compute.image.delete.start/.end
 +
|  Image
 +
|-
 +
|  compute.image.exists
 +
|  Image Audit
 +
|-
 +
|  compute.ip.allocate.start/.end
 +
|  IP
 +
|-
 +
|  compute.ip.release.start/.end
 +
|  IP
 +
|-
 +
|  compute.instance.create_ip ***
 +
|  IP
 +
|-
 +
|  compute.instance.delete_ip ***
 +
|  IP
 +
|-
 +
|  compute.ip.exists
 +
|  IP Audit
 
|}
 
|}
| compute.instance.create.start/.end \*\* | Instance | Usage notification upon creation of a new instance |
 
| compute.instance.delete.start/.end *\* | Instance | Usage notification upon deletion of an instance |
 
| compute.instance.rebuild.start/.end *\* | Instance | Usage notification upon rebuild of instance |
 
| compute.instance.resize.prep.start/.end *\* | Instance | Usage notification when a resize/migrate of an instance is begun |
 
| compute.instance.resize.confirm.start/.end *\* | Instance | Usage notification when a resize/migration of an instance has been confirmed by the user |
 
| compute.instance.resize.revert.start/.end *\* | Instance | Usage notification when a resize/migration of an instance has been reverted by the user |
 
| compute.instance.exists *\* | Instance Audit | Periodic usage notification generated by the instance-usage-audit cron job. |
 
| {color:#ff0000}compute.instance.reboot.start/.end *{color} | Instance | Usage notification when a reboot of the instance has been executed |
 
| compute.volume.create.start/.end | Volume | Creation or a new volume notification |
 
| compute.volume.delete.start/.end | Volume | Deletion of a volume notification |
 
| compute.volume.attached.start/.end | Volume | Usage notification upon attaching of a volume to an instance |
 
| compute.volume.detached.start/.end | Volume | Usage notification upon a volume has been detached from an instance |
 
| compute.volume.exists | Volume Audit | Periodic usage notification generated by the volume-usage-audit cron job. |
 
| compute.image.create.start/.end | Image | Creation of a new image |
 
| compute.image.delete.start/.end | Image | Deletion of an image |
 
| compute.image.exists | Image Audit | Periodic usage notification generated by the image-usage-audit cron job. |
 
| compute.ip.allocate.start/.end | IP | Allocation of an IP address notification |
 
| compute.ip.release.start/.end | IP | Deletion of an IP address notification |
 
| compute.ip.associate.start/.end {color:#ff0000}(compute.instance.create_ip)\*{color} | IP | Association of an IP address to an instance |
 
| compute.ip.disassociate.start/.end {color:#ff0000}(compute.instance.delete_ip)\*{color} | IP | Disassociation of an IP address to an instance |
 
| compute.ip.exists | IP Audit | Periodic usage notification generated by the ip-usage-audit cron job. |
 
  
\*\* Events from the previous version: [http://wiki.openstack.org/SystemUsageData]
+
* * Events from the previous version: [http://wiki.openstack.org/SystemUsageData]
 +
* ** Seen in the reviewed code [nova/compute/manager.py|https://review.openstack.org/#patch,sidebyside,2460,10,nova/compute/manager.py]
  
{color:#ff0000}\* Seen in the reviewed code {color}[nova/compute/manager.py|https://review.openstack.org/#patch,sidebyside,2460,10,nova/compute/manager.py]
+
Example of a reboot event:
  
Example of a reboot event:
+
<pre><nowiki>#!highlight javascript
{code}
 
 
{
 
{
 
   "_context_roles": ["admin"],
 
   "_context_roles": ["admin"],
Line 120: Line 165:
 
   "_context_remote_address": null
 
   "_context_remote_address": null
 
}
 
}
{code}
+
</nowiki></pre>
 +
 
  
 
Create instance event:
 
Create instance event:
{code}
+
 
 +
<pre><nowiki>#!highlight javascript
 
{
 
{
 
   "_context_roles": ["admin"],
 
   "_context_roles": ["admin"],
Line 163: Line 210:
 
   "_context_remote_address": null
 
   "_context_remote_address": null
 
}
 
}
{code}
+
</nowiki></pre>
 +
 
  
 
=== Resources Model ===
 
=== Resources Model ===
Line 169: Line 217:
 
==== Instance ====
 
==== Instance ====
  
<!-- # tenant_id: Tenant ID that owns the this instance (string) -->
+
* tenant_id: Tenant ID that owns the this instance (string)
<!-- # user_id: User ID that owns this instance (string) -->
+
* user_id: User ID that owns this instance (string)
<!-- # instance_id: Nova instance ID of this instance (string) -->
+
* instance_id: Nova instance ID of this instance (string)
<!-- # instance_type: Name of the instance type ('flavor') of this instance. (string) -->
+
* instance_type: Name of the instance type ('flavor') of this instance. (string)
<!-- # instance_type_id: Nova ID for instance type ('flavor') of this instance. (string) -->
+
* instance_type_id: Nova ID for instance type ('flavor') of this instance. (string)
<!-- # display_name: User selected display name for instance. -->
+
* display_name: User selected display name for instance.
<!-- # created_at: Timestamp for when this instance's record was created in Nova (string, formatted "YYYY-MM-DD hh:mm:ss.ssssss") -->
+
* created_at: Timestamp for when this instance's record was created in Nova (string, formatted "YYYY-MM-DD hh:mm:ss.ssssss")
<!-- # launched_at: Timestamp for when this instance was last launched by hypervisor. (string, formatted "YYYY-MM-DD hh:mm:ss.ssssss") -->
+
* launched_at: Timestamp for when this instance was last launched by hypervisor. (string, formatted "YYYY-MM-DD hh:mm:ss.ssssss")
<!-- # image_ref_url: Image URL (from Glance) that this instance was created from. (string) -->
+
* image_ref_url: Image URL (from Glance) that this instance was created from. (string)
<!-- # state: Current state of instance. (string, such as 'active' or 'deleted') -->
+
* state: Current state of instance. (string, such as 'active' or 'deleted')
<!-- # state_description: Additional human readable description of current state of instance. -->
+
* state_description: Additional human readable description of current state of instance.
<!-- # fixed_ips: list of ip addresses (as strings) assigned to instance. -->
+
* fixed_ips: list of ip addresses (as strings) assigned to instance.
<!-- # memory_mb: memory allocation for this instance (in mb) -->
+
* memory_mb: memory allocation for this instance (in mb)
<!-- # disk_gb: disk allocation for this instance (in gb) -->
+
* disk_gb: disk allocation for this instance (in gb)
  
 
==== Instance Audit ====
 
==== Instance Audit ====
  
 
All instance attributes and:
 
All instance attributes and:
<!-- # audit_period_begining: Timestamp of beginning of audit period. (string, formatted "YYYY-MM-DD hh:mm:ss.ssssss") -->
+
* audit_period_begining: Timestamp of beginning of audit period. (string, formatted "YYYY-MM-DD hh:mm:ss.ssssss")
<!-- # audit_period_ending: Timestamp of end of audit period. (string, formatted "YYYY-MM-DD hh:mm:ss.ssssss") -->
+
* audit_period_ending: Timestamp of end of audit period. (string, formatted "YYYY-MM-DD hh:mm:ss.ssssss")
<!-- # bandwidth: Hash listing bandwidth used for each network this instance is connected to. Keys will be network labels, values will be hashes containing the keys 'bw_in', and 'bw_out', listing the incoming, and outgoing bandwith, respectively, used by that instance, on that network, for that audit period. Bandwith is reported in bytes. -->
+
* bandwidth: Hash listing bandwidth used for each network this instance is connected to. Keys will be network labels, values will be hashes containing the keys 'bw_in', and 'bw_out', listing the incoming, and outgoing bandwith, respectively, used by that instance, on that network, for that audit period. Bandwith is reported in bytes.
  
 
==== Image ====
 
==== Image ====
Line 198: Line 246:
  
 
----
 
----
[[NovaSpec]]
+
NovaSpec

Latest revision as of 23:31, 17 February 2013

  • Launchpad Entry: NovaSpec:nova-notifications
  • Created: 28 Sep 2011
  • Contributors: Juan Lebrijo

Summary

Nova administrators want data on system usage for billing, chargeback, or monitoring purposes.

Release Note

This paper is based on "System Usage Data" http://wiki.openstack.org/SystemUsageData implemented for Essex realease.

We are planning to implement this features for Folsom

Rationale

The best way to gather this data is from a notifications system based on AMQP queues (RabbitMQ messaging System).

Then the main challenge of this BluePrint is to define the most useful events to be transmitted from OpenStack subsystems to AMQP queues. In order to be consumed by billing, chargeback or monitoring systems.

User stories

  1. As a systems integrator, I need to retrieve usage data so that I can properly bill my customers.
  2. As a systems integrator or enterprise cloud administrator, I want to monitor usage data over time so that I can optimize the utilization of my resources.
  3. As a systems integrator, I need to provide usage data to multiple third-party systems without building custom interfaces to each one so that I can efficiently utilize my manpower.

Design

There are two events generated for every activity <activity>.start and <activity>.end. The "start" one specifies the beginning of the activity, and the "end" one specifies the success of the activity.

The Events resources are the objects in the platform that we are interested to scan:

  • Instances
    • Memory increase/decrease
    • Disk increase/decrease
    • Image mounted
    • BW consumed
  • Volumes
    • Created/deleted
    • Attached/detached
  • Images:
    • Created/Deleted
  • Floating IPs
    • Associated/dissasociated (to an instance)
    • Allocated/released (by a user/tenant)
  • Snapshots

On other projects (keystone, quantum, ...)

  • Fixed Networks
  • Tenants
  • Users
  • Inbound traffic
  • Outbound traffic

Implementation

Event Types

Type Resource
compute.instance.create.start/.end ** Instance
compute.instance.delete.start/.end ** Instance
compute.instance.rebuild.start/.end ** Instance
compute.instance.resize.prep.start/.end ** Instance
compute.instance.resize.confirm.start/.end ** Instance
compute.instance.resize.revert.start/.end ** Instance
compute.instance.exists ** Instance Audit
compute.instance.reboot.start/.end *** Instance
compute.volume.create.start/.end Volume
compute.volume.delete.start/.end Volume
compute.volume.attached.start/.end Volume
compute.volume.detached.start/.end Volume
compute.volume.exists Volume Audit
compute.image.create.start/.end Image
compute.image.delete.start/.end Image
compute.image.exists Image Audit
compute.ip.allocate.start/.end IP
compute.ip.release.start/.end IP
compute.instance.create_ip *** IP
compute.instance.delete_ip *** IP
compute.ip.exists IP Audit

Example of a reboot event:

#!highlight javascript
{
  "_context_roles": ["admin"],
  "_context_request_id": "req-b41fca68-5fe4-4c9a-b3ba-956b72bae971",
  "event_type": "compute.instance.reboot.end",
  "timestamp": "2012-02-29 00:07:23.442571",
  "_context_auth_token": null,
  "_context_user_id": null,
  "payload": {
    "state_description": "rebooting_hard",
    "display_name": "instance03",
    "disk_gb": 0,
    "tenant_id": "54deadcc2c724dadbb3f59b139c97c09",
    "created_at": "2012-02-28 23:56:10",
    "instance_id": "8c12bedb-1817-4543-9223-2484b138dc66",
    "instance_type_id": 2,
    "memory_mb": 512,
    "instance_type": "m1.tiny",
    "state": "active",
    "user_id": "admin",
    "launched_at": "2012-02-28 23:56:17",
    "image_ref_url": "http:\/\/192.168.10.179:9292\/images\/9c5bbe68-04eb-4687-ae90-f96b876c9d50"
  },
  "_context_read_deleted": "no",
  "priority": "INFO",
  "_context_strategy": "noauth",
  "_context_is_admin": true,
  "_context_project_id": null,
  "_context_timestamp": "2012-02-29T00:07:23.442607",
  "publisher_id": "compute.devstack",
  "message_id": "88d14623-fa5a-4e2e-aff6-3289b7ae72dd",
  "_context_remote_address": null
}


Create instance event:

#!highlight javascript
{
  "_context_roles": ["admin"],
  "_context_request_id": "req-67c9cc6a-c6a0-4bd6-901d-4cab5fac2704",
  "event_type": "compute.instance.create.end",
  "timestamp": "2012-02-29 00:16:51.844539",
  "_context_auth_token": null,
  "_context_user_id": null,
  "payload": {
    "state_description": "",
    "display_name": "instance07",
    "disk_gb": 0,
    "tenant_id": "464e0e5fac1747e1b81f482d80525ea6",
    "created_at": "2012-02-29 00:16:43",
    "instance_id": "576b3a16-b8ee-4105-b84c-623fbbda1f04",
    "instance_type_id": 2,
    "memory_mb": 512,
    "instance_type": "m1.tiny",
    "state": "active",
    "user_id": "admin",
    "fixed_ips": [{
      "meta": {},
      "type": "fixed",
      "version": 4,
      "address": "10.0.0.3",
      "floating_ips": []
    }],
    "launched_at": "2012-02-29 00:16:51.564723",
    "image_ref_url": "http:\/\/192.168.10.179:9292\/images\/9c5bbe68-04eb-4687-ae90-f96b876c9d50"
  },
  "_context_read_deleted": "no",
  "priority": "INFO",
  "_context_strategy": "noauth",
  "_context_is_admin": true,
  "_context_project_id": null,
  "_context_timestamp": "2012-02-29T00:16:51.844580",
  "publisher_id": "compute.devstack",
  "message_id": "be0d96ba-73ce-4a37-b6a9-923eadd8f86a",
  "_context_remote_address": null
}


Resources Model

Instance

  • tenant_id: Tenant ID that owns the this instance (string)
  • user_id: User ID that owns this instance (string)
  • instance_id: Nova instance ID of this instance (string)
  • instance_type: Name of the instance type ('flavor') of this instance. (string)
  • instance_type_id: Nova ID for instance type ('flavor') of this instance. (string)
  • display_name: User selected display name for instance.
  • created_at: Timestamp for when this instance's record was created in Nova (string, formatted "YYYY-MM-DD hh:mm:ss.ssssss")
  • launched_at: Timestamp for when this instance was last launched by hypervisor. (string, formatted "YYYY-MM-DD hh:mm:ss.ssssss")
  • image_ref_url: Image URL (from Glance) that this instance was created from. (string)
  • state: Current state of instance. (string, such as 'active' or 'deleted')
  • state_description: Additional human readable description of current state of instance.
  • fixed_ips: list of ip addresses (as strings) assigned to instance.
  • memory_mb: memory allocation for this instance (in mb)
  • disk_gb: disk allocation for this instance (in gb)

Instance Audit

All instance attributes and:

  • audit_period_begining: Timestamp of beginning of audit period. (string, formatted "YYYY-MM-DD hh:mm:ss.ssssss")
  • audit_period_ending: Timestamp of end of audit period. (string, formatted "YYYY-MM-DD hh:mm:ss.ssssss")
  • bandwidth: Hash listing bandwidth used for each network this instance is connected to. Keys will be network labels, values will be hashes containing the keys 'bw_in', and 'bw_out', listing the incoming, and outgoing bandwith, respectively, used by that instance, on that network, for that audit period. Bandwith is reported in bytes.

Image

Volume

IP


NovaSpec