Difference between revisions of "SystemUsageData"
(converted account id to tenant id) |
|||
| Line 1: | Line 1: | ||
__NOTOC__ | __NOTOC__ | ||
| − | |||
| − | |||
| − | |||
| − | |||
<<[[TableOfContents]]()>> | <<[[TableOfContents]]()>> | ||
== Summary == | == Summary == | ||
| − | |||
Nova administrators want data on system usage for billing, chargeback, or monitoring purposes. | Nova administrators want data on system usage for billing, chargeback, or monitoring purposes. | ||
== Release Note == | == Release Note == | ||
| + | System usage data is emitted from Nova in the form of usage events from Nova's Notifications system. The notifications system can be configured to emit events either through nova's logging facility, or send them to a series of AMQP queues (one per notification priority). System usages are emitted as notification events with the INFO priority. Different types of usage events are distinguished via the notifications' event_type, which is a hierarchical dotted string such as ''compute.instance.create'', which allows usages to be easily grouped for aggregation. Usage notifications can be immediate, created when a specific increment of usage occurs (such as creation of an instance) or periodic, generated by a periodic task, like a cron job, and covering usage for a certain period of time. Besides the standard Nova Notification priority, notification timestamp, and event_type, usage notifications contain a payload of data that will vary depending on the event_type. This is presented as a json-formatted hash of key-value pairs. Some of the keys, such as tenant_id will always be present in any usage notification, others will be data relevent to that event_type (For example, instance related notifications will contain data describing the instance.) | ||
| − | + | An external application (https://github.com/Cerberus98/yagi) is available that can read notifications from AMQP queues and provide an Atom API. This application generates Atom feeds of the notifications, and can publish feeds via [[PubSubHubbub]] (PSH) for efficient distribution of data to subscribers. | |
== Rationale == | == Rationale == | ||
| + | As a systems integrator, we need the ability for 3rd party systems to query usage information from Nova to determine how to charge customers for use of the platform. | ||
| − | + | Enterprise customers also need the ability to determine usage data, though for different reasons. For example, a cloud administrator may want to determine VMs that have not been used in several month so that they can be reclaimed and the resources reallocated. Some IT departments may implement a quota system for their users. | |
| − | |||
| − | Enterprise customers also need the ability to determine usage data, though for different reasons. For example, a cloud administrator may want to determine VMs that have not been used in several month so that they can be reclaimed and the resources reallocated. | ||
| − | Note that Nova itself should not be concerned with billing, but it needs to | + | Note that Nova itself should not be concerned with billing, but it needs to present a feed of usage data that an external application could use to aggregate the billing data for a requested time period. |
== User stories == | == User stories == | ||
| − | |||
# As a systems integrator, I need to retrieve usage data so that I can properly bill my customers. | # As a systems integrator, I need to retrieve usage data so that I can properly bill my customers. | ||
# As a systems integrator or enterprise cloud administrator, I want to monitor usage data over time so that I can optimize the utilization of my resources. | # As a systems integrator or enterprise cloud administrator, I want to monitor usage data over time so that I can optimize the utilization of my resources. | ||
# As an business or agency that supports multiple projects, I need to account for the resources consumed by each project so that I can properly meet accounting and budget standards. | # As an business or agency that supports multiple projects, I need to account for the resources consumed by each project so that I can properly meet accounting and budget standards. | ||
| − | # As a systems integrator, I need to provide usage data to multiple third-party systems without building custom interfaces to each one so that I can efficiently utilize my manpower. | + | # As a systems integrator, I need to provide usage data to multiple third-party systems without building custom interfaces to each one so that I can efficiently utilize my manpower. |
== Assumptions == | == Assumptions == | ||
| + | Usage data can be aggregated by Tenant ID (as defined in the [[openstack-accounting]] blueprint).(i.e. Tenant ID is always present) | ||
| − | + | There is a billing system that is not a part of [[OpenStack]]. Invoicing, billing, and customer management are handled externally to [[OpenStack]]/Nova. Nova only has knowledge of an "tenant ID" (as per the [[openstack-accounting]] blueprint). | |
| − | + | The Tenant ID is the same as the tenant id concept in Keystone, and is known internally to Nova as the "Project ID". | |
| − | + | If Nova is installed in multiple regions (aka "zones"), usage data will not span zones; even if the same tenant ID is used in multiple zones, that data must be aggregated external to Nova. | |
| − | |||
| − | |||
| − | |||
| − | If Nova is installed in multiple regions (aka "zones"), usage data will not span zones; even if the same tenant ID is used in multiple zones, that data must be aggregated external to Nova. | ||
| − | |||
| − | |||
== Design and Implementation == | == Design and Implementation == | ||
| + | === Event Types and Payload data: === | ||
| + | ==== compute.instance.create: ==== | ||
| + | * Usage notification upon creation of a new instance. | ||
| + | * '''tenant_id''': Tenant ID that owns the this instance (string) | ||
| + | * '''user_id''': User ID that owns this instance (string) | ||
| + | * '''instance_id''': Nova instance ID of this instance (string) | ||
| + | * '''instance_type''': Name of the instance type ('flavor') of this instance. (string) | ||
| + | * '''instance_type_id''': Nova ID for instance type ('flavor') of this instance. (string) | ||
| + | * '''display_name''': User selected display name for instance. | ||
| + | * '''created_at''': Timestamp for when this instance's record was created in Nova (string, formatted "YYYY-MM-DD hh:mm:ss.ssssss") | ||
| + | * '''launched_at''': Timestamp for when this instance was last launched by hypervisor. (string, formatted "YYYY-MM-DD hh:mm:ss.ssssss") | ||
| + | * '''image_ref''': Image URL (from Glance) that this instance was created from. (string) | ||
| − | === | + | ==== compute.instance.delete: ==== |
| − | + | * Usage notification upon deletion of an instance. | |
| − | + | * '''tenant_id''': Tenant ID that owns the this instance (string) | |
| − | + | * '''user_id''': User ID that owns this instance (string) | |
| − | + | * '''instance_id''': Nova instance ID of this instance (string) | |
| − | + | * '''instance_type''': Name of the instance type ('flavor') of this instance. (string) | |
| − | + | * '''instance_type_id''': Nova ID for instance type ('flavor') of this instance. (string) | |
| − | + | * '''display_name''': User selected display name for instance. | |
| − | + | * '''created_at''': Timestamp for when this instance's record was created in Nova (string, formatted "YYYY-MM-DD hh:mm:ss.ssssss") | |
| − | + | * '''launched_at''': Timestamp for when this instance was last launched by hypervisor. (string, formatted "YYYY-MM-DD hh:mm:ss.ssssss") | |
| − | + | * '''image_ref''': Image URL (from Glance) that this instance was created from. (string) | |
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | '' | ||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | '' | ||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | == | + | ==== compute.instance.rebuild: ==== |
| + | * Usage notification upon rebuild of instance. | ||
| + | * '''tenant_id''': Tenant ID that owns the this instance (string) | ||
| + | * '''user_id''': User ID that owns this instance (string) | ||
| + | * '''instance_id''': Nova instance ID of this instance (string) | ||
| + | * '''instance_type''': Name of the instance type ('flavor') of this instance. (string) | ||
| + | * '''instance_type_id''': Nova ID for instance type ('flavor') of this instance. (string) | ||
| + | * '''display_name''': User selected display name for instance. | ||
| + | * '''created_at''': Timestamp for when this instance's record was created in Nova (string, formatted "YYYY-MM-DD hh:mm:ss.ssssss") | ||
| + | * '''launched_at''': Timestamp for when this instance was last launched by hypervisor. (string, formatted "YYYY-MM-DD hh:mm:ss.ssssss") | ||
| + | * '''image_ref''': Image URL (from Glance) that this instance <u>is being rebuilt</u> from. (string) | ||
| − | + | ==== compute.instance.resize.prep: ==== | |
| + | * Usage notification when a resize/migrate of an instance is begun. | ||
| + | * '''tenant_id''': Tenant ID that owns the this instance (string) | ||
| + | * '''user_id''': User ID that owns this instance (string) | ||
| + | * '''instance_id''': Nova instance ID of this instance (string) | ||
| + | * '''instance_type''': Name of the <u>current</u> (before resize) instance type ('flavor') of this instance. (string) | ||
| + | * '''instance_type_id''': Nova ID for <u>current</u> (before resize) instance type ('flavor') of this instance. (string) | ||
| + | * '''new_instance_type''': Name of the new (after resize) instance type ('flavor') of this instance. (string) | ||
| + | * '''new_instance_type_id''': Nova ID for new (after resize) instance type ('flavor') of this instance. (string) | ||
| + | * '''display_name''': User selected display name for instance. | ||
| + | * '''created_at''': Timestamp for when this instance's record was created in Nova (string, formatted "YYYY-MM-DD hh:mm:ss.ssssss") | ||
| + | * '''launched_at''': Timestamp for when this instance was last launched by hypervisor. (string, formatted "YYYY-MM-DD hh:mm:ss.ssssss") | ||
| + | * '''image_ref''': Image URL (from Glance) that this instance was created from. (string) | ||
| − | == | + | ==== compute.instance.resize.confirm: ==== |
| + | * Usage notification when a resize/migration of an instance has been confirmed by the user. | ||
| + | * '''tenant_id''': Tenant ID that owns the this instance (string) | ||
| + | * '''user_id''': User ID that owns this instance (string) | ||
| + | * '''instance_id''': Nova instance ID of this instance (string) | ||
| + | * '''instance_type''': Name of the current instance type ('flavor') of this instance. (i.e. the one it has been resized to.) (string) | ||
| + | * '''instance_type_id''': Nova ID for the current instance type ('flavor') of this instance. (i.e. the one it has been resized to.) (string) | ||
| + | * '''display_name''': User selected display name for instance. | ||
| + | * '''created_at''': Timestamp for when this instance's record was created in Nova (string, formatted "YYYY-MM-DD hh:mm:ss.ssssss") | ||
| + | * '''launched_at''': Timestamp for when this instance was last launched by hypervisor. (string, formatted "YYYY-MM-DD hh:mm:ss.ssssss") | ||
| + | * '''image_ref''': Image URL (from Glance) that this instance was created from. (string) | ||
| − | + | ==== compute.instance.resize.revert: ==== | |
| + | * Usage notification when a resize/migration of an instance has been reverted by the user. | ||
| + | * '''tenant_id''': Tenant ID that owns the this instance (string) | ||
| + | * '''user_id''': User ID that owns this instance (string) | ||
| + | * '''instance_id''': Nova instance ID of this instance (string) | ||
| + | * '''instance_type''': Name of the current instance type ('flavor') of this instance. (i.e. the one it has been reverted back to.) (string) | ||
| + | * '''instance_type_id''': Nova ID for the current instance type ('flavor') of this instance. (i.e. the one it has been reverted back to.) (string) | ||
| + | * '''display_name''': User selected display name for instance. | ||
| + | * '''created_at''': Timestamp for when this instance's record was created in Nova (string, formatted "YYYY-MM-DD hh:mm:ss.ssssss") | ||
| + | * '''launched_at''': Timestamp for when this instance was last launched by hypervisor. (string, formatted "YYYY-MM-DD hh:mm:ss.ssssss") | ||
| + | * '''image_ref''': Image URL (from Glance) that this instance was created from. (string) | ||
| − | ---- | + | ==== compute.instance.exists: ==== |
| − | + | * Periodic usage notification generated by the instance-usage-audit cron job. These usages are generated for '''each''' instance that has been neither '''created''' not '''deleted''' during the specified audit period. Aggregating these usages with the compute.instance.create and compute.instance.delete usages for the audit period will give a complete account of instance usage for each tenant during the audit period. | |
| + | * '''tenant_id''': Tenant ID that owns the this instance (string) | ||
| + | * '''user_id''': User ID that owns this instance (string) | ||
| + | * '''instance_id''': Nova instance ID of this instance (string) | ||
| + | * '''instance_type''': Name of the instance type ('flavor') of this instance. (string) | ||
| + | * '''instance_type_id''': Nova ID for instance type ('flavor') of this instance. (string) | ||
| + | * '''display_name''': User selected display name for instance. | ||
| + | * '''created_at''': Timestamp for when this instance's record was created in Nova (string, formatted "YYYY-MM-DD hh:mm:ss.ssssss") | ||
| + | * '''launched_at''': Timestamp for when this instance was last launched by hypervisor. (string, formatted "YYYY-MM-DD hh:mm:ss.ssssss") | ||
| + | * '''image_ref''': Image URL (from Glance) that this instance was created from. (string) | ||
| + | * '''audit_period_begining''': Timestamp of beginning of audit period. (string, formatted "YYYY-MM-DD hh:mm:ss.ssssss") | ||
| + | * '''audit_period_ending''': Timestamp of end of audit period. (string, formatted "YYYY-MM-DD hh:mm:ss.ssssss") | ||
Revision as of 10:54, 12 July 2011
<<TableOfContents()>>
Summary
Nova administrators want data on system usage for billing, chargeback, or monitoring purposes.
Release Note
System usage data is emitted from Nova in the form of usage events from Nova's Notifications system. The notifications system can be configured to emit events either through nova's logging facility, or send them to a series of AMQP queues (one per notification priority). System usages are emitted as notification events with the INFO priority. Different types of usage events are distinguished via the notifications' event_type, which is a hierarchical dotted string such as compute.instance.create, which allows usages to be easily grouped for aggregation. Usage notifications can be immediate, created when a specific increment of usage occurs (such as creation of an instance) or periodic, generated by a periodic task, like a cron job, and covering usage for a certain period of time. Besides the standard Nova Notification priority, notification timestamp, and event_type, usage notifications contain a payload of data that will vary depending on the event_type. This is presented as a json-formatted hash of key-value pairs. Some of the keys, such as tenant_id will always be present in any usage notification, others will be data relevent to that event_type (For example, instance related notifications will contain data describing the instance.)
An external application (https://github.com/Cerberus98/yagi) is available that can read notifications from AMQP queues and provide an Atom API. This application generates Atom feeds of the notifications, and can publish feeds via PubSubHubbub (PSH) for efficient distribution of data to subscribers.
Rationale
As a systems integrator, we need the ability for 3rd party systems to query usage information from Nova to determine how to charge customers for use of the platform.
Enterprise customers also need the ability to determine usage data, though for different reasons. For example, a cloud administrator may want to determine VMs that have not been used in several month so that they can be reclaimed and the resources reallocated. Some IT departments may implement a quota system for their users.
Note that Nova itself should not be concerned with billing, but it needs to present a feed of usage data that an external application could use to aggregate the billing data for a requested time period.
User stories
- As a systems integrator, I need to retrieve usage data so that I can properly bill my customers.
- As a systems integrator or enterprise cloud administrator, I want to monitor usage data over time so that I can optimize the utilization of my resources.
- As an business or agency that supports multiple projects, I need to account for the resources consumed by each project so that I can properly meet accounting and budget standards.
- As a systems integrator, I need to provide usage data to multiple third-party systems without building custom interfaces to each one so that I can efficiently utilize my manpower.
Assumptions
Usage data can be aggregated by Tenant ID (as defined in the openstack-accounting blueprint).(i.e. Tenant ID is always present)
There is a billing system that is not a part of OpenStack. Invoicing, billing, and customer management are handled externally to OpenStack/Nova. Nova only has knowledge of an "tenant ID" (as per the openstack-accounting blueprint).
The Tenant ID is the same as the tenant id concept in Keystone, and is known internally to Nova as the "Project ID".
If Nova is installed in multiple regions (aka "zones"), usage data will not span zones; even if the same tenant ID is used in multiple zones, that data must be aggregated external to Nova.
Design and Implementation
Event Types and Payload data:
compute.instance.create:
- Usage notification upon creation of a new instance.
- tenant_id: Tenant ID that owns the this instance (string)
- user_id: User ID that owns this instance (string)
- instance_id: Nova instance ID of this instance (string)
- instance_type: Name of the instance type ('flavor') of this instance. (string)
- instance_type_id: Nova ID for instance type ('flavor') of this instance. (string)
- display_name: User selected display name for instance.
- created_at: Timestamp for when this instance's record was created in Nova (string, formatted "YYYY-MM-DD hh:mm:ss.ssssss")
- launched_at: Timestamp for when this instance was last launched by hypervisor. (string, formatted "YYYY-MM-DD hh:mm:ss.ssssss")
- image_ref: Image URL (from Glance) that this instance was created from. (string)
compute.instance.delete:
- Usage notification upon deletion of an instance.
- tenant_id: Tenant ID that owns the this instance (string)
- user_id: User ID that owns this instance (string)
- instance_id: Nova instance ID of this instance (string)
- instance_type: Name of the instance type ('flavor') of this instance. (string)
- instance_type_id: Nova ID for instance type ('flavor') of this instance. (string)
- display_name: User selected display name for instance.
- created_at: Timestamp for when this instance's record was created in Nova (string, formatted "YYYY-MM-DD hh:mm:ss.ssssss")
- launched_at: Timestamp for when this instance was last launched by hypervisor. (string, formatted "YYYY-MM-DD hh:mm:ss.ssssss")
- image_ref: Image URL (from Glance) that this instance was created from. (string)
compute.instance.rebuild:
- Usage notification upon rebuild of instance.
- tenant_id: Tenant ID that owns the this instance (string)
- user_id: User ID that owns this instance (string)
- instance_id: Nova instance ID of this instance (string)
- instance_type: Name of the instance type ('flavor') of this instance. (string)
- instance_type_id: Nova ID for instance type ('flavor') of this instance. (string)
- display_name: User selected display name for instance.
- created_at: Timestamp for when this instance's record was created in Nova (string, formatted "YYYY-MM-DD hh:mm:ss.ssssss")
- launched_at: Timestamp for when this instance was last launched by hypervisor. (string, formatted "YYYY-MM-DD hh:mm:ss.ssssss")
- image_ref: Image URL (from Glance) that this instance is being rebuilt from. (string)
compute.instance.resize.prep:
- Usage notification when a resize/migrate of an instance is begun.
- tenant_id: Tenant ID that owns the this instance (string)
- user_id: User ID that owns this instance (string)
- instance_id: Nova instance ID of this instance (string)
- instance_type: Name of the current (before resize) instance type ('flavor') of this instance. (string)
- instance_type_id: Nova ID for current (before resize) instance type ('flavor') of this instance. (string)
- new_instance_type: Name of the new (after resize) instance type ('flavor') of this instance. (string)
- new_instance_type_id: Nova ID for new (after resize) instance type ('flavor') of this instance. (string)
- display_name: User selected display name for instance.
- created_at: Timestamp for when this instance's record was created in Nova (string, formatted "YYYY-MM-DD hh:mm:ss.ssssss")
- launched_at: Timestamp for when this instance was last launched by hypervisor. (string, formatted "YYYY-MM-DD hh:mm:ss.ssssss")
- image_ref: Image URL (from Glance) that this instance was created from. (string)
compute.instance.resize.confirm:
- Usage notification when a resize/migration of an instance has been confirmed by the user.
- tenant_id: Tenant ID that owns the this instance (string)
- user_id: User ID that owns this instance (string)
- instance_id: Nova instance ID of this instance (string)
- instance_type: Name of the current instance type ('flavor') of this instance. (i.e. the one it has been resized to.) (string)
- instance_type_id: Nova ID for the current instance type ('flavor') of this instance. (i.e. the one it has been resized to.) (string)
- display_name: User selected display name for instance.
- created_at: Timestamp for when this instance's record was created in Nova (string, formatted "YYYY-MM-DD hh:mm:ss.ssssss")
- launched_at: Timestamp for when this instance was last launched by hypervisor. (string, formatted "YYYY-MM-DD hh:mm:ss.ssssss")
- image_ref: Image URL (from Glance) that this instance was created from. (string)
compute.instance.resize.revert:
- Usage notification when a resize/migration of an instance has been reverted by the user.
- tenant_id: Tenant ID that owns the this instance (string)
- user_id: User ID that owns this instance (string)
- instance_id: Nova instance ID of this instance (string)
- instance_type: Name of the current instance type ('flavor') of this instance. (i.e. the one it has been reverted back to.) (string)
- instance_type_id: Nova ID for the current instance type ('flavor') of this instance. (i.e. the one it has been reverted back to.) (string)
- display_name: User selected display name for instance.
- created_at: Timestamp for when this instance's record was created in Nova (string, formatted "YYYY-MM-DD hh:mm:ss.ssssss")
- launched_at: Timestamp for when this instance was last launched by hypervisor. (string, formatted "YYYY-MM-DD hh:mm:ss.ssssss")
- image_ref: Image URL (from Glance) that this instance was created from. (string)
compute.instance.exists:
- Periodic usage notification generated by the instance-usage-audit cron job. These usages are generated for each instance that has been neither created not deleted during the specified audit period. Aggregating these usages with the compute.instance.create and compute.instance.delete usages for the audit period will give a complete account of instance usage for each tenant during the audit period.
- tenant_id: Tenant ID that owns the this instance (string)
- user_id: User ID that owns this instance (string)
- instance_id: Nova instance ID of this instance (string)
- instance_type: Name of the instance type ('flavor') of this instance. (string)
- instance_type_id: Nova ID for instance type ('flavor') of this instance. (string)
- display_name: User selected display name for instance.
- created_at: Timestamp for when this instance's record was created in Nova (string, formatted "YYYY-MM-DD hh:mm:ss.ssssss")
- launched_at: Timestamp for when this instance was last launched by hypervisor. (string, formatted "YYYY-MM-DD hh:mm:ss.ssssss")
- image_ref: Image URL (from Glance) that this instance was created from. (string)
- audit_period_begining: Timestamp of beginning of audit period. (string, formatted "YYYY-MM-DD hh:mm:ss.ssssss")
- audit_period_ending: Timestamp of end of audit period. (string, formatted "YYYY-MM-DD hh:mm:ss.ssssss")
