Revision as of 14:53, 30 April 2012

Efficient Metering in OpenStack Blueprint

Project and code : https://launchpad.net/ceilometer

Meetings : http://wiki.openstack.org/Meetings/MeteringAgenda

Uses cases

• need a tool to collect per customer usage
• need an API to query collected data from existing billing system
• data needed per customer, with an hour level granularity, includes:
  • Compute - Nova:
    • instances (type, availability zone) - hourly usage
    • cpu - hourly usage
    • ram - hourly usage
    • nova volume block device (type, availability zone) - hourly usage
      • reserved
      • used
  • network (data in/out, availability zone) - hourly bytes + total bytes
    • differentiate between internal and external end-points
    • External floating IP - hourly bytes + total bytes
• Storage - Swift
  • total data stored
  • data in/out - hourly bytes + total bytes
  • differentiate between internal and external end-points

^l

Proposed design

Counters

The following is a first list of counters that needs to be collected in order to allow billing systems to perform their tasks. This list must be expandable over time and each administrator must have the possibility to enable or disable each counter based on his local needs.

Other possible counters:

• service handlers (load balancer, databases, queues...)
• service usage

Note for the id column: The unique ID of a resource. If the id is missing, the counter is agregated. For instance if the ip of n4 is missing, the value is the total of Kbytes sent, regardless of the IP from which it originated, as long as the IP has been allocated to the tenant. There may be multiple counters with the same name : some with an ID and one with no id to aggregate all measures.

Note for network counters (n1-n4): the distinction between internal and external traffic requires that internal networks be explicitly listed in the agent configuration.

Storage

• Data is stored on a per account basis in a db on a per availability zone basis
• Per account records hold
  • account_id (same as keystone’s)
  • account_state (enabled, credit disabled, admin disabled)
• Per event records hold
  • account_id
  • counter_type
  • counter_volume
  • counter_duration
  • counter_datetime
  • message_signature
  • message_id
• db is not directly accessible by any other mean than API
• a process must collect messages from agent and store data
• a process may validate counters against nova event database
• a process may verify that messages were not lost
• a process may verify that accounts states are in sync with keystone

Alternative gauge design

During the Folsom ODS session, an alternate design was discussed where events instead of recoding deltas, would record the absolute value of a gauge. That would require to extend the event to include the 'object id' (instance, network, volume) associated with the counter.

The delta model can be derived from the absolute model, and means it's resilient in the face of missing delta registration.

Agents

• Agent on each nova compute node to accumulate and send counters for c1, c2, c3, c4, c5, n1, n2, n3, n4. The agent is likely to be pulling this information from libvirt.
  • c5 could get disk I/O stats with libvirt's virDomainBlockStats
  • n3 / n4 could use iptables accounting rules ? (for external traffic ?)
  • n1 / n2 could use libvirt's virDomainInterfaceStats ? (for all traffic ?)
• Agent on each nova volume node to accumulate and send counters for v1, v2
• Agent on each swift proxy to forward existing accounting data o1 and accumulate and send o2-o5

Note: nova network node need not accumulate and send counters for n5 because they can be pulled directly from the nova database ( see nova-manage floating list for instance )

Architecture

• An agent runs on each OpenStack node ( Bare Metal machine ) and harvests the data localy
• A storage daemon communicates with the agents to collect their data and aggregate them
• The data is sent from agents to the storage daemon via a trusted messaging system (RabbitMQ?)
• The message queue is separate from other queues (such as the nova queue)
• The messages in queue are signed and non repudiable (http://en.wikipedia.org/wiki/Non-repudiation)

Note: document some use case scenarios to really nail down the architecture. Who signals the metering service? The API service or nova, quantum, swift, glance, volume?

Messaging use cases

Instance creation

• An instance is created, nova issues a message ( http://wiki.openstack.org/SystemUsageData )
• The metering storage agent listens on the nova queue and picks up the creation message
• The metering storage agent stores the creation event locally, with a timestamp
• The metering storage daemon is notified by the agent that the instance has been created five minutes ago and aggregates this information in the tenant records

API

• Database can only be queried via a REST API (i.e. the database schema is not a supported API and can change in a non backward compatible way from one version to the other).
• Requests must be authenticated (separate from keystone, or only linked to accounting type account)
• API Server must be able to be redundant
• Requests allow to
  • GET account_id list
  • GET list of counter_type
  • GET list of events per account
    • optional start and end for counter_datetime
    • optional counter_type
• GET sum of (counter_volume, counter_duration) for counter_type and account_id
  • optional start and end for counter_datetime

Note: At the Folsom design session, the SET account_id call designed to change the status of the tenant in keystone was pointed more as a wart at this stage, since the billing system will need to talk to Keystone API anyway to make sense of the account id.

Free Software Billing Systems

A list of the billing system implementations that could use the Metering system when it becomes available.

• Dough https://github.com/lzyeval/dough
• trystack.org billing https://github.com/trystack/dash_billing
• nova-billing https://github.com/griddynamics/nova-billing

Related resources

• Definition of a Storage Accounting Record http://www.ogf.org/Public_Comment_Docs/Documents/2012-02/EMI-StAR-OGF-info-doc-v2.pdf
• UsageRecord format (UR) http:///documents/GFD.98.pdf
• Capturing exchanges https://github.com/rackspace/stacktach
• Messages about system usage http://wiki.openstack.org/SystemUsageData
• http://etherpad.openstack.org/EfficientMetering
• Use https://github.com/stackforge
• lzyeval codebase:
  • billing https://github.com/lzyeval/dough
  • metering https://github.com/lzyeval/kanyun
• trystack.org codebase:
  • https://github.com/trystack/dash_billing
• http://wiki.openstack.org/utilizationdata
• Nova billing https://github.com/griddynamics/nova-billing
• Swift
  • Retrieve Account Metadata http://docs.openstack.org/bexar/openstack-object-storage/developer/content/ch03s01.html#d5e388
  • swift middlewares examples :
    • https://github.com/spilgames/swprobe (https://lists.launchpad.net/openstack/msg07794.html)
    • https://github.com/pandemicsyn/swift-informant (https://lists.launchpad.net/openstack/msg07795.html)
• April 2012 mailing list thread on billing https://lists.launchpad.net/openstack/msg10334.html
• Virgo (scriptable agent for counter collection): https://github.com/racker/virgo
  • Contact Brandon Philips at Rackspace - brandon.philips@rackspace.com

FAQ

Q: why reinvent the wheel ? XXXX already does it.

A: please mail about the tool you think does the work, unless it is listed below.

• http://wiki.openstack.org/SystemUsageData for instance is specific to nova while the metering aims at aggregating all OpenStack components
• collectd, munin etc. all have pieces of the puzzle but do not have all of them and they are not designed with billing in mind and are not a good fit for this blueprint