NetworkBandwidthEntitlement

• Launchpad Entry: NovaSpec:network-bandwidth-entitlement
• Created: 6th May 2013
• Contributors: Phil Day, HP Cloud Services

Summary

Currently the Nova host_manager keeps track of the Disk, Memory and #_vCPU resources of each host which are then available for use in the scheduler filters. These are also reflected in the flavour definitions.

We want to extend this model to add a measure of network bandwidth available to instances of this flavour. Each physical host would report its total network bandwidth capacity. This would allow support of hosts with mixed network configurations (10Gb, multi 10Gb, etc), and allow the definition and scheduling of "high bandwidth" instance types.

There are a number of aspects to this change:

• How to define these values as part of the instance type
• How to define the corresponding physical host capacities
• How to make this information available to the scheduler
• How to make the information available to the resource management layer on the physical host
• How to add this in a way that doesn't break current systems, and can be ignored by anyone that doesn't want/need this additional functionality

How to define these values as part of the instance type ?

Instance type already has a related rxtx_factor but current use in the system is limited to Xen. rxtx_factor (float): Is supported by the API extensions. Is passed into both Nova Networking and Quantum APIs. In Nova Network it's used to calculate an rxtx_cap for the instance by multiplying with the rxtx_base factor of the network. rxtx_cap is then used in the Xen driver to set a rate limit cap on a VIF.

Issues with this:

• rxtx_factor is used as a scaling factor on logical networks (i.e. attribute of a Network in the Nova DB), doesn't take into account host configuration.
• rxtx_factor means that instance types get different capacity depending on the networks it is connected to
• It is considered in the mutable resources tracked by the host_manager.

For Network bandwidth we really want to have an rxtx_cap as an alternative to rxtx_factor. We would then need to pass this to Quantum

We would need to also provide quota controls for these new attributes

How to define the corresponding physical host capacities:

Total host Network Bandwidth capacity is similarly something that it’s hard to see being derived automatically. The physical NICS of a host might be presented in a number of different ways to Nova.

Add a total_rxtx attribute to each host that describes the total bandwidth available to instances.

Values would be read from nova.conf and saved in the compute_nodes table

How to make this information available to the scheduler :

rxtx_total is property of each host, and consumed rxtx is the sum of the rxtx_caps of each instance type. (Implies that rxtx_cap is aggregated accross all NICs)

A new scheduler filter will be created to apply the Network Bandwidth capacity.

With more resource types being managed there may need to be a different weighting function (e.g. rank by % consumed of most consumed resource rather than just memory)

Visibility of rxtx_factor is already controlled by its own API extension, which is fine at the API layer. But as we want the scheduler to take these into account does this mean that the scheduler now needs to know which API extensions are enabled ?

How to make the information available to the resource management layer on the physical host:

This is really a hypervisor / quantum driver implementation issue. (The Xen driver for example already has it’s implementation and interpretation of vcpu_weight). In general each hypervisor drivers would need to convert to units appropriate to their resource mgmt system (e.g. vcpu_weight/cpu_base)

Current instance type attributes (#vcpus, memory) are easy to translate into VM definitions. Bandwidth probably less so as different cloud provided may have different views on implementation (shares vs cap)

Q) How much of this can be handled directly in the Hypervisor layer ? There are so many possible ways of doing this (shares vs quotas, which threads to limit, etc) that any build in model will need to have too many configuration options to meet everyone's needs.

Instead we prefer a simpler approach of making the flavour data available and letting an external script act on it. For example: Make sure all flavor data is in the notification messages Create an additional file in the instance directory that contains all of the flavour values

How to add this in a way that doesn't break current systems, and can be ignored by anyone that doesn't want/need this additional functionality

• Provide default values for host attributes (e.g. cpu_base=100, vcpu_weight=100)
• New scheduler filters can be left out of the scheduler by default
• Making flavour data available to external resource mgmt avoids trying to build a universal approach in the hypervisor driver
• Keeping compatibility with Xen will need work (assuming we re-use the vcpu_weight and add rxtx_cap attributes)