Difference between revisions of "WholeHostAllocation"

Revision as of 08:55, 24 July 2013

• Launchpad Entry: NovaSpec:whole-host-allocation
• Created: 6th May 2013
• Contributors: Phil Day, HP Cloud Services

Summary

Allow a tenant to allocate all of the capacity of a host for their exclusive use. The host remains part of the Nova configuration, i.e this is different from bare metal provisioning in that the tenant is not getting access to the Host OS - just a dedicated pool of compute capacity. This gives the tenant guaranteed isolation for their instances, at the premium of paying for a whole host.

Extending this further in the future could form the basis of hosted private clouds – i.e. schematics of having a private could without the operational overhead.

The proposal splits into three parts:

1. Allowing the user to define and manage a pool of servers
2. Adding / Removing servers to that pool, and generating the associated notification messages for billing
3. Allowing the user to schedule instances to a specific pool that they own or have rights to use

Defining and managing pools of servers

The basic mechanism for grouping servers into pools with associated metadata already exists in the form of Host aggregates, and there are already various parts of the code that rely on specific metadata values for things like scheduling schematics.    However host aggregates were never designed or intended to be directly use facing:

• There is no concept of an aggregate owner
• They have a simple ID (rather than a uuid)
• Metadata key values are effectively internal details of Nova
• There are no quotas associated with aggregates

So instead of changing aggregates to be user facing we will provide a wrapper layer that creates aggregrates in a controlled way with a limited set of pre-defined metadata. We call these "wrapped" aggregates Pclouds (Private Clouds)

A user can perform the following opertaions on a Pcloud:

Create a Pcloud: User specifies a descriptive name of the Pcloud. The system will validate the request against quotas and create a host aggregate with the following properties:

• The aggregate name is set to a generated uuid (This is what the user will use to identify the Pcloud in subsequent calls)
• The user supplied name is stored as a metadata value of the aggregate
• The project_id of the owner is stored as a metadata value of the aggregate

Add a project to a Pcloud: Pclouds are owned by a single project for quota, billing, and mgmt purposes. But the owner may want to allow other projects to use the resources within a Pcloud (a common use case in enterprises). So the owner of a Pcloud is allowed to add project_Ids as tenants of the Pcloud, which are then recorded in the aggregate metadata. Because metadata keys have to be unique, and metadata values are limited in length, project access rights are recorded as a metadata item per project using the project ID as the key. Note that owner of a Pcloud is automatically added as a tenant and creation time, but they can remove themselves if required.

Remove a project from a Pcloud: Simply deletes the metadata key for that project

Change the properties of a Pcloud:

(Future): Provides a controlled mechanism to update other aggregate metadata values that affect scheduling - for example providing a different value of the CPU or Memory over commit

List Pclouds: List all Pclouds that are available to a project. This includes Pclouds that the project has access rights to (so Projects can "discover" which Plcouds they can use without having to ask the Pcloud owner). Both the UUID and Name are shown in the list, along with the role (Owner or Tenant). If the project is the Owner of a Pcloud then they can also see the number of hosts in the Pcloud

Show Pcloud Details: Only available to the owner of the Pcloud, show details such as the list of projects allowed access, list of hosts, and instances running on those hosts. Hosts will be presented as a hash of hostname and project_id (as in the server details)

Delete a Pcloud: Remove a Pcloud. Can only be performed when there are no hosts associated with the Pcloud.

Adding / Removing Servers

There are a number of different ways that this could be specified and implemented, but to start with we propose to keep to a very simple definition and mechanism:

• The types of hosts available will be specified as "host flavors" (this is consistent with Ironic). Each host flavor will define the #CPUs, Memory, and Disk space of a server, possibly with other attributes in extra_specs. The flavor table will be extended to include an attribute which defines if a flavor is an instance or host flavour. The current flavor APIs will be constrained to show and work on just instance flavors.

• There will be a pool of prebuild but disabled servers in a specific aggregate (the Pcloud_free_pool aggregate) that are used exclusively for whole host allocation. (Don't want the complexity at this stage of interacting with the scheduler to find a suitable host with no VMs on it)

• A user will be able to request that a host of a particular host flavour and from a particular AZ is added to a Pcloud. The system will look for a suitable host from the pcloud_free_pool move it to the aggregate behind the Pcloud, and enable it.

• A user will be able to request that a specific host (identified by its hash) is removed from a Pcloud. Providing there are no instances on the host, the system will disable the service, and move the host back to the pcloud_free_pool

Schedule instances to a specific pool

Users should be able to specify whether they want to schedule to a specific Pcloud or the general resource pool (I.e they are not constrained to one or the other). Neither of the Current Scheduler isolation filters provide this. The user can specify teh uuid of a Pcloud to use via scheduler hints

The basic flow of the Pcloud filter is as follows:

   If a pcloud is specified in scheduler hints
        if the host is not part of the specified Pcloud:
             Fail
        else if the user does not has access to Pcloud:
            Fail
        else
            Pass
   else if the host is part of a Pcloud
       Fail
   else
       Pass

Use Cases

The required features are explored by stepping through the main use cases:

1. Tenant allocates a host: 1.1 Make the request to the API.

Q) What do they need to be able to specify ? Required size characteristics. Flavors would seem to be the natural thing to use here. Maybe we extend the flavor definition to include a flavor type (instance, host, bare metal). Note that Bare Metal also uses flavors in this way, adding a type would allow combined BM and non BM systems.

Q) Should this have to be an exact match (Mem,Disk,CPU), a minimum, or should we allow the cloud provider to tag the hosts in some way to match the flavors ?

Pool name: The tenant might want to have several pools of servers. Behind the scenes the system would create a host aggregate (project_id-pool_name). Having a default pool ensures at least one aggregate, and allow us to use aggregates as a way to manage the hosts allocated to a tenant. This saves us having to expose the aggregate API directly to tenants (Don’t want to do this as aggregates work on host names, don’t’ have an owner, etc)

Availability Zone: Tenant should be able to build pools of hosts that span AZs.

Min/max count: Tenant should be able to request multiple hosts in a single call

Don’t need image, Security Group, key name, metadata, nics, config_drive as these only apply to an instance.

Q) Should this be a new API call, or overloaded onto “server create” as with Bare Metal Provisioning ? Don’t really want to return an instance, and the parameter overlap with server create is pretty limited, so think it should be a new API

1.2) System validates request against the Quotas: Need a new set of quotas to cover hosts: #hosts, #cpus, memory(GB), disk(GB)

1.3) System allocates a free host, and adds it to the aggregate corresponding to the specified pool. Q) Do we need a scheduler for this, or can we do it in the API ? There is no need to interact with the compute manager, and no complex sequence of state changes (the host is already in the required state), so no real need for a separate scheduler.

Q) Do we allocate from the general pool of hosts, or does there need to be a way to create a reserved pool ? Can see a case for either, depending on how the cloud provider wants to manage their capacity. So provide configuration option to identify the host aggregate that hosts are allocated from. Add an aggregate filter to the main scheduler so that it can also be limited to allocating from specific aggregates. These aggregates used for instance creation and whole host allocation can then overlap or be disjoint.

Q) How does the system record that the host allocated ? Just transferring it into the tenant’s aggregate is enough – providing we make this an atomic operation when moving it from the "allocation aggregate". Otherwise we would need to do something like adding a project_id to the compute_hosts table.

Q) What do we return to the tenant ? Don’t really want to return host names, as these should be private to the cloud provider. Can't use the same hash of hostname and tenant that are used in server details, as we need to be able to extract the hostname for deallocation (see 3). So maybe we return id of the compute_nodes entry. Would be better if this was a uuid rather than an id.

1.4) System generates a notification event for the billing system Need a new class of even to represent hosts. Also need periodic “exists” event (billing systems like regular confirmation for self correction in case they miss an event)

Q) What else does the billing system need to be able to charge differently for instances on hosts allocated to tenants (the tenant is already paying for the host)? Add a list of aggregates the host belongs to in the instance notification events ?

2) Tenant creates an instance on one of their allocated hosts Since we’re using host aggregates to manage the hosts, this just requires a scheduler hint that can be used by a filter to identify the pool and translate into the aggregate name. Tenants can still also create instances on the general pool of hosts by not giving the scheduler hint.

3) Tenant deallocates a host System moves the host out of the project specific aggregate and back into the aggregate that it is using for whole host scheduling If there are instances running on the host then the API should refuse to de-allocate it.

Extended use cases (maybe future additional Blue Prints ?)

4) Tenant defines their own flavor(s) to be used within their pool(s) One of the use cases for having dedicated hosts is that the tenant wants to be able to use the full capacity of a host, or to create “off size” flavours that that Cloud Provider doesn’t want to have generally available (as they become hard to build pricing plans around).

There is a flavoraccess API extension, but looks like this just allows an admin to define access policies on flavours, not give control of specific flavours at the tenant level. Also not clear if the access polict is actually validated during instance creation

Maybe need a new TenantFlavor extension which creates the flavors as private to the calling tenant and adds the required extra_data to limit them to the aggregates owned by this tenant ?

5) Tenant can define scheduling policy for their hosts For example they may want to be able to specify a different rate of over provisioning, or maybe even a different set of filters.