Launchpad Entry: xenapi-sm-support
Created: 1 February 2011
Contributors: Renuka Apte, Armando Migliaccio
Summary
Design and implement a new driver for Nova-Volume, based on XenAPI Storage Manager. This will provide basic storage functionality (like volume creation, and destruction) on a number of different storage back-ends, but it will enable the capability of using more sophisticated storage back-ends for operations like cloning/snapshotting etc. To have an idea of the benefits of using XenAPI SM to provide back-end storage services, the list below shows some of the storage plugins already supported in XenServer/XCP:
- NFS VHD: SR plugin which stores disks as VHD files on a remote NFS filesystem
- Local VHD on LVM: SR plugin which represents disks as VHD disks on Logical Volumes within a locally-attached Volume Group
- HBA LUN-per-VDI driver: SR plugin which represents LUNs as VDIs sourced by hardware HBA adapters, e.g. hardware-based iSCSI or FC support
NetApp: SR driver for mapping of LUNs to VDIs on a NETAPP server, providing use of fast snapshot and clone features on the filer
- LVHD over FC: SR plugin which represents disks as VHDs on Logical Volumes within a Volume Group created on an HBA LUN, e.g. hardware-based iSCSI or FC support
- iSCSI: Base ISCSI SR driver, provides a LUN-per-VDI. Does not support creation of VDIs but accesses existing LUNs on a target.
- LVHD over iSCSI: SR plugin which represents disks as Logical Volumes within a Volume Group created on an iSCSI LUN
EqualLogic: SR driver for mapping of LUNs to VDIs on a EQUALLOGIC array group, providing use of fast snapshot and clone features on the array
Glossary
XenServer: Commercial, supported product from Citrix
Xen Cloud Platform (XCP): Open-source equivalent of XenServer (and the development project for the toolstack). Everything said about XenServer below applies equally to XCP
XenAPI: The management API exposed by XenServer and XCP
xapi: The primary daemon on XenServer and Xen Cloud Platform; the one that exposes the XenAPI
Release Note
No direct impact on user consuming the external HTTP API services. This blueprint enables the use of several storage back-ends by writing a single nova-volume driver, rather than several ones, as many as the specific storage back-ends required to be supported are needed. In due course, leveraging capabilities of specific back-ends, will enable operations like volume snapshotting, cloning, etc. that can be exposed via official API.
Rationale
This new driver will sit alongside drivers already developed like AoE, iSCSI and SAN (both running on commodity boxes running Linux). However, the latter is slightly different as it will still run on commodity hardware, but it access remote controllers on SAN hardware by some protocol like SSH, CLI or custom API. This is clearly undesirable because every time there is a need to support new hardware, a new driver must be designed and developed. The adoption of XenAPI SM will enable a number of storage back-end that are abstracted by a common API. Moreover, this becomes premium value to customers of service providers and may become differentiator for enterprise private clouds.
User stories
The system administrator of a medium sized company deploys a private cloud based on OpenStack. He had previously bought and deployed a NetApp filer to provide back-end storage to employees' virtual machines. He now uses XenAPISM volume driver to provide block storage to VMs created using the OpenStack private cloud.
A service provider wants to deliver premium storage to its customers. Storage back-ends like NetApp or EqualLogic can be easily supported using XenAPISM volume driver.
Assumptions
For the design and implementation of the new Nova-Volume driver, XenAPI for XenServer/XCP is going to be used.
Design
Definitions
- Backend: A term for a particular storage backend. This could be iSCSI, NFS, Netapp etc.
- Backend-config: All the parameters required to connect to a specific backend. For e.g. For NFS, this would be the server, path, etc.
- Flavor: A user friendly term to specify some notion of quality of service. For example, "gold" might mean that the volumes will use a backend where backups are possible.
The concept of flavor has been introduced in keeping with the LUNR work. No API changes have been introduced to allow for creating a volume of a particular flavor. This will be deferred until the LUNR code is available.
A flavor can be associated with multiple backends. Some kind of policy will decide which backend will be used to create a volume of a particular flavor. Until we have a way of selecting flavor, we will use a simple "first-fit" policy, where the first backend that can successfully create this volume is the one that is used.
The entity relationship diagram is given below:
Operation
Using the nova-manage command detailed in the implementation, an admin can add flavors and backends.
One or more nova-volume service instances will be deployed per availability zone. When an instance is started, it will create storage repositories (SRs) to connect to the backends available within that zone. All nova-volume instances within a zone can see all the available backends. These instances are completely symmetric and hence should be able to service any create_volume request within the zone.
Implementation
Commands
A category called “sm” has been added to nova-manage in the class StorageManagerCommands
The following actions will be added:
- flavor_list
- flavor_create
- flavor_delete
- backend_list
- backend_add
- backend_remove
Usage:
nova-manage sm flavor_create <label> <description>
nova-manage sm flavor_delete<label>
nova-manage sm backend_add <flavor label> <SR type> [config connection parameters]
Note: SR type and config connection parameters are in keeping with the Xen Command Line Interface. http://support.citrix.com/article/CTX124887
nova-manage sm backend_delete <backend-id>
Examples:
nova-manage sm flavor_create gold "Not all that glitters"
nova-manage sm flavor_delete gold
nova-manage sm backend_add gold nfs name_label=toybox-renuka server=myserver serverpath=/local/scratch/myname
nova-manage sm backend_remove 1
API Changes
As mentioned above, the API changes required to create/delete a volume of a particular flavor will be deferred till the LUNR code is available. It is therefore not possible to create a volume for a particular flavor yet, although the code/design exists.
The existing euca-create-volume and euca-delete-volume commands should be used.
New Tables
SM Flavor:
Field |
Type |
Null |
Key |
Default |
Extra |
created_at |
datetime |
YES |
|
NULL |
|
updated_at |
datetime |
YES |
|
NULL |
|
deleted_at |
datetime |
YES |
|
NULL |
|
deleted |
tinyint(1) |
YES |
|
NULL |
|
id |
int(11) |
NO |
PRI |
NULL |
auto_increment |
label |
varchar(255) |
YES |
|
NULL |
|
description |
varchar(255) |
YES |
|
NULL |
|
Backend-config:
Field |
Type |
Null |
Key |
Default |
Extra |
created_at |
datetime |
YES |
|
NULL |
|
updated_at |
datetime |
YES |
|
NULL |
|
deleted_at |
datetime |
YES |
|
NULL |
|
deleted |
tinyint(1) |
YES |
|
NULL |
|
id |
int(11) |
NO |
PRI |
NULL |
auto_increment |
flavor_id |
int(11) |
NO |
MUL |
NULL |
|
sr_uuid |
varchar(255) |
YES |
|
NULL |
|
sr_type |
varchar(255) |
YES |
|
NULL |
|
config_params |
varchar(2047) |
YES |
|
NULL |
|
SM Volume:
Field |
Type |
Null |
Key |
Default |
Extra |
created_at |
datetime |
YES |
|
NULL |
|
updated_at |
datetime |
YES |
|
NULL |
|
deleted_at |
datetime |
YES |
|
NULL |
|
deleted |
tinyint(1) |
YES |
|
NULL |
|
id |
int(11) |
NO |
PRI |
NULL |
|
backend_id |
int(11) |
NO |
MUL |
NULL |
|
vdi_uuid |
varchar(255) |
YES |
|
NULL |
|
Code Changes
A new volume driver called XenSMDriver has been added to nova/volume. It leverages the xenapi storage manager to control the available backends. Functions have been exposed to create/forget storage repositories and connect to them.
A do_setup function is added to volume driver in order to perform initialization (which includes creating backend repositories) when the service is started.
Test/Demo Plan
Unit tests will be provided as code is developed.
Unresolved issues
None at the time of writing
BoF agenda and discussion
Related topics:
- Having VM's root devices running on block storage
- Thin provisioning
- Fast cloning
- Advanced volume API (Volume as a Service)
Fibre channel and StorageLink
- Dedup