NexentaVolumeDriver


 * Launchpad Entry: nexenta-volume-driver
 * Created: 11 Nov 2011
 * Contributors: Yuriy Taraday

Summary
We would like to introduce the NexentaStor support for the nova-volume service. This blueprint proposes the driver based on the iSCSI driver already integrated into the OpenStack nova-volume service, with the basic capabilities to create, export and delete volumes and snapshots.

The driver can interact with the NexentaStor Appliance using its REST API, or using DBus API. The default API is REST, and the DBus requires patched libdbus to function properly.

The proposal is to add NexentaStor driver to the nova-volume service.

Release Note
NexentaStor Appliance is NAS/SAN software platform designed for building reliable and fast network storage arrays. The NexentaStor is based on the OpenSolaris and uses ZFS as a disk management system. NexentaStor can serve as a storage node for the OpenStack and provide block-level volumes for the virtual servers via iSCSI protocol.

Requirements

 * Openstack nova-volume service installed
 * NexentaStor Appliance installed and configured
 * Dedicated ‘volume’ object created on appliance

Rationale
NexentaStor Appliance provides reliable block-level volumes which are easy to manage via simple REST interface. It automates many common storage administration tasks. It is intended to be used with commodity hardware and basically is vendor-independent. This makes NexentaStor Appliance software useful as a base for storage nodes in IaaS clouds.

User stories
User installs nova-volume service on the Openstack node and configures volume_driver, specifies communication transport and IP address of the NexentaStor appliance. Additional parameters for volumes can be specified in the flag file, i.e. block size. User can then create volumes with standard Nova API calls.

Nexenta Appliance Configuration
To get the volumes stored to the Nexenta appliance, it has to be configured appropriately. Configuration of appliance includes the following steps:
 * install and configure NexentaStor software via management console and web interface using initial configuration wizard;
 * use management console or web interface to create ZFS volume, for example, ‘nova-volumes’;
 * specify base volume name in the nova.conf on the Openstack node that runs nova-volume.

Example Configuration
For example, we have an appliance with 8 disks, two of which are used for system pool 'syspool'. This leaves us 6 disks to create storage pool:

nmc@nexentastor:/$ show lun disk LUN ID     Device    Type         Size         Volume     Mounted Attach GUID c1t0d0     sd0       disk         147GB        syspool    no      mpt    00825b934e690f9000059a03000b c1t1d0     sd2       disk         500GB                   no      mpt    00c469d24e68c469000b4c830006 c1t2d0     sd3       disk         500GB                   no      mpt    00c469d24e6a0070000e519b0011 c1t3d0     sd4       disk         500GB                   no      mpt    00c469d24e6a053400070c500012 c1t4d0     sd5       disk         500GB                   no      mpt    00c469d24e6f25ab000bb7770008 c1t5d0     sd6       disk         500GB                   no      mpt    00c469d24e6f25ab000bcd6d0009 c1t6d0     sd7       disk         500GB                   no      mpt    00c469d24e6f514500045e94000c c1t9d0     sd9       disk         147GB        syspool    no      mpt    00c469d24e79b0ca0007a8000004

We want to create a 'nova-volumes' volume from this disks as 2 three-way mirrors to improve data protection. The pool can be created by the following command:

nmc@nexentastor:/$ create volume nova-volumes mirror c1t1d0 c1t2d0 c1t3d0 mirror c1t4d0 c1t5d0 c1t6d0 -y

This will non-interactively create volume 'nova-volumes'. We can see its properties with the command:

nmc@nexentastor:/$ show volume nova status volume: nova state: ONLINE scan: none requested config:

NAME       STATE     READ WRITE CKSUM nova       ONLINE       0     0     0 mirror-0 ONLINE       0     0     0 c1t1d0 ONLINE       0     0     0 c1t2d0 ONLINE       0     0     0 c1t3d0 ONLINE       0     0     0 mirror-1 ONLINE       0     0     0 c1t4d0 ONLINE       0     0     0 c1t5d0 ONLINE       0     0     0 c1t6d0 ONLINE       0     0     0

errors: No known data errors

OpenStack Nova configuration
You can set up the driver by setting following flags in nova.conf:
 * volume_driver - should be set to nexenta.volume.NexentaDriver;
 * nouse_local_volumes - this is necessary for all SAN drivers;
 * nexenta_host - should be hostname or IP address of Nexenta appliance;
 * nexenta_volume - should be set to the name of ZFS volume that should contain all Nova volumes;
 * nexenta_user - name of user at Nexenta appliance with REST API access (e.g. admin);
 * nexenta_password - password for that user.

Example config following:

--volume_driver=nexenta.volume.NexentaDriver --nouse_local_volumes --nexenta_host=172.16.1.10 --nexenta_volume=nova --nexenta_user=admin --nexenta_password=secrete

User Experience
The NexentaStor appliance using SAS disk drives and connected to the Compute node by 1Gbps Ethernet provides following input and output rates:
 * up to 60 Mbytes per second on sequential writes
 * up to 132 Mbytes per second on random reads
 * up to 1000 volumes per appliance tested

Assumptions
This driver assumes user has installed NexentaStor software on at least one appliance and configured it to have a logical volume dedicated to Openstack volumes.

Design
Driver extends iSCSI driver in the same way as all SAN drivers. All operations are performed using one of two transport modules: either D-Bus or JSON RPC transport supported.

Volumes Back-End
ZFS zvols are used as base for exported volumes. Zvol is a ZFS entity that essentialy is virtual block device. Its flexible configuration allows block sizes from 512 bytes to 128 Kbytes, and use of advanced ZFS specific features like built-in integrity control, deduplication and caching. Zvols can be sparse and consume only so much disk space as the size of actually written data.

The driver supports the following features of underlying ZFS zvols:
 * block size can be specified in bytes or Kbytes with --nexenta_blocksize flag;
 * use of sparse volumes can be configured with --nexenta_sparse flag;

Volumes Export
Common iSCSI initiator-target scheme is used for export. Each volume is exported through dedicated target which is placed in separate target group to ensure correct iSCSI LUN numbering.

Implementation
Driver functionality is separated into three modules. Two modules implement transports: nexenta.jsonrpc for JSON RPC and nexenta.NZA for D-Bus. Latter requires patched libdbus installed in the system. The nexenta.volume module contains the driver code.

Minor exception handling issue was fixed in the main nova code to enhance the behavior of the driver.

Code Changes
Nexenta driver code should be included as package under nova.volume package.

Test/Demo Plan
You can see the demonstration of the function of this driver in this video: http://www.youtube.com/watch?v=q76OONdWo8I

Unresolved issues

 * Currently one cannot delete a snapshot if there is a volume created from it due to Nexenta limitations.