- Launchpad Entry: nexenta-volume-driver
- Created: 11 Nov 2011
- Contributors: Yuriy Taraday
- 1 Summary
- 2 Release Note
- 3 Rationale
- 4 User stories
- 5 Assumptions
- 6 Design
- 7 Implementation
- 8 Test/Demo Plan
- 9 Unresolved issues
- 10 External links
We would like to introduce the NexentaStor support for the nova-volume service. This blueprint proposes the driver based on the iSCSI driver already integrated into the OpenStack nova-volume service, with the basic capabilities to create, export and delete volumes and snapshots.
The driver can interact with the NexentaStor Appliance using its REST API, or using DBus API. The default API is REST, and the DBus requires patched libdbus to function properly.
The proposal is to add NexentaStor driver to the nova-volume service.
NexentaStor Appliance is NAS/SAN software platform designed for building reliable and fast network storage arrays. The NexentaStor is based on the OpenSolaris and uses ZFS as a disk management system. NexentaStor can serve as a storage node for the OpenStack and provide block-level volumes for the virtual servers via iSCSI protocol.
- Openstack nova-volume service installed
- NexentaStor Appliance installed and configured
- Dedicated ‘volume’ object created on appliance
NexentaStor Appliance provides reliable block-level volumes which are easy to manage via simple REST interface. It automates many common storage administration tasks. It is intended to be used with commodity hardware and basically is vendor-independent. This makes NexentaStor Appliance software useful as a base for storage nodes in IaaS clouds.
User installs nova-volume service on the Openstack node and configures volume_driver, specifies communication transport and IP address of the NexentaStor appliance. Additional parameters for volumes can be specified in the flag file, i.e. block size. User can then create volumes with standard Nova API calls.
Nexenta Appliance Configuration
To get the volumes stored to the Nexenta appliance, it has to be configured appropriately. Configuration of appliance includes the following steps:
- install and configure NexentaStor software via management console and web interface using initial configuration wizard;
- use management console or web interface to create ZFS volume, for example, ‘nova-volumes’;
- specify base volume name in the nova.conf on the Openstack node that runs nova-volume.
For example, we have an appliance with 8 disks, two of which are used for system pool 'syspool'. This leaves us 6 disks to create storage pool:
nmc@nexentastor:/$ show lun disk LUN ID Device Type Size Volume Mounted Attach GUID c1t0d0 sd0 disk 147GB syspool no mpt 00825b934e690f9000059a03000b c1t1d0 sd2 disk 500GB no mpt 00c469d24e68c469000b4c830006 c1t2d0 sd3 disk 500GB no mpt 00c469d24e6a0070000e519b0011 c1t3d0 sd4 disk 500GB no mpt 00c469d24e6a053400070c500012 c1t4d0 sd5 disk 500GB no mpt 00c469d24e6f25ab000bb7770008 c1t5d0 sd6 disk 500GB no mpt 00c469d24e6f25ab000bcd6d0009 c1t6d0 sd7 disk 500GB no mpt 00c469d24e6f514500045e94000c c1t9d0 sd9 disk 147GB syspool no mpt 00c469d24e79b0ca0007a8000004
We want to create a 'nova-volumes' volume from this disks as 2 three-way mirrors to improve data protection. The pool can be created by the following command:
nmc@nexentastor:/$ create volume nova-volumes mirror c1t1d0 c1t2d0 c1t3d0 mirror c1t4d0 c1t5d0 c1t6d0 -y
This will non-interactively create volume 'nova-volumes'. We can see its properties with the command:
nmc@nexentastor:/$ show volume nova status volume: nova state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM nova ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 c1t1d0 ONLINE 0 0 0 c1t2d0 ONLINE 0 0 0 c1t3d0 ONLINE 0 0 0 mirror-1 ONLINE 0 0 0 c1t4d0 ONLINE 0 0 0 c1t5d0 ONLINE 0 0 0 c1t6d0 ONLINE 0 0 0 errors: No known data errors
OpenStack Nova configuration
You can set up the driver by setting following flags in nova.conf:
- volume_driver - should be set to nexenta.volume.NexentaDriver;
- nouse_local_volumes - this is necessary for all SAN drivers;
- nexenta_host - should be hostname or IP address of Nexenta appliance;
- nexenta_volume - should be set to the name of ZFS volume that should contain all Nova volumes;
- nexenta_user - name of user at Nexenta appliance with REST API access (e.g. admin);
- nexenta_password - password for that user.
Example config following:
--volume_driver=nexenta.volume.NexentaDriver --nouse_local_volumes --nexenta_host=172.16.1.10 --nexenta_volume=nova --nexenta_user=admin --nexenta_password=secrete
The NexentaStor appliance using SAS disk drives and connected to the Compute node by 1Gbps Ethernet provides following input and output rates:
- up to 60 Mbytes per second on sequential writes
- up to 132 Mbytes per second on random reads
- up to 1000 volumes per appliance tested
This driver assumes user has installed NexentaStor software on at least one appliance and configured it to have a logical volume dedicated to Openstack volumes.
Driver extends iSCSI driver in the same way as all SAN drivers. All operations are performed using one of two transport modules: either D-Bus or JSON RPC transport supported.
ZFS zvols are used as base for exported volumes. Zvol is a ZFS entity that essentialy is virtual block device. Its flexible configuration allows block sizes from 512 bytes to 128 Kbytes, and use of advanced ZFS specific features like built-in integrity control, deduplication and caching. Zvols can be sparse and consume only so much disk space as the size of actually written data.
The driver supports the following features of underlying ZFS zvols:
- block size can be specified in bytes or Kbytes with --nexenta_blocksize flag;
- use of sparse volumes can be configured with --nexenta_sparse flag;
Common iSCSI initiator-target scheme is used for export. Each volume is exported through dedicated target which is placed in separate target group to ensure correct iSCSI LUN numbering.
Driver functionality is separated into three modules. Two modules implement transports: nexenta.jsonrpc for JSON RPC and nexenta.NZA for D-Bus. Latter requires patched libdbus installed in the system. The nexenta.volume module contains the driver code.
Minor exception handling issue was fixed in the main nova code to enhance the behavior of the driver.
Nexenta driver code should be included as package under nova.volume package.
You can see the demonstration of the function of this driver in this video: http://www.youtube.com/watch?v=q76OONdWo8I
- Currently one cannot delete a snapshot if there is a volume created from it due to Nexenta limitations.
- http://www.nexenta.com/ -- official Nexenta Systems Inc. website. Documentation and downloads for NexentaStor software.
- http://www.nexentastor.org/ -- NexentaStor Community website. Open source projects, discussion boards and community downloads.
- ] -- driver's download page, you can find [driver itself and patch for libdbus here.