Jump to: navigation, search



VMware VMDK cinder driver

The goal of this blueprint is to implement VMDK driver for cinder. The driver will allow management of cinder volume on any VMware vCenter Server or ESX managed datastore. In this project, we are essentially mapping the Cinder Volume construct to VMDK file(s) that form the persistent block storage for virtual machines within the VMware stack. Today, there is no cinder driver implementation for the VMware stack and the nova driver allows only attaching/detaching discovered iSCSI targets as RDM. This driver will allow life cycle management for a cinder volume that is backed by a VMDK file(s) within a VMware datastore. This project also positions Cinder to take advantage of features provided by VMFS and upcoming technologies such as vSAN, vVol and others.

Because of the design of vCenter, each VMDK needs to be a "child" object of one or more VM's. In this implementation, we use a "shadow" VM to back the Cinder Volume. This ensures that VMware specific features such as snapshots, fast cloning, vMotion, etc. will continue to work without breaking any of the Cinder Volume constructs or abstractions. This virtual machine backing a volume will never be powered on and is only an abstraction for performing operations such as snapshots or cloning of the cinder volume. By using virtual machine as a representation for cinder volume we can perform any operation on a cinder volume that can be done on the corresponding virtual machine using the public SDK.

Work Items

Driver configuration

  • The following are mandatory properties, specific to the driver and are to be specified in the /etc/cinder/cinder.conf file.
volume_driver=cinder.volume.drivers.vmware.vmdk.VMwareVcVmdkDriver (for vCenter server)
volume_driver=cinder.volume.drivers.vmware.vmdk.VMwareEsxVmdkDriver (for ESX/ESXi server)
  • The following are optional properties, specific to the driver and are to be specified in the /etc/cinder/cinder.conf file.
  • Assumption is cinder and nova are configured to the same server.
  • There is restriction to the nova driver(s) while using VMDK driver. Use the appropriate nova driver as shown while using VMDK driver.
vCenter server:

ESX/ESXi server:

Driver details

  • Name: VMwareVcVmdkDriver and VMwareEsxVmdkDriver
  • Root: cinder/volume/drivers/vmware/*
class VMwareEsxVmdkDriver(driver.VolumeDriver)
class VMwareVcVmdkDriver(VMwareEsxVmdkDriver)


  • Collect stats of datastores being managed by the VMware server. Stats will be used by schedulers such as filter_scheduler for filtering (e.g. CapacityFilter) and weighing (e.g. CapacityWeigher).
  • VMware server uses aggregates of datastores. Using an aggregated capacity/free space or that of a randomly chosen datastore may not be the right metric.
  • 'unknown' will be used for total and free capacity.


  • This is creation of volume from scratch.
    • Backing is not created for the empty volume in the inventory. A backing will be created when the empty volume is being attached to an instance for the first time.
  • VMDK type can be specified for the backing. This type will be used at the time of creating backing for the volume.
    • Thin provisioning - extra spec vmware:vmdk_type=thin.
    • Thick provisioning - extra spec vmware:vmdk_type=thick.
    • Eager zeroed thick provisioning - extra spec vmware:vmdk_type=eagerZeroedThick.
    • Default is thin provisioning.


  • This operation deletes the volume from inventory if present.


  • This is a contract between nova and driver.
  • This method is called before initializing/terminating connections to the volume service.
  • Must return moref of the virtual machine if it is already created.
  • See initialize_connection how the connector info is used.


  • This method is called when nova decides to initialize connection to the volume service.
  • The driver decides where to create or relocate volume. This is a contract between nova and driver.

Case - 1

If the virtual machine instance to which the volume is being attached is already present.

  • If the volume does not have a backing in the inventory.
    • Create volume backing with the specified vmdk_type (see create_volume) on appropriate datastore.
  • If the volume has a backing and its datastore is not visible to the ESX managing the virtual machine instance.
    • Relocate the volume onto one of datastores that is visible to the ESX managing the virtual machine instance.
  • At this point the virtual machine instance and volume are under a common ESX.
  • Returns the volume's moref so that nova can attach it as a new disk device to the virtual machine instance.

Case - 2

If the virtual machine instance does not exist. This is the case of booting a virtual machine instance from a volume.

  • If the volume does not have a backing. (Ideally we do not expect user to use a fresh volume as a bootable device)
    • We create backing for volume in the inventory on appropriate datastore.
  • At this point we have volume with backing in the inventory.
  • We return the volume's moref.

Datastore selection

  • Datastore with highest (free_space/total_space) and that can accommodate the volume is chosen for creation.


  • Reconfigure the existing virtual machine instance by creating a new disk device backed by the volume's VMDK.
  • At this point, the virtual machine instance's ESX can view the volume's datastore.
  • Save the volume's VMDK uuid as part of instance's extraConfig.
  • See initialize_connection how the input connection_info is formed.

Mount point

  • The volume is not mounted at the specified mount point. The guest OS user must manually mount the discovered disk device.


  • Reconfigure the virtual machine instance by removing volume's disk device.
    • See attach_volume how we set extraConfig entry to identify the volume device.
  • The virtual machine instance may have been moved to another datastore by SDRS by copying the volume's VMDK chain along. Consolidation of disk chains must be done before detach.


  • This method is called when nova decides to terminate connection with volume service.
  • At this point, the volume's VMDK is either moved to a new datastore or not moved by SDRS.
  • If the volume's VMDK has been moved to a new datastore. (See get_volume_connector how connector data is collected.)
    • We relocate the existing volume to the new datastore with VirtualMachineRelocateDiskMoveOptions=moveAllDiskBackingsAndAllowSharing so that we share the read only disks that are part of the volume's VMDK chain (the chain could be due to snapshotting the volume, see create_snapshot)
    • At this point we have two writable VMDK deltas with a common parent VMDK chain. We need to delete the VMDK child that is connected to the volume's backing VM and then add the child VMDK that is attached to or has been detached from the virtual machine instance to the volume's backing VM.
def terminate_connection(self, volume, connector, force=False, **kwargs):
    """Consolidate volume state if possible.

    if not connector['volume_datastore_moref']:
        # The volume is not being detached from the virtual machine instance

    volume_moref = get_volume_by_name(volume['name'])
    datastore_path = get_vmdk_path(volume_moref)
    if datastore_path == connector['volume_datastore_path']:
        # The volume is not moved from its original location.
        # No consolidation is required.

    # The volume has been moved from its original location.
    # We first relocate volume to the new datastore.
    # Delete the existing disk device.
    # Add disk device with connector['volume_datastore_path'] as file backing.

    # Move the volume to the new datastore.
    relocate_volume(volume_moref, connector['volume_datastore'])

    # Create volume_group folder and register volume to that folder

    # Delete disk device.

    # Add new disk with connector['volume_datastore_path'] as file backing.
    add_disk_device(volume_moref, connector['volume_datastore_path'])

Terminate connection.png


  • Here we create snapshot of the volume's VM backing. We use the snapshot ID as name.
  • Note openstack allows snapshoting only detached or available volume.
def create_snapshot(self, snapshot):
    """Snapshot the volume.

    volume_moref = get_volume_by_name(snapshot['volume_name'])
    snapshot_vm(volume_moref, snapshot['name'])


  • Delete the snapshot uniquely identified by name.
def delete_snapshot(self, snapshot):
    """Delete snapshot of the volume.

    volume_moref = get_volume_by_name(snapshot['volume_name'])
    delete_snapshot_by_name(volume_moref, snapshot['name'])


Case 1:

  • This is the default case, where we perform full clone from a snapshot point.
  • We create a new VM backing the new volume. We use VirtualMachineRelocateDiskMoveOptions='moveAllDiskBackingsAndDisallowSharing' as part of clone spec.

Case 2:

  • This is a case where we perform linked clone from snapshot point provided, volume type has extra spec clone_type=fast.
  • We create a new VM backing the new volume. We use VirtualMachineRelocateDiskMoveOptions='createNewChildDiskBacking' as part of clone spec.
def create_volume_from_snapshot(self, volume, snapshot):
    """ Creates volume from given snapshot """

    volume_moref = get_volume_by_name(snapshot['volume_name'])
    snapshot_moref = get_snapshot_by_name(volume_moref, snapshot['volume_name'])
    clone_type = volume_types.get_volume_type(volume['volume_type_id'])

    if clone_type == 'fast':
        relocate_option = 'createNewChildDiskBacking'
        relocate_option = 'moveAllDiskBackingsAndDisallowSharing'

    new_volume_moref = clone_vm_from_snapshot(volume_moref, snapshot_moref, relocate_option)

    return {'metadata':
                {'volume_moref': new_volume_moref}


  • TODO


  • TODO


  • TODO

nova#spawn - booting from a volume workflow

  • TODO

devstack support

The following code will need to be added to devstack (lib/cinder)

   elif [ "$CINDER_DRIVER" == "vsphere" ]; then
       echo_summary "Using VMware vCenter driver"
       iniset $CINDER_CONF DEFAULT enabled_backends vmware
       iniset $CINDER_CONF vmware host_ip "$VMWAREAPI_IP"
       iniset $CINDER_CONF vmware host_username "$VMWAREAPI_USER"
       iniset $CINDER_CONF vmware host_password "$VMWAREAPI_PASSWORD"
       iniset $CINDER_CONF vmware volume_driver "cinder.volume.drivers.vmware.vmdk.VMwareVcVmdkDriver"