Jump to: navigation, search

Difference between revisions of "Cinder/GuestAssistedSnapshotting"

(Snapshot creation: Include finalize-snapshot-metadata in Nova steps)
 
(11 intermediate revisions by one other user not shown)
Line 1: Line 1:
 
= QEMU guest-assisted snapshotting =
 
= QEMU guest-assisted snapshotting =
  
==== Related blueprints ====
+
== Related blueprints ==
  
 
* https://blueprints.launchpad.net/nova/+spec/qemu-assisted-snapshots
 
* https://blueprints.launchpad.net/nova/+spec/qemu-assisted-snapshots
 
* https://blueprints.launchpad.net/cinder/+spec/qemu-assisted-snapshots
 
* https://blueprints.launchpad.net/cinder/+spec/qemu-assisted-snapshots
 +
* Gerrit topic: https://review.openstack.org/#/q/status:open+branch:master+topic:bp/qemu-assisted-snapshots,n,z
  
==== Goals ====
+
== Goals ==
  
1. Add snapshot support for Cinder backing stores which lack internal snapshots (NFS, Gluster, etc.)
+
# Add snapshot support for Cinder backing stores which lack internal snapshots (NFS, Gluster, etc.)
 +
#  Create snapshots w/ quiesced I/O
 +
#  (Phase 2) Enable snapshotting of all volumes on a VM
  
==== Prerequisites ====  
+
== Prerequisites ==
  
* QEMU/libvirt live snapshot support
+
* QEMU guest agent installed (for quiescing), but will still work if not installed
* QEMU guest agent installed (for quiescing)
 
  
==== Overview ====  
+
== Overview ==
  
 
Currently, GlusterFS + Cinder does not support snapshots.  Snapshot support can be enabled by storing volume data as QCOW2 files on Cinder volumes rather than as flat raw files (as is done today), and leveraging QCOW2's snapshot functionality.
 
Currently, GlusterFS + Cinder does not support snapshots.  Snapshot support can be enabled by storing volume data as QCOW2 files on Cinder volumes rather than as flat raw files (as is done today), and leveraging QCOW2's snapshot functionality.
  
Creation of Snapshot:
 
# User calls new nova API call to execute an assisted snapshot
 
# Nova will quiesce guest (use existing pause functionality if guest assisted quiesce is not available)
 
# Nova will execute snapshot API call in Cinder for each Cinder Volume
 
# Cinder creates snapshot(s)
 
# Nova resumes VM on completion of the snapshot(s)
 
 
The snapshot can then be managed like any other Cinder snapshot.
 
  
Changes required for Cinder QCOW2 volumes:
+
====Creation of Snapshot====
* Cinder code to create them (per-driver code & options)
+
# User calls Cinder's snapshot-create API
* Cinder code to translate/process them for operations like upload-to-image, backup_create, clone
+
# If the volume is detached, Cinder's GlusterFS driver will manipulate the qcow2 files with qemu-img to create a snapshot, skip to below.
* Possibly DB information tracking type (qcow2 or raw) - if needed
+
# If the volume is attached, Cinder will:
 +
##  Create a new snapshot with status 'creating'
 +
##  Create a new (empty) qcow2 file on the GlusterFS share which references the current image as its backing file
 +
##  Call Nova's create_volume_snapshot() via novaclient and give it this filename and type 'qcow2'
 +
##  Nova (compute/libvirt): scan VM for volumes matching the supplied volume_id (disk serial)
 +
##  Nova (compute/libvirt): calls libvirt's createSnapshotXML() operation with REUSE_EXT to create a snapshot
 +
##*    First w/ QUIESCE flag, again w/o QUIESCE if that fails
 +
##* The new qcow2 file (created by Cinder, populated by libvirt) becomes the active image for the VM
 +
#  Record information about current qcow2 active file into volume's snapshot info store (this will be a file stored alongside the volume-<uuid> files, like volume-<uuid>.info.
 +
#  Update the snapshot's status to 'available' or 'error' based on the above
  
== Related Notes ==
 
  
* Some examples on libvirt's blockcommit and blockpull -- http://kashyapc.fedorapeople.org/virt/lc-2012/snapshots-handout.html
+
====Deletion of Snapshot====
* More notes on libvirt qcow2 based block operations -- http://kashyapc.fedorapeople.org/virt/lc-2012
+
# User calls Cinder's snapshot-delete API
 +
# If the volume is detached, Cinder's GlusterFS driver will manipulate the qcow2 files with qemu-img to merge the snapshot, skip to below.
 +
# If the volume is attached, Cinder will:
 +
##  Set snapshot status to 'deleting'
 +
##  Call Nova's delete_volume_snapshot() operation via novaclient with the identifier(*) of the snapshot being merged
 +
##  Nova will call libvirt's blockPull/blockCommit operations as appropriate to merge the snapshot data
 +
##  Nova/libvirt will delete the qcow2 file that is no longer needed
 +
#  Update volume's snapshot info store as needed (if this snapshot was the active one, change active image file)
 +
#  Update the snapshot's status to 'deleted' or 'error_deleting'
  
 +
====Notes====
 +
# With this system, the snapshot can be managed like any other Cinder snapshot.  This implies Horizon support as well.
 +
# The Nova APIs will not be exposed to end users.  They will be extensions only present via the internal admin API.  This API will be similar to keystone's admin API in that it is only intended to be exposed internally to the deployment.  This API does not exist yet, but is being added now for this feature (among other ideas that people have for this for the future).
  
== API Details ==
+
====Volume Attach====
 +
# GlusterFS driver's initialize_connection must read the volume's snapshot info store to determine the appropriate filename for the active image
 +
#* This filename is passed to Nova (as it is today)
  
===New APIs===
 
  
 +
====Snapshot data format====
 +
* Initial volume filename is still volume-<uuid>
 +
* When a snapshot is created, the filename will be volume-<uuid>.<snap_uuid>
 +
** This qcow2 image has a backing file pointer to another volume-<uuid>[.<snap_uuid>] file.
  
====Cinder====
 
=====new volume_actions API "create-snapshot-metadata"=====
 
(Note: this may actually need to be two APIs, or use different parameters to specify "creating" vs "done, created successfully".)
 
  
- Allow creation of a snapshot by providing metadata rather than Cinder creating snapshot. (i.e. it was created by Nova.) Cinder driver snapshot code is not called.
+
====Changes required for Cinder====
- Metadata:
+
* Cinder code to create qcow2 snapshots (per-driver code & options)
            volume_id
+
* Cinder code to translate/process qcow2 snapshots for operations like upload-to-image, backup_create, clone
            created_at
+
* Cinder code to track active image's filename (qcow2 chain information) and use this to determine filename for initialize_connection()
            display_name (maybe?)
+
* Cinder needs to embed novaclient
            display_description (optional?)
 
            size
 
- Leave snapshot in "creating" status
 
- This will be implemented as a volume_action
 
  
=====new snapshot_actions module to be added, similar to volume_actions=====
+
====Changes required for Nova====
 +
* libvirt driver implementation of volume_snapshot()
 +
* compute API support for volume_snapshot()    (nova/compute/api.py, rpcapi.py)
 +
* Nova API changes for volume_snapshot
  
=====new snapshot_actions API "finalize-snapshot-metadata"=====
+
* libvirt driver implementation of volume_snapshot_delete()
- Finalize snapshot creation process, set status to available or failed
+
* compute API support for volume_snapshot_delete()    (nova/compute/api.py, rpcapi.py)
- This will be implemented as a volume_action
+
* Nova API changes for volume_snapshot_delete()
  
=====new snapshot_actions API "snapshot-delete-metadata"=====
+
== New API Details ==
- Deletes a snapshot without performing any real storage operation
 
- Is this needed?  Maybe not if the GlusterFS driver's snapshot-delete is smart enough.
 
  
====Nova====
+
===Nova===
  New API to create snapshots of multiple volumes
+
====New create_volume_snapshot() call====
  - Allow "all volumes" or a subset of volumes to be specified
+
* Can create a single snapshot
  - Available via nova client CLI
+
** Each volume passed in will be provided with the information { volume_uuid, type : 'qcow2' or 'cinder', path: '/path/to/new/qcow2.img' }
 +
** type 'qcow2' = handle via libvirt in Nova
 +
** type 'cinder' = call cinderclient to snapshot a non-qcow2 volume (used if snapshotting multiple volumes at once)
 +
* Intended to be called by Cinder
 +
* Needs to be added to novaclient so that Cinder can call it
  
===Snapshot creation===
+
====New delete_volume_snapshot() call====
Currently, it is assumed that file names are:
+
* Deletes a single volume snapshot
volume-<UUID> for the original volume (as is done today), and volume-<UUID>.<snap-UUID> where snap-UUID is the snapshot which depends directly on this qcow2 file as a backing file.
+
** Takes parameters { volume_uuid, path }
The offline Cinder case works this way, ideally Nova can match the same behavior.
+
** Interacts with libvirt, using blockpull or blockcommit operations to merge the snapshot into the qcow2 chain
  
Case 1: Nova-driven snaps (attached)
+
===Cinder===
 +
'''Note: the below Cinder APIs are not required at this time and will likely be dropped.'''
 +
====new volume_actions API "create-snapshot-metadata"====
  
Nova:
+
* Allow creation of a snapshot by providing metadata rather than Cinder creating snapshot. (i.e. it was created by Nova.) Cinder driver snapshot code is not called.
  Create multiple-snapshots Nova API called, specifying which volumes, or all volumes
+
* Metadata:
    Determine which volumes are local files, which are iSCSI/FC-attached
+
** volume_id
    Create a Cinder snapshot for each volume (cinder create-snapshot-metadata API call)
+
* Leave snapshot in "creating" status
    Call libvirt create-snapshot-as for each local file (on Gluster volume)
+
* This will be implemented as a volume_action
    Finalize Cinder snapshot for each volume (cinder finalize-snapshot-metadata API call)
 
    For other attached Cinder volumes, call Cinder snapshot API
 
   
 
  Done
 
  
Case 2: Cinder-driven snapshot
+
====new snapshot_actions module to be added, similar to volume_actions====
  Only works if volume not attached
 
  create-snapshot goes through GlusterFS driver like a typical Cinder driver snapshot
 
  Snapshot created via qemu-img manipulation
 
  
===Snapshot deletion===
+
====new snapshot_actions API "finalize-snapshot-metadata"====
 +
* Finalize snapshot creation process, set status to available or failed
 +
* This will be implemented as a snapshot_action
  
Case 1: Nova-driven (attached)
+
====new snapshot_actions API "snapshot-delete-metadata"====
 +
* Deletes a snapshot without performing any real storage operation
  
Nova (method 1 - should work today):
 
  Create libvirt snapshot from metadata retrieved from Cinder
 
  Call libvirt blockpull operation (merge base into snapshot)
 
  Rename snapshot file to previous base name (libvirt operation: ?)
 
  Delete snapshot from Cinder (snapshot-delete-metadata, or maybe just snapshot-delete)
 
  
Nova (method 2 - more efficient, may not work today):
+
== Next Phases ==
  Create libvirt snapshot from metadata retrieved from Cinder
+
Port Cinder code out of GlusterFS driver as much as possible so that drivers like NFS can use this same scheme.
  Call libvirt blockcommit operation (merge snapshot into base)
 
    - In current (or at least recent) libvirt, live blockcommit is not supported. Should be added in the future.
 
  Delete snapshot from Cinder (snapshot-delete-metadata, or maybe just snapshot-delete)
 
  
Case 2: Cinder-driven
+
== Related Notes ==
  Only works if volume not attached
+
 
  Delete via qemu-img manipulation, works like a typical Cinder driver
+
* Some examples on libvirt's blockcommit and blockpull -- http://kashyapc.fedorapeople.org/virt/lc-2012/snapshots-handout.html
 +
* More notes on libvirt qcow2 based block operations -- http://kashyapc.fedorapeople.org/virt/lc-2012

Latest revision as of 12:35, 8 August 2013

QEMU guest-assisted snapshotting

Related blueprints

Goals

  1. Add snapshot support for Cinder backing stores which lack internal snapshots (NFS, Gluster, etc.)
  2. Create snapshots w/ quiesced I/O
  3. (Phase 2) Enable snapshotting of all volumes on a VM

Prerequisites

  • QEMU guest agent installed (for quiescing), but will still work if not installed

Overview

Currently, GlusterFS + Cinder does not support snapshots. Snapshot support can be enabled by storing volume data as QCOW2 files on Cinder volumes rather than as flat raw files (as is done today), and leveraging QCOW2's snapshot functionality.


Creation of Snapshot

  1. User calls Cinder's snapshot-create API
  2. If the volume is detached, Cinder's GlusterFS driver will manipulate the qcow2 files with qemu-img to create a snapshot, skip to below.
  3. If the volume is attached, Cinder will:
    1. Create a new snapshot with status 'creating'
    2. Create a new (empty) qcow2 file on the GlusterFS share which references the current image as its backing file
    3. Call Nova's create_volume_snapshot() via novaclient and give it this filename and type 'qcow2'
    4. Nova (compute/libvirt): scan VM for volumes matching the supplied volume_id (disk serial)
    5. Nova (compute/libvirt): calls libvirt's createSnapshotXML() operation with REUSE_EXT to create a snapshot
      • First w/ QUIESCE flag, again w/o QUIESCE if that fails
      • The new qcow2 file (created by Cinder, populated by libvirt) becomes the active image for the VM
  4. Record information about current qcow2 active file into volume's snapshot info store (this will be a file stored alongside the volume-<uuid> files, like volume-<uuid>.info.
  5. Update the snapshot's status to 'available' or 'error' based on the above


Deletion of Snapshot

  1. User calls Cinder's snapshot-delete API
  2. If the volume is detached, Cinder's GlusterFS driver will manipulate the qcow2 files with qemu-img to merge the snapshot, skip to below.
  3. If the volume is attached, Cinder will:
    1. Set snapshot status to 'deleting'
    2. Call Nova's delete_volume_snapshot() operation via novaclient with the identifier(*) of the snapshot being merged
    3. Nova will call libvirt's blockPull/blockCommit operations as appropriate to merge the snapshot data
    4. Nova/libvirt will delete the qcow2 file that is no longer needed
  4. Update volume's snapshot info store as needed (if this snapshot was the active one, change active image file)
  5. Update the snapshot's status to 'deleted' or 'error_deleting'

Notes

  1. With this system, the snapshot can be managed like any other Cinder snapshot. This implies Horizon support as well.
  2. The Nova APIs will not be exposed to end users. They will be extensions only present via the internal admin API. This API will be similar to keystone's admin API in that it is only intended to be exposed internally to the deployment. This API does not exist yet, but is being added now for this feature (among other ideas that people have for this for the future).

Volume Attach

  1. GlusterFS driver's initialize_connection must read the volume's snapshot info store to determine the appropriate filename for the active image
    • This filename is passed to Nova (as it is today)


Snapshot data format

  • Initial volume filename is still volume-<uuid>
  • When a snapshot is created, the filename will be volume-<uuid>.<snap_uuid>
    • This qcow2 image has a backing file pointer to another volume-<uuid>[.<snap_uuid>] file.


Changes required for Cinder

  • Cinder code to create qcow2 snapshots (per-driver code & options)
  • Cinder code to translate/process qcow2 snapshots for operations like upload-to-image, backup_create, clone
  • Cinder code to track active image's filename (qcow2 chain information) and use this to determine filename for initialize_connection()
  • Cinder needs to embed novaclient

Changes required for Nova

  • libvirt driver implementation of volume_snapshot()
  • compute API support for volume_snapshot() (nova/compute/api.py, rpcapi.py)
  • Nova API changes for volume_snapshot
  • libvirt driver implementation of volume_snapshot_delete()
  • compute API support for volume_snapshot_delete() (nova/compute/api.py, rpcapi.py)
  • Nova API changes for volume_snapshot_delete()

New API Details

Nova

New create_volume_snapshot() call

  • Can create a single snapshot
    • Each volume passed in will be provided with the information { volume_uuid, type : 'qcow2' or 'cinder', path: '/path/to/new/qcow2.img' }
    • type 'qcow2' = handle via libvirt in Nova
    • type 'cinder' = call cinderclient to snapshot a non-qcow2 volume (used if snapshotting multiple volumes at once)
  • Intended to be called by Cinder
  • Needs to be added to novaclient so that Cinder can call it

New delete_volume_snapshot() call

  • Deletes a single volume snapshot
    • Takes parameters { volume_uuid, path }
    • Interacts with libvirt, using blockpull or blockcommit operations to merge the snapshot into the qcow2 chain

Cinder

Note: the below Cinder APIs are not required at this time and will likely be dropped.

new volume_actions API "create-snapshot-metadata"

  • Allow creation of a snapshot by providing metadata rather than Cinder creating snapshot. (i.e. it was created by Nova.) Cinder driver snapshot code is not called.
  • Metadata:
    • volume_id
  • Leave snapshot in "creating" status
  • This will be implemented as a volume_action

new snapshot_actions module to be added, similar to volume_actions

new snapshot_actions API "finalize-snapshot-metadata"

  • Finalize snapshot creation process, set status to available or failed
  • This will be implemented as a snapshot_action

new snapshot_actions API "snapshot-delete-metadata"

  • Deletes a snapshot without performing any real storage operation


Next Phases

Port Cinder code out of GlusterFS driver as much as possible so that drivers like NFS can use this same scheme.

Related Notes