Jump to: navigation, search

Difference between revisions of "Cinder/GuestAssistedSnapshotting"

m (Open Questions)
 
(4 intermediate revisions by one other user not shown)
Line 1: Line 1:
 
= QEMU guest-assisted snapshotting =
 
= QEMU guest-assisted snapshotting =
  
==== Related blueprints ====
+
== Related blueprints ==
  
 
* https://blueprints.launchpad.net/nova/+spec/qemu-assisted-snapshots
 
* https://blueprints.launchpad.net/nova/+spec/qemu-assisted-snapshots
Line 7: Line 7:
 
* Gerrit topic: https://review.openstack.org/#/q/status:open+branch:master+topic:bp/qemu-assisted-snapshots,n,z
 
* Gerrit topic: https://review.openstack.org/#/q/status:open+branch:master+topic:bp/qemu-assisted-snapshots,n,z
  
==== Goals ====
+
== Goals ==
  
1. Add snapshot support for Cinder backing stores which lack internal snapshots (NFS, Gluster, etc.)
+
# Add snapshot support for Cinder backing stores which lack internal snapshots (NFS, Gluster, etc.)
 +
#  Create snapshots w/ quiesced I/O
 +
#  (Phase 2) Enable snapshotting of all volumes on a VM
  
==== Prerequisites ====  
+
== Prerequisites ==
  
* QEMU/libvirt live snapshot support
+
* QEMU guest agent installed (for quiescing), but will still work if not installed
* QEMU guest agent installed (for quiescing)
 
  
==== Overview ====  
+
== Overview ==
  
 
Currently, GlusterFS + Cinder does not support snapshots.  Snapshot support can be enabled by storing volume data as QCOW2 files on Cinder volumes rather than as flat raw files (as is done today), and leveraging QCOW2's snapshot functionality.
 
Currently, GlusterFS + Cinder does not support snapshots.  Snapshot support can be enabled by storing volume data as QCOW2 files on Cinder volumes rather than as flat raw files (as is done today), and leveraging QCOW2's snapshot functionality.
  
Creation of Snapshot:
 
# User calls new nova API call to execute an assisted snapshot
 
# Nova calls Cinder's create-snapshot-metadata call for each volume
 
# Nova will quiesce and snapshot guest through libvirt create_snapshot
 
# Based on that result, call Cinder's finalize-snapshot-metadata call for each snapshot
 
  
The snapshot can then be managed like any other Cinder snapshot.
+
====Creation of Snapshot====
 +
# User calls Cinder's snapshot-create API
 +
# If the volume is detached, Cinder's GlusterFS driver will manipulate the qcow2 files with qemu-img to create a snapshot, skip to below.
 +
# If the volume is attached, Cinder will:
 +
##  Create a new snapshot with status 'creating'
 +
##  Create a new (empty) qcow2 file on the GlusterFS share which references the current image as its backing file
 +
##  Call Nova's create_volume_snapshot() via novaclient and give it this filename and type 'qcow2'
 +
##  Nova (compute/libvirt): scan VM for volumes matching the supplied volume_id (disk serial)
 +
##  Nova (compute/libvirt): calls libvirt's createSnapshotXML() operation with REUSE_EXT to create a snapshot
 +
##*    First w/ QUIESCE flag, again w/o QUIESCE if that fails
 +
##* The new qcow2 file (created by Cinder, populated by libvirt) becomes the active image for the VM
 +
#  Record information about current qcow2 active file into volume's snapshot info store (this will be a file stored alongside the volume-<uuid> files, like volume-<uuid>.info.
 +
#  Update the snapshot's status to 'available' or 'error' based on the above
  
Changes required for Cinder QCOW2 volumes:
 
* Cinder code to create them (per-driver code & options)
 
* Cinder code to translate/process them for operations like upload-to-image, backup_create, clone
 
* Possibly DB information tracking type (qcow2 or raw) - if needed (does not seem to be needed)
 
  
== Related Notes ==
+
====Deletion of Snapshot====
 +
# User calls Cinder's snapshot-delete API
 +
# If the volume is detached, Cinder's GlusterFS driver will manipulate the qcow2 files with qemu-img to merge the snapshot, skip to below.
 +
# If the volume is attached, Cinder will:
 +
##  Set snapshot status to 'deleting'
 +
##  Call Nova's delete_volume_snapshot() operation via novaclient with the identifier(*) of the snapshot being merged
 +
##  Nova will call libvirt's blockPull/blockCommit operations as appropriate to merge the snapshot data
 +
##  Nova/libvirt will delete the qcow2 file that is no longer needed
 +
#  Update volume's snapshot info store as needed (if this snapshot was the active one, change active image file)
 +
#  Update the snapshot's status to 'deleted' or 'error_deleting'
  
* Some examples on libvirt's blockcommit and blockpull -- http://kashyapc.fedorapeople.org/virt/lc-2012/snapshots-handout.html
+
====Notes====
* More notes on libvirt qcow2 based block operations -- http://kashyapc.fedorapeople.org/virt/lc-2012
+
# With this system, the snapshot can be managed like any other Cinder snapshot.  This implies Horizon support as well.
 +
# The Nova APIs will not be exposed to end users. They will be extensions only present via the internal admin API. This API will be similar to keystone's admin API in that it is only intended to be exposed internally to the deployment. This API does not exist yet, but is being added now for this feature (among other ideas that people have for this for the future).
  
 +
====Volume Attach====
 +
# GlusterFS driver's initialize_connection must read the volume's snapshot info store to determine the appropriate filename for the active image
 +
#* This filename is passed to Nova (as it is today)
  
== API Details ==
 
  
===New APIs===
+
====Snapshot data format====
 +
* Initial volume filename is still volume-<uuid>
 +
* When a snapshot is created, the filename will be volume-<uuid>.<snap_uuid>
 +
** This qcow2 image has a backing file pointer to another volume-<uuid>[.<snap_uuid>] file.
  
  
====Cinder====
+
====Changes required for Cinder====
=====new volume_actions API "create-snapshot-metadata"=====
+
* Cinder code to create qcow2 snapshots (per-driver code & options)
(Note: this may actually need to be two APIs, or use different parameters to specify "creating" vs "done, created successfully".)
+
* Cinder code to translate/process qcow2 snapshots for operations like upload-to-image, backup_create, clone
 +
* Cinder code to track active image's filename (qcow2 chain information) and use this to determine filename for initialize_connection()
 +
* Cinder needs to embed novaclient
  
- Allow creation of a snapshot by providing metadata rather than Cinder creating snapshot. (i.e. it was created by Nova.) Cinder driver snapshot code is not called.
+
====Changes required for Nova====
- Metadata:
+
* libvirt driver implementation of volume_snapshot()
            volume_id
+
* compute API support for volume_snapshot()   (nova/compute/api.py, rpcapi.py)
            created_at
+
* Nova API changes for volume_snapshot
            display_name (maybe?)
 
            display_description (optional?)
 
            size
 
- Leave snapshot in "creating" status
 
- This will be implemented as a volume_action
 
  
=====new snapshot_actions module to be added, similar to volume_actions=====
+
* libvirt driver implementation of volume_snapshot_delete()
 +
* compute API support for volume_snapshot_delete()    (nova/compute/api.py, rpcapi.py)
 +
* Nova API changes for volume_snapshot_delete()
  
=====new snapshot_actions API "finalize-snapshot-metadata"=====
+
== New API Details ==
- Finalize snapshot creation process, set status to available or failed
 
- This will be implemented as a volume_action
 
  
=====new snapshot_actions API "snapshot-delete-metadata"=====
+
===Nova===
- Deletes a snapshot without performing any real storage operation
+
====New create_volume_snapshot() call====
 +
* Can create a single snapshot
 +
** Each volume passed in will be provided with the information { volume_uuid, type : 'qcow2' or 'cinder', path: '/path/to/new/qcow2.img' }
 +
** type 'qcow2' = handle via libvirt in Nova
 +
** type 'cinder' = call cinderclient to snapshot a non-qcow2 volume (used if snapshotting multiple volumes at once)
 +
* Intended to be called by Cinder
 +
* Needs to be added to novaclient so that Cinder can call it
  
====Nova====
+
====New delete_volume_snapshot() call====
  New API to create snapshots of multiple volumes
+
* Deletes a single volume snapshot
  - Allow "all volumes" or a subset of volumes to be specified
+
** Takes parameters { volume_uuid, path }
  - Available via nova client CLI
+
** Interacts with libvirt, using blockpull or blockcommit operations to merge the snapshot into the qcow2 chain
  
===Snapshot creation===
+
===Cinder===
Currently, it is assumed that file names are:
+
'''Note: the below Cinder APIs are not required at this time and will likely be dropped.'''
volume-<UUID> for the original volume (as is done today), and volume-<UUID>.<snap-UUID> where snap-UUID is the snapshot which depends directly on this qcow2 file as a backing file.
+
====new volume_actions API "create-snapshot-metadata"====
The offline Cinder case works this way, ideally Nova can match the same behavior.
 
  
Case 1: Nova-driven snaps (attached)
+
* Allow creation of a snapshot by providing metadata rather than Cinder creating snapshot. (i.e. it was created by Nova.) Cinder driver snapshot code is not called.
 +
* Metadata:
 +
** volume_id
 +
* Leave snapshot in "creating" status
 +
* This will be implemented as a volume_action
  
Nova:
+
====new snapshot_actions module to be added, similar to volume_actions====
  Create multiple-snapshots Nova API called, specifying which volumes, or all volumes
 
    Determine which volumes are local files, which are iSCSI/FC-attached
 
    Create a Cinder snapshot for each volume (cinder create-snapshot-metadata API call)
 
    Call libvirt create-snapshot-as for each local file (on Gluster volume)
 
    Finalize Cinder snapshot for each volume (cinder finalize-snapshot-metadata API call)
 
    For other attached Cinder volumes, call Cinder snapshot API
 
   
 
  Done
 
  
Case 2: Cinder-driven snapshot
+
====new snapshot_actions API "finalize-snapshot-metadata"====
  Only works if volume not attached
+
* Finalize snapshot creation process, set status to available or failed
  create-snapshot goes through GlusterFS driver like a typical Cinder driver snapshot
+
* This will be implemented as a snapshot_action
  Snapshot created via qemu-img manipulation
 
  
===Snapshot deletion===
+
====new snapshot_actions API "snapshot-delete-metadata"====
 +
* Deletes a snapshot without performing any real storage operation
  
Case 1: Nova-driven (attached)
 
  
Nova (method 1 - should work today):
+
== Next Phases ==
  Create libvirt snapshot from metadata retrieved from Cinder
+
Port Cinder code out of GlusterFS driver as much as possible so that drivers like NFS can use this same scheme.
  Call libvirt blockpull operation (merge base into snapshot)
 
  Rename snapshot file to previous base name (libvirt operation: ?)
 
  Delete snapshot from Cinder (snapshot-delete-metadata, or maybe just snapshot-delete)
 
  
Nova (method 2 - more efficient, may not work today):
+
== Related Notes ==
  Create libvirt snapshot from metadata retrieved from Cinder
 
  Call libvirt blockcommit operation (merge snapshot into base)
 
    - In current (or at least recent) libvirt, live blockcommit is not supported.  Should be added in the future.
 
  Delete snapshot from Cinder (snapshot-delete-metadata, or maybe just snapshot-delete)
 
  
Case 2: Cinder-driven
+
* Some examples on libvirt's blockcommit and blockpull -- http://kashyapc.fedorapeople.org/virt/lc-2012/snapshots-handout.html
  Only works if volume not attached
+
* More notes on libvirt qcow2 based block operations -- http://kashyapc.fedorapeople.org/virt/lc-2012
  Delete via qemu-img manipulation, works like a typical Cinder driver
 
 
 
==Open Questions==
 
Cinder needs to be able to call Nova's new create-snapshots API when the snapshot-create operation is called.
 
 
 
Need to understand situations like Cinder asking to detach a volume while a snap create/delete is running on an attached volume.
 

Latest revision as of 12:35, 8 August 2013

QEMU guest-assisted snapshotting

Related blueprints

Goals

  1. Add snapshot support for Cinder backing stores which lack internal snapshots (NFS, Gluster, etc.)
  2. Create snapshots w/ quiesced I/O
  3. (Phase 2) Enable snapshotting of all volumes on a VM

Prerequisites

  • QEMU guest agent installed (for quiescing), but will still work if not installed

Overview

Currently, GlusterFS + Cinder does not support snapshots. Snapshot support can be enabled by storing volume data as QCOW2 files on Cinder volumes rather than as flat raw files (as is done today), and leveraging QCOW2's snapshot functionality.


Creation of Snapshot

  1. User calls Cinder's snapshot-create API
  2. If the volume is detached, Cinder's GlusterFS driver will manipulate the qcow2 files with qemu-img to create a snapshot, skip to below.
  3. If the volume is attached, Cinder will:
    1. Create a new snapshot with status 'creating'
    2. Create a new (empty) qcow2 file on the GlusterFS share which references the current image as its backing file
    3. Call Nova's create_volume_snapshot() via novaclient and give it this filename and type 'qcow2'
    4. Nova (compute/libvirt): scan VM for volumes matching the supplied volume_id (disk serial)
    5. Nova (compute/libvirt): calls libvirt's createSnapshotXML() operation with REUSE_EXT to create a snapshot
      • First w/ QUIESCE flag, again w/o QUIESCE if that fails
      • The new qcow2 file (created by Cinder, populated by libvirt) becomes the active image for the VM
  4. Record information about current qcow2 active file into volume's snapshot info store (this will be a file stored alongside the volume-<uuid> files, like volume-<uuid>.info.
  5. Update the snapshot's status to 'available' or 'error' based on the above


Deletion of Snapshot

  1. User calls Cinder's snapshot-delete API
  2. If the volume is detached, Cinder's GlusterFS driver will manipulate the qcow2 files with qemu-img to merge the snapshot, skip to below.
  3. If the volume is attached, Cinder will:
    1. Set snapshot status to 'deleting'
    2. Call Nova's delete_volume_snapshot() operation via novaclient with the identifier(*) of the snapshot being merged
    3. Nova will call libvirt's blockPull/blockCommit operations as appropriate to merge the snapshot data
    4. Nova/libvirt will delete the qcow2 file that is no longer needed
  4. Update volume's snapshot info store as needed (if this snapshot was the active one, change active image file)
  5. Update the snapshot's status to 'deleted' or 'error_deleting'

Notes

  1. With this system, the snapshot can be managed like any other Cinder snapshot. This implies Horizon support as well.
  2. The Nova APIs will not be exposed to end users. They will be extensions only present via the internal admin API. This API will be similar to keystone's admin API in that it is only intended to be exposed internally to the deployment. This API does not exist yet, but is being added now for this feature (among other ideas that people have for this for the future).

Volume Attach

  1. GlusterFS driver's initialize_connection must read the volume's snapshot info store to determine the appropriate filename for the active image
    • This filename is passed to Nova (as it is today)


Snapshot data format

  • Initial volume filename is still volume-<uuid>
  • When a snapshot is created, the filename will be volume-<uuid>.<snap_uuid>
    • This qcow2 image has a backing file pointer to another volume-<uuid>[.<snap_uuid>] file.


Changes required for Cinder

  • Cinder code to create qcow2 snapshots (per-driver code & options)
  • Cinder code to translate/process qcow2 snapshots for operations like upload-to-image, backup_create, clone
  • Cinder code to track active image's filename (qcow2 chain information) and use this to determine filename for initialize_connection()
  • Cinder needs to embed novaclient

Changes required for Nova

  • libvirt driver implementation of volume_snapshot()
  • compute API support for volume_snapshot() (nova/compute/api.py, rpcapi.py)
  • Nova API changes for volume_snapshot
  • libvirt driver implementation of volume_snapshot_delete()
  • compute API support for volume_snapshot_delete() (nova/compute/api.py, rpcapi.py)
  • Nova API changes for volume_snapshot_delete()

New API Details

Nova

New create_volume_snapshot() call

  • Can create a single snapshot
    • Each volume passed in will be provided with the information { volume_uuid, type : 'qcow2' or 'cinder', path: '/path/to/new/qcow2.img' }
    • type 'qcow2' = handle via libvirt in Nova
    • type 'cinder' = call cinderclient to snapshot a non-qcow2 volume (used if snapshotting multiple volumes at once)
  • Intended to be called by Cinder
  • Needs to be added to novaclient so that Cinder can call it

New delete_volume_snapshot() call

  • Deletes a single volume snapshot
    • Takes parameters { volume_uuid, path }
    • Interacts with libvirt, using blockpull or blockcommit operations to merge the snapshot into the qcow2 chain

Cinder

Note: the below Cinder APIs are not required at this time and will likely be dropped.

new volume_actions API "create-snapshot-metadata"

  • Allow creation of a snapshot by providing metadata rather than Cinder creating snapshot. (i.e. it was created by Nova.) Cinder driver snapshot code is not called.
  • Metadata:
    • volume_id
  • Leave snapshot in "creating" status
  • This will be implemented as a volume_action

new snapshot_actions module to be added, similar to volume_actions

new snapshot_actions API "finalize-snapshot-metadata"

  • Finalize snapshot creation process, set status to available or failed
  • This will be implemented as a snapshot_action

new snapshot_actions API "snapshot-delete-metadata"

  • Deletes a snapshot without performing any real storage operation


Next Phases

Port Cinder code out of GlusterFS driver as much as possible so that drivers like NFS can use this same scheme.

Related Notes