Jump to: navigation, search

Difference between revisions of "Cinder/GuestAssistedSnapshotting"

m (→‎Open Questions)
(Redesign: More knowledge of on-disk data handled by Cinder rather than Nova. Snap filename scheme changes, original filename no longer re-used for snapshots.)
Line 1: Line 1:
 
= QEMU guest-assisted snapshotting =
 
= QEMU guest-assisted snapshotting =
  
−
==== Related blueprints ====
+
== Related blueprints ==
  
 
* https://blueprints.launchpad.net/nova/+spec/qemu-assisted-snapshots
 
* https://blueprints.launchpad.net/nova/+spec/qemu-assisted-snapshots
Line 7: Line 7:
 
* Gerrit topic: https://review.openstack.org/#/q/status:open+branch:master+topic:bp/qemu-assisted-snapshots,n,z
 
* Gerrit topic: https://review.openstack.org/#/q/status:open+branch:master+topic:bp/qemu-assisted-snapshots,n,z
  
−
==== Goals ====
+
== Goals ==
  
−
1. Add snapshot support for Cinder backing stores which lack internal snapshots (NFS, Gluster, etc.)
+
# Add snapshot support for Cinder backing stores which lack internal snapshots (NFS, Gluster, etc.)
 +
#  Create snapshots w/ quiesced I/O
 +
#  Enable snapshotting of all volumes on a VM
  
−
==== Prerequisites ====  
+
== Prerequisites ==
  
 
* QEMU/libvirt live snapshot support
 
* QEMU/libvirt live snapshot support
 
* QEMU guest agent installed (for quiescing)
 
* QEMU guest agent installed (for quiescing)
  
−
==== Overview ====  
+
== Overview ==
  
 
Currently, GlusterFS + Cinder does not support snapshots.  Snapshot support can be enabled by storing volume data as QCOW2 files on Cinder volumes rather than as flat raw files (as is done today), and leveraging QCOW2's snapshot functionality.
 
Currently, GlusterFS + Cinder does not support snapshots.  Snapshot support can be enabled by storing volume data as QCOW2 files on Cinder volumes rather than as flat raw files (as is done today), and leveraging QCOW2's snapshot functionality.
  
−
Creation of Snapshot:
 
−
# User calls new nova API call to execute an assisted snapshot
 
−
# Nova calls Cinder's create-snapshot-metadata call for each volume
 
−
# Nova will quiesce and snapshot guest through libvirt create_snapshot
 
−
# Based on that result, call Cinder's finalize-snapshot-metadata call for each snapshot
 
  
−
The snapshot can then be managed like any other Cinder snapshot.
+
====Creation of Snapshot====
 +
# User calls Cinder's snapshot-create API
 +
# If the volume is detached, Cinder's GlusterFS driver will manipulate the qcow2 files with qemu-img to create a snapshot, skip to below.
 +
# If the volume is attached, Cinder will:
 +
##  Create a new snapshot with status 'creating'
 +
##  Create a new (empty) qcow2 file on the GlusterFS share which references the current image as its backing file
 +
##  Call Nova's create_volume_snapshot() operation via novaclient and give it this filename and type 'qcow2'
 +
##  Nova (compute/libvirt): scan VM for volumes matching the supplied volume_id (disk serial)
 +
##  Nova (compute/libvirt): calls libvirt's createSnapshotXML() operation with REUSE_EXT to create a snapshot
 +
##*    First w/ QUIESCE flag, again w/o QUIESCE if that fails
 +
##* The new qcow2 file (created by Cinder, populated by libvirt) becomes the active image for the VM
 +
#  Record information about current qcow2 active file into volume's snapshot info store (this will be a file stored alongside the volume-<uuid> files, like volume-<uuid>.info.
 +
#  Update the snapshot's status to 'available' or 'error' based on the above
  
−
Changes required for Cinder QCOW2 volumes:
 
−
* Cinder code to create them (per-driver code & options)
 
−
* Cinder code to translate/process them for operations like upload-to-image, backup_create, clone
 
−
* Possibly DB information tracking type (qcow2 or raw) - if needed (does not seem to be needed)
 
  
−
== Related Notes ==
+
====Deletion of Snapshot====
 +
# User calls Cinder's snapshot-delete API
 +
# If the volume is detached, Cinder's GlusterFS driver will manipulate the qcow2 files with qemu-img to merge the snapshot, skip to below.
 +
# If the volume is attached, Cinder will:
 +
##  Set snapshot status to 'deleting'
 +
##  Call Nova's delete_volume_snapshot() operation via novaclient with the identifier(*) of the snapshot being merged
 +
##  Nova will call libvirt's blockPull/blockCommit operations as appropriate to merge the snapshot data
 +
##  Nova/libvirt will delete the qcow2 file that is no longer needed
 +
#  Update volume's snapshot info store as needed (if this snapshot was the active one, change active image file)
 +
#  Update the snapshot's status to 'deleted' or 'error_deleting'
  
−
* Some examples on libvirt's blockcommit and blockpull -- http://kashyapc.fedorapeople.org/virt/lc-2012/snapshots-handout.html
+
====Notes====
−
* More notes on libvirt qcow2 based block operations -- http://kashyapc.fedorapeople.org/virt/lc-2012
+
With this system, the snapshot can be managed like any other Cinder snapshot. This implies Horizon support as well.
  
  
−
== API Details ==
+
====Volume Attach====
 +
# GlusterFS driver's initialize_connection must read the volume's snapshot info store to determine the appropriate filename for the active image
 +
#* This filename is passed to Nova (as it is today)
  
−
===New APIs===
 
  
 +
====Snapshot data format====
 +
* Initial volume filename is still volume-<uuid>
 +
* When a snapshot is created, the filename will be volume-<uuid>.<snap_uuid>
 +
** This qcow2 image has a backing file pointer to another volume-<uuid>[.<snap_uuid>] file.
  
−
====Cinder====
 
−
=====new volume_actions API "create-snapshot-metadata"=====
 
−
(Note: this may actually need to be two APIs, or use different parameters to specify "creating" vs "done, created successfully".)
 
  
−
- Allow creation of a snapshot by providing metadata rather than Cinder creating snapshot. (i.e. it was created by Nova.) Cinder driver snapshot code is not called.
+
====Changes required for Cinder====
−
- Metadata:
+
* Cinder code to create qcow2 snapshots (per-driver code & options)
−
            volume_id
+
* Cinder code to translate/process qcow2 snapshots for operations like upload-to-image, backup_create, clone
−
            created_at
+
* Cinder code to track active image's filename (qcow2 chain information) and use this to determine filename for initialize_connection()
−
            display_name (maybe?)
+
* Cinder needs to embed novaclient
−
            display_description (optional?)
 
−
            size
 
−
- Leave snapshot in "creating" status
 
−
- This will be implemented as a volume_action
 
  
−
=====new snapshot_actions module to be added, similar to volume_actions=====
+
====Changes required for Nova====
 +
* libvirt driver implementation of volume_snapshot()
 +
* compute API support for volume_snapshot()    (nova/compute/api.py, rpcapi.py)
 +
* Nova API changes for volume_snapshot
  
−
=====new snapshot_actions API "finalize-snapshot-metadata"=====
+
* libvirt driver implementation of volume_snapshot_delete()
−
- Finalize snapshot creation process, set status to available or failed
+
* compute API support for volume_snapshot_delete()    (nova/compute/api.py, rpcapi.py)
−
- This will be implemented as a volume_action
+
* Nova API changes for volume_snapshot_delete()
  
−
=====new snapshot_actions API "snapshot-delete-metadata"=====
+
== New API Details ==
−
- Deletes a snapshot without performing any real storage operation
 
  
−
====Nova====
+
===Nova===
−
  New API to create snapshots of multiple volumes
+
====New create_volume_snapshots() call====
−
  - Allow "all volumes" or a subset of volumes to be specified
+
* Can create snapshots of multiple volumes
−
  - Available via nova client CLI
+
** Each volume passed in will be provided with the information { volume_uuid, type : 'qcow2' or 'cinder', path: '/path/to/new/qcow2.img' }
 +
** type 'qcow2' = handle via libvirt in Nova
 +
** type 'cinder' = call cinderclient to snapshot a non-qcow2 volume
 +
* Allow a set of volumes to be specified
 +
* Intended to be called by Cinder
 +
* Needs to be piped into novaclient so that Cinder can call it
  
−
===Snapshot creation===
 
−
Currently, it is assumed that file names are:
 
−
volume-<UUID> for the original volume (as is done today), and volume-<UUID>.<snap-UUID> where snap-UUID is the snapshot which depends directly on this qcow2 file as a backing file.
 
−
The offline Cinder case works this way, ideally Nova can match the same behavior.
 
  
−
Case 1: Nova-driven snaps (attached)
+
===Cinder===
 +
'''Note: the below Cinder APIs are not required at this time and will likely be dropped.'''
 +
====new volume_actions API "create-snapshot-metadata"====
  
−
Nova:
+
* Allow creation of a snapshot by providing metadata rather than Cinder creating snapshot. (i.e. it was created by Nova.) Cinder driver snapshot code is not called.
−
  Create multiple-snapshots Nova API called, specifying which volumes, or all volumes
+
* Metadata:
−
    Determine which volumes are local files, which are iSCSI/FC-attached
+
** volume_id
−
    Create a Cinder snapshot for each volume (cinder create-snapshot-metadata API call)
+
* Leave snapshot in "creating" status
−
    Call libvirt create-snapshot-as for each local file (on Gluster volume)
+
* This will be implemented as a volume_action
−
    Finalize Cinder snapshot for each volume (cinder finalize-snapshot-metadata API call)
 
−
    For other attached Cinder volumes, call Cinder snapshot API
 
−
   
 
−
  Done
 
  
−
Case 2: Cinder-driven snapshot
+
====new snapshot_actions module to be added, similar to volume_actions====
−
  Only works if volume not attached
 
−
  create-snapshot goes through GlusterFS driver like a typical Cinder driver snapshot
 
−
  Snapshot created via qemu-img manipulation
 
  
−
===Snapshot deletion===
+
====new snapshot_actions API "finalize-snapshot-metadata"====
 +
* Finalize snapshot creation process, set status to available or failed
 +
* This will be implemented as a snapshot_action
  
−
Case 1: Nova-driven (attached)
+
====new snapshot_actions API "snapshot-delete-metadata"====
 +
* Deletes a snapshot without performing any real storage operation
  
−
Nova (method 1 - should work today):
 
−
  Create libvirt snapshot from metadata retrieved from Cinder
 
−
  Call libvirt blockpull operation (merge base into snapshot)
 
−
  Rename snapshot file to previous base name (libvirt operation: ?)
 
−
  Delete snapshot from Cinder (snapshot-delete-metadata, or maybe just snapshot-delete)
 
  
−
Nova (method 2 - more efficient, may not work today):
+
== Next Phases ==
−
  Create libvirt snapshot from metadata retrieved from Cinder
+
Port Cinder code out of GlusterFS driver as much as possible so that drivers like NFS can use this same scheme.
−
  Call libvirt blockcommit operation (merge snapshot into base)
 
−
    - In current (or at least recent) libvirt, live blockcommit is not supported. Should be added in the future.
 
−
  Delete snapshot from Cinder (snapshot-delete-metadata, or maybe just snapshot-delete)
 
  
−
Case 2: Cinder-driven
+
== Related Notes ==
−
  Only works if volume not attached
 
−
  Delete via qemu-img manipulation, works like a typical Cinder driver
 
  
−
==Open Questions==
+
* Some examples on libvirt's blockcommit and blockpull -- http://kashyapc.fedorapeople.org/virt/lc-2012/snapshots-handout.html
−
Cinder needs to be able to call Nova's new create-snapshots API when the snapshot-create operation is called.
+
* More notes on libvirt qcow2 based block operations -- http://kashyapc.fedorapeople.org/virt/lc-2012
−
 
 
−
Need to understand situations like Cinder asking to detach a volume while a snap create/delete is running on an attached volume.
 

Revision as of 20:11, 7 August 2013

QEMU guest-assisted snapshotting

Related blueprints

Goals

  1. Add snapshot support for Cinder backing stores which lack internal snapshots (NFS, Gluster, etc.)
  2. Create snapshots w/ quiesced I/O
  3. Enable snapshotting of all volumes on a VM

Prerequisites

  • QEMU/libvirt live snapshot support
  • QEMU guest agent installed (for quiescing)

Overview

Currently, GlusterFS + Cinder does not support snapshots. Snapshot support can be enabled by storing volume data as QCOW2 files on Cinder volumes rather than as flat raw files (as is done today), and leveraging QCOW2's snapshot functionality.


Creation of Snapshot

  1. User calls Cinder's snapshot-create API
  2. If the volume is detached, Cinder's GlusterFS driver will manipulate the qcow2 files with qemu-img to create a snapshot, skip to below.
  3. If the volume is attached, Cinder will:
    1. Create a new snapshot with status 'creating'
    2. Create a new (empty) qcow2 file on the GlusterFS share which references the current image as its backing file
    3. Call Nova's create_volume_snapshot() operation via novaclient and give it this filename and type 'qcow2'
    4. Nova (compute/libvirt): scan VM for volumes matching the supplied volume_id (disk serial)
    5. Nova (compute/libvirt): calls libvirt's createSnapshotXML() operation with REUSE_EXT to create a snapshot
      • First w/ QUIESCE flag, again w/o QUIESCE if that fails
      • The new qcow2 file (created by Cinder, populated by libvirt) becomes the active image for the VM
  4. Record information about current qcow2 active file into volume's snapshot info store (this will be a file stored alongside the volume-<uuid> files, like volume-<uuid>.info.
  5. Update the snapshot's status to 'available' or 'error' based on the above


Deletion of Snapshot

  1. User calls Cinder's snapshot-delete API
  2. If the volume is detached, Cinder's GlusterFS driver will manipulate the qcow2 files with qemu-img to merge the snapshot, skip to below.
  3. If the volume is attached, Cinder will:
    1. Set snapshot status to 'deleting'
    2. Call Nova's delete_volume_snapshot() operation via novaclient with the identifier(*) of the snapshot being merged
    3. Nova will call libvirt's blockPull/blockCommit operations as appropriate to merge the snapshot data
    4. Nova/libvirt will delete the qcow2 file that is no longer needed
  4. Update volume's snapshot info store as needed (if this snapshot was the active one, change active image file)
  5. Update the snapshot's status to 'deleted' or 'error_deleting'

Notes

With this system, the snapshot can be managed like any other Cinder snapshot. This implies Horizon support as well.


Volume Attach

  1. GlusterFS driver's initialize_connection must read the volume's snapshot info store to determine the appropriate filename for the active image
    • This filename is passed to Nova (as it is today)


Snapshot data format

  • Initial volume filename is still volume-<uuid>
  • When a snapshot is created, the filename will be volume-<uuid>.<snap_uuid>
    • This qcow2 image has a backing file pointer to another volume-<uuid>[.<snap_uuid>] file.


Changes required for Cinder

  • Cinder code to create qcow2 snapshots (per-driver code & options)
  • Cinder code to translate/process qcow2 snapshots for operations like upload-to-image, backup_create, clone
  • Cinder code to track active image's filename (qcow2 chain information) and use this to determine filename for initialize_connection()
  • Cinder needs to embed novaclient

Changes required for Nova

  • libvirt driver implementation of volume_snapshot()
  • compute API support for volume_snapshot() (nova/compute/api.py, rpcapi.py)
  • Nova API changes for volume_snapshot
  • libvirt driver implementation of volume_snapshot_delete()
  • compute API support for volume_snapshot_delete() (nova/compute/api.py, rpcapi.py)
  • Nova API changes for volume_snapshot_delete()

New API Details

Nova

New create_volume_snapshots() call

  • Can create snapshots of multiple volumes
    • Each volume passed in will be provided with the information { volume_uuid, type : 'qcow2' or 'cinder', path: '/path/to/new/qcow2.img' }
    • type 'qcow2' = handle via libvirt in Nova
    • type 'cinder' = call cinderclient to snapshot a non-qcow2 volume
  • Allow a set of volumes to be specified
  • Intended to be called by Cinder
  • Needs to be piped into novaclient so that Cinder can call it


Cinder

Note: the below Cinder APIs are not required at this time and will likely be dropped.

new volume_actions API "create-snapshot-metadata"

  • Allow creation of a snapshot by providing metadata rather than Cinder creating snapshot. (i.e. it was created by Nova.) Cinder driver snapshot code is not called.
  • Metadata:
    • volume_id
  • Leave snapshot in "creating" status
  • This will be implemented as a volume_action

new snapshot_actions module to be added, similar to volume_actions

new snapshot_actions API "finalize-snapshot-metadata"

  • Finalize snapshot creation process, set status to available or failed
  • This will be implemented as a snapshot_action

new snapshot_actions API "snapshot-delete-metadata"

  • Deletes a snapshot without performing any real storage operation


Next Phases

Port Cinder code out of GlusterFS driver as much as possible so that drivers like NFS can use this same scheme.

Related Notes

Retrieved from "https://wiki.openstack.org/w/index.php?title=Cinder/GuestAssistedSnapshotting&oldid=27701"