Openstack should have the ability to take a snapshot of a running instance that includes all of its attached volumes. A coordinated snapshot of multiple volumes for backup purposes. The snapshot operation should occur while the instance is in a paused and quiesced state so that each snapshot is both consistent within itself and with respect to its sibling snapshots.
There are two snapshot operations that exist in Openstack at the moment.
- A user can create an image of a running instance using Nova's createImage API call. This will create a "template" that can be used to spin up new instances from and is referred to by Glance, regardless of where it is stored. This operation will only capture the contents of the root volume.
- A user can create a snapshot of an attached volume through Cinder. This snapshot is a new volume that can be managed just as any other Cinder volume.
The user must choose between one of the two single-volume calls, thus I propose adding functionality that allows a user to create a snapshot that includes both the root volume and all attached cinder volumes in a single pseudo-atomic operation.
Maybe Icehouse 3 dependent on required libvirt changes. Libvirt may need to expose additional operations for this feature to be properly supported. See below for further details.
There are two paths that can be taken and I'm hoping to get consensus on the better path.
- Nova already has a command createImage for creating an image of an existing instance. This command could be extended to take an additional parameter all-volumes that signals the underlying code to capture all attached volumes in addition to the root volume. The semantic here is important, createImage is used to create a template image stored in Glance for later reuse. If the primary intent of this new feature is for backup only, then it may not be wise to overlap the two operations in this way. On the other hand, this approach would introduce the least amount of change to the existing API, requiring only modification of an existing command instead of the addition of an entirely new one.
- If the feature's primary use is for backup purposes, then a new API call may be a better approach, and leave createImage untouched. This new call could be called createBackup and take as a parameter the name of the instance. Although it introduces a new member to the API reference, it would allow this feature to evolve without introducing regressions in any existing calls. These two calls could share code at some point in the future.
We should be able to support crash-consistent snapshots without too many requirements on libvirt by simply pausing the VM, requesting snapshots, and resuming. A stretch-goal would be to quiesce the instance to provide filesystem-consistent snapshots. The end goal is to use guest-assisted snapshots for application-consistent snapshots across all volumes.
Volume Naming Scheme
We will need a way for the user to tell which Cinder volumes are members of a particular snapshot. I propose we can store the Glance snapshot UUID in the volume snapshot metadata for each Cinder volume.
New Snapshot Logic
The basic pseudocode might look like:
volumes = get_all_attached_volumes() PAUSE(quiesce=True) UUID = snapshot_root_volume() for volume in volumes: snapshot_volume(volume, UUID) [volume_api.create_snapshot() via cinder] wait for snaphots to complete RESUME() return response
In the createImage path, Nova issues a call to libvirt's snapshot routine. This routine performs a pause, quiesce, snapshot, and resume in one operation as viewed from Nova. To implement multi-volume snapshots, each of these sub-actions must be exposed by libvirt so that Nova can control the snapshot flow. In other words, Nova must be able to issue separate pause, quiesce, snapshot, and resume commands so that the correct logic can be implemented from the Nova layer.
I still have some pending questions about what is possible in the current version of libvirt.
- Can I request a snapshot for a VM that's already in a paused state?
- Can I quiesce a VM that's in a paused state without requesting a snapshot?
Depending on the answers here, libvirt may require modifications before multi-volume snapshots can be implemented in Nova.
Initially I assumed that the snapshots would be stored next to their parents, whether that be Glance or Cinder. But it might be nice to combine all of the snapshot images into a single OVF file that contains all volumes attached to the instance at the time of snapshot. Additional metadata could be included such as RAM and CPU architecture. The OVF file could then be uploaded to Glance so new instances could be created from it.