Fuel/External Snapshots

External Snapshots
We consider external snapshots as snapshots of 'system checkpoint' type according to libvirt documentation in its external form.

What is External Snapshot?
External Snapshot is a combination of 2 types of snapshots in external form: External form means snapshot is created not in one file but in different files unlike internal form. Disk snapshot is a couple of files: at least 1 for snapshot itself and another one for writing changes since the snapshot is created (snapshot file - is a backing store for this file). Memory state is also a separate file in case of external form.
 * disk snapshot - contains content of disks
 * memory (or VM) state - tracks only the state of RAM and all other resources in use by the VM

Volume chains in External Snapshots
When working with external snapshots one feature have to be taken into account: multiple sequencial external snapshots of disk create a volumes chain. Every next snapshot is a 'delta' from previous one in a separate volume and all volumes related to parent snapshot itself. And so on down to the base volume. It happens due to design of disk external snapshots.

External Snapshots in Fuel system tests environment
External Snapshot support has been added to fuel-devops application. To start using external snapshots in fuel-devops the only thing to do is to define SNAPSHOTS_EXTERNAL environment variable and set its value to 'true'.

How does it work?
Fuel-devops performs the following steps for every node in an environment.

Create snapshot

 * Creates a volume with currently set disk as a backing store to save changes after snapshot (makes for every disk attached to the VM) This is done to have new volume in fuel-devops database to be able to control the volume in the future.
 * Defines a path for memory state file
 * Makes the snapshot from generated XML definition using snapshot libvirt functionality
 * Redefines the snapshot as a current one

Revert VM to the state from snapshot

 * Destroys a node
 * Chooses volumes to use after VM reverting:
 * If it's the latest snapshot in a chain the same volume will be used (recreate volume)
 * Otherwise new volume will be created
 * Starts VM using restore libvirt functionality

Known Issues
It turned out that external snapshots can't be created when using host-passthrough cpu mode. The reasons why it doesn't work are: Full snapshot XML definition is needed to redefine the snapshot as a current one. To avoid such problem functionality to get cpu model from domain definition and inject one in snapshot definition has been implemented. Also as a workaround host-passthrough mode could be disabled by setting DRIVER_USE_HOST_CPU variable to false before environment creation.
 * virDomainSnapshotGetXMLDesc function does not return cpu model in a snapshot XML definition
 * libvirt forbids changing the cpu model during modifications of snapshot definition

Export and Import environments
Let's assume that we have limited number of nodes (e.g. Jenkins slaves) where it is possible to execute deployment tests. In case of test failures it is necessary to disable the node where it occurred to be able to perform root cause analysis. This leads to scarcity of resources and slow down overall performance of CI environment. In such cases it would be nice to have possibility to export environment from one node (let's name it source node, e.g. Jenkins slave) and import it to another node (target node, e.g. backup node for the troubleshooting and analysis). This way will provide possibility to free the node up from the failed environment and return it back to the CI environment.

As this task is not simple, at the beginning let's consider that the target node for importing environments is empty, i.e. does not contain any nodes, items, and etc. in libvirt environment. Also all local paths and UUID should be the same to avoid modifications in binary files such as memory state files and volumes qcow2 files.

System tests environment consists of the following parts:
 * 1) Records in fuel-devops DB (to be able to work with libvirt objects)
 * 2) Libvirt environment (domains, networks, volumes, snapshots, etc.)

Currently steps to export and import systest environments have been implemented as shell scripts.

Export environment
NOTE: Steps 7 - 9 are implemented in a separate script as it needs root privileges to run. There's a list of steps to export environment (using libvirt virsh and psql) :
 * 1) Export records from fuel-devops DB (psql -c '\copy ...' )
 * 2) Dump definition of storage pool (pool-dumpxml)
 * 3) Dump definitions of networks (net-dumpxml)
 * 4) Dump definitions of volumes (vol-dumpxml)
 * 5) Dump definitions of domains (dumpxml)
 * 6) Download volumes (vol-download). It shows poor performance with libvirt 1.2.12 which is in use now. Related bugs: (https://bugzilla.redhat.com/show_bug.cgi?id=1026136 https://bugzilla.redhat.com/show_bug.cgi?id=1026137). So we need either upgrade libvirt or implement direct copying of the volumes (move to the separate script as it is done for 7-9 steps)
 * 7) Dump definitions of snapshots. At the moment direct copying is used because it was easier to implement, assnapshot-dumpxml returns xml file without cpu model if cpu mode is 'host-passthrough' which is set by default, and without tag needed by libvirt.
 * 8) Set   to the XML definition of the snapshot which will be set as current (1st in the list of snapshots to export) and   to all another snapshots definitions
 * 9) Copy memory state files (for snapshots)

Import environment
NOTE: There're 2 ways to import snapshots: 1. Replay snapshots, i.e. create snapshots in the same order as it was done on the source node   and then redefine ones using exported definitions 2. Add snapshots definitions to /var/lib/libvirt/qemu/snapshot/  directory 1st way is a complex one as in case of many snapshots and complex scheme of snapshot creation order there are many steps to do. Also it needs to take into account that all 'replayed' snapshots must be of the same state as source snapshots respectively. 2nd way is much easier as it needs only to put snapshots definitions to the libvirt snapshots directory and copy memory state files to the same directories as on the source node. To do this we need root privileges. Currently 2nd way is implemented. There's a list of steps to import environment (using libvirt virsh) :
 * 1) Import records to fuel-devops DB (psql -c '\copy ...' )
 * 2) Create a storage pool from the definition and autostart it (pool-define, pool-start, pool-autostart)
 * 3) Create networks from the definitions and autostart ones (net-define, net-start, net-autostart)
 * 4) Create volumes from the definitions (vol-create)
 * 5) Upload content from the exported files to the volumes (vol-upload)
 * 6) Create domains from the definitions (define)
 * 7) Put memory state files to the same paths as ones were placed on the source node
 * 8) Copy snapshot XML definitions to /var/lib/libvirt/qemu/snapshot/  directory
 * 9) Restart libvirt daemon