This guide is for OpenStack operators and does not seek to be an exhaustive reference for users, but as an operator, you should have a basic understanding of how to use the cloud facilities. This chapter looks at OpenStack from a basic user perspective, which helps you understand your users’ needs and determine, when you get a trouble ticket, whether it is a user issue or a service issue. The main concepts covered are images, flavors, security groups, block storage, shared file system storage, and instances.
- 1 Images
- 2 Flavors
- 3 Security Groups
- 4 Block Storage
- 5 Shared File Systems Service
- 6 Instances
- 7 Associating Security Groups
- 8 Floating IPs
- 9 Attaching Block Storage
- 10 Taking Snapshots
- 11 Instances in the Database
- 12 Good Luck
OpenStack images can often be thought of as “virtual machine templates.” Images can also be standard installation media such as ISO images. Essentially, they contain bootable file systems that are used to launch instances.
Several pre-made images exist and can easily be imported into the Image service. A common image to add is the CirrOS image, which is very small and used for testing purposes. To add this image, simply do:
The openstack image create command provides a large set of options for working with your image. For example, the
--min-disk option is useful for images that require root disks of a certain size (for example, large Windows images). To view these options, run:
Run the following command to view the properties of existing images:
Adding Signed Images
To provide a chain of trust from an end user to the Image service, and the Image service to Compute, an end user can import signed images that can be initially verified in the Image service, and later verified in the Compute service. Appropriate Image service properties need to be set to enable this signature feature.
The following are steps needed to create the signature used for the signed images:
Retrieve image for upload
Use private key to create a signature of the image
- Create context
- Encode certificate in DER format
- Upload Certificate in DER format to Castellan
- Upload Image to Image service, with Signature Metadata
Verify the Keystone URL
Signature verification will occur when Compute boots the signed image
Obtain the UUID of the image:
Obtain the UUID of the project with which you want to share your image, let’s call it target project. Unfortunately, non-admin users are unable to use the openstack command to do this. The easiest solution is to obtain the UUID either from an administrator of the cloud or from a user located in the target project.
Once you have both pieces of information, run the openstack image add project command:
You now need to act in the target project scope.
To accept the sharing, you need to update the member status:
771ed149ef7e4b2b88665cc1c98f77cawill now have access to image
You can explicitly ask for pending member status to view shared images not yet accepted:
Generate signature of image and convert it to a base64 representation:
In a multi-tenant cloud environment, users sometimes want to share their personal images or snapshots with other projects. This can be done on the command line with the
glance tool by the owner of the image.
To share an image or snapshot with another project, do the following:
To delete an image, just execute:
Other CLI Options
A full set of options can be found using:
or the Command-Line Interface Reference.
The Image service and the Database
The only thing the Image service does not store in a database is the image itself. The Image service database has two main tables:
Working directly with the database and SQL queries can provide you with custom lists and reports of images. Technically, you can update properties about images through the database, although this is not generally recommended.
Example Image service Database Queries
One interesting example is modifying the table of images and the owner of that image. This can be easily done if you simply display the unique ID of the owner. This example goes one step further and displays the readable name of the owner:
Another example is displaying all properties for a certain image:
Virtual hardware templates are called “flavors” in OpenStack, defining sizes for RAM, disk, number of cores, and so on. The default install provides five flavors.
These are configurable by admin users (the rights may also be delegated to other users by redefining the access controls for
/etc/nova/policy.json on the
nova-api server). To get the list of available flavors on your system, run:
The openstack flavor create command allows authorized users to create new flavors. Additional flavor manipulation commands can be shown with the following command:
Flavors define a number of parameters, resulting in the user having a choice of what type of virtual machine to run—just like they would have if they were purchasing a physical server. Table. Flavor parameters lists the elements that can be set. Note in particular
extra_specs, which can be used to define free-form characteristics, giving a lot of flexibility beyond just the size of RAM, CPU, and Disk.
|ID||Unique ID (integer or UUID) for the flavor.|
|Name||A descriptive name, such as xx.size_name, is conventional but not required, though some third-party tools may rely on it.|
|Memory_MB||Virtual machine memory in megabytes.|
|Disk||Virtual root disk size in gigabytes. This is an ephemeral disk the base image is copied into. You don’t use it when you boot from a persistent volume. The “0” size is a special case that uses the native base image size as the size of the ephemeral root volume.|
|Ephemeral||Specifies the size of a secondary ephemeral data disk. This is an empty, unformatted disk and exists only for the life of the instance.|
|Swap||Optional swap space allocation for the instance.|
|VCPUs||Number of virtual CPUs presented to the instance.|
|RXTX_Factor||Optional property that allows created servers to have a different bandwidth cap from that defined in the network they are attached to. This factor is multiplied by the rxtx_base property of the network. Default value is 1.0 (that is, the same as the attached network).|
|Is_Public|| Boolean value that indicates whether the flavor is available to all users or private. Private flavors do not get the current tenant assigned to them. Defaults to |
|extra_specs||Additional optional restrictions on which compute nodes the flavor can run on. This is implemented as key-value pairs that must match against the corresponding key-value pairs on compute nodes. Can be used to implement things like special resources (such as flavors that can run only on compute nodes with GPU hardware).|
A user might need a custom flavor that is uniquely tuned for a project she is working on. For example, the user might require 128 GB of memory. If you create a new flavor as described above, the user would have access to the custom flavor, but so would all other tenants in your cloud. Sometimes this sharing isn’t desirable. In this scenario, allowing all users to have access to a flavor with 128 GB of memory might cause your cloud to reach full capacity very quickly. To prevent this, you can restrict access to the custom flavor using the nova flavor-access-add command:
To view a flavor’s access list, do the following:
It’s also helpful to allocate a specific numeric range for custom and private flavors. On UNIX-based systems, nonsystem accounts usually have a UID starting at 500. A similar approach can be taken with custom flavors. This helps you easily identify which flavors are custom, private, and public for the entire cloud.
How Do I Modify an Existing Flavor?
The OpenStack dashboard simulates the ability to modify a flavor by deleting an existing flavor and creating a new one with the same name.
A common new-user issue with OpenStack is failing to set an appropriate security group when launching an instance. As a result, the user is unable to contact the instance on the network.
Security groups are sets of IP filter rules that are applied to an instance’s networking. They are project specific, and project members can edit the default rules for their group and add new rules sets. All projects have a “default” security group, which is applied to instances that have no other security group defined. Unless changed, this security group denies all incoming traffic.
End-User Configuration of Security Groups
Security groups for the current project can be found on the OpenStack dashboard under Access & Security. To see details of an existing group, select the Edit Security Group action for that security group. Obviously, modifying existing groups can be done from this edit interface. There is a Create Security Group button on the main Access & Security page for creating new groups. We discuss the terms used in these fields when we explain the command-line equivalents.
Setting with openstack command
If your environment is using Neutron, you can configure security groups settings using the openstack command. Get a list of security groups for the project you are acting in, by using following command:
To view the details of a security group:
These rules are all “allow” type rules, as the default is deny. This example shows the full port range for all protocols allowed from all IPs. This section describes the most common security group rule parameters:
- The direction in which the security group rule is applied. Valid values are
- This attribute value matches the specified IP prefix as the source IP address of the IP packet.
- The protocol that is matched by the security group rule. Valid values are
- The minimum port number in the range that is matched by the security group rule. If the protocol is TCP or UDP, this value must be less than or equal to the
port_range_maxattribute value. If the protocol is ICMP or ICMPv6, this value must be an ICMP or ICMPv6 type, respectively.
- The maximum port number in the range that is matched by the security group rule. The
port_range_minattribute constrains the
port_range_maxattribute. If the protocol is ICMP or ICMPv6, this value must be an ICMP or ICMPv6 type, respectively.
- Must be
IPv6, and addresses represented in CIDR must match the ingress or egress rules.
When adding a new security group, you should pick a descriptive but brief name. This name shows up in brief descriptions of the instances that use it where the longer description field often does not. Seeing that an instance is using security group
http is much easier to understand than
This example creates a security group that allows web traffic anywhere on the Internet. We’ll call this group
global_http, which is clear and reasonably concise, encapsulating what is allowed and from where. From the command line, do:
Immediately after create, the security group has only an allow egress rule. To make it do what we want, we need to add some rules:
Despite only outputting the newly added rule, this operation is additive:
The inverse operation is called openstack security group rule delete, specifying security-group-rule ID. Whole security groups can be removed with openstack security group delete.
To create security group rules for a cluster of instances, use RemoteGroups.
RemoteGroups are a dynamic way of defining the CIDR of allowed sources. The user specifies a RemoteGroup (security group name) and then all the users’ other instances using the specified RemoteGroup are selected dynamically. This dynamic selection alleviates the need for individual rules to allow each new member of the cluster.
The code is similar to the above example of openstack security group rule create. To use RemoteGroup, specify
--remote-group instead of
--remote-ip. For example:
The “cluster” rule allows SSH access from any other instance that uses the
OpenStack volumes are persistent block-storage devices that may be attached and detached from instances, but they can be attached to only one instance at a time. Similar to an external hard drive, they do not provide shared storage in the way a network file system or object store does. It is left to the operating system in the instance to put a file system on the block device and mount it, or not.
As with other removable disk technology, it is important that the operating system is not trying to make use of the disk before removing it. On Linux instances, this typically involves unmounting any file systems mounted from the volume. The OpenStack volume service cannot tell whether it is safe to remove volumes from an instance, so it does what it is told. If a user tells the volume service to detach a volume from an instance while it is being written to, you can expect some level of file system corruption as well as faults from whatever process within the instance was using the device.
There is nothing OpenStack-specific in being aware of the steps needed to access block devices from within the instance operating system, potentially formatting them for first use and being cautious when removing them. What is specific is how to create new volumes and attach and detach them from instances. These operations can all be done from the Volumes page of the dashboard or by using the
openstack command-line client.
To add new volumes, you need only a volume size in gigabytes. Either put these into the Create Volume web form or use the command line:
This creates a 10 GB volume. To list existing volumes and the instances they are connected to, if any:
OpenStack Block Storage also allows creating snapshots of volumes. Remember that this is a block-level snapshot that is crash consistent, so it is best if the volume is not connected to an instance when the snapshot is taken and second best if the volume is not in use on the instance it is attached to. If the volume is under heavy use, the snapshot may have an inconsistent file system. In fact, by default, the volume service does not take a snapshot of a volume that is attached to an image, though it can be forced to. To take a volume snapshot, either select Create Snapshot from the actions column next to the volume name on the dashboard Volumes page, or run this from the command line:
Block Storage Creation Failures
If a user tries to create a volume and the volume immediately goes into an error state, the best way to troubleshoot is to grep the cinder log files for the volume’s UUID. First try the log files on the cloud controller, and then try the storage node where the volume was attempted to be created:
Similar to Block Storage, the Shared File System is a persistent storage, called share, that can be used in multi-tenant environments. Users create and mount a share as a remote file system on any machine that allows mounting shares, and has network access to share exporter. This share can then be used for storing, sharing, and exchanging files. The default configuration of the Shared File Systems service depends on the back-end driver the admin chooses when starting the Shared File Systems service. For more information about existing back-end drivers, see Share Backends of Shared File Systems service Developer Guide. For example, in case of OpenStack Block Storage based back-end is used, the Shared File Systems service cares about everything, including VMs, networking, keypairs, and security groups. Other configurations require more detailed knowledge of shares functionality to set up and tune specific parameters and modes of shares functioning.
Shares are a remote mountable file system, so users can mount a share to multiple hosts, and have it accessed from multiple hosts by multiple users at a time. With the Shared File Systems service, you can perform a large number of operations with shares:
- Create, update, delete, and force-delete shares
- Change access rules for shares, reset share state
- Specify quotas for existing users or tenants
- Create share networks
- Define new share types
- Perform operations with share snapshots: create, change name, create a share from a snapshot, delete
- Operate with consistency groups
- Use security services
For more information on share management see Share management of chapter “Shared File Systems” in OpenStack Administrator Guide. As to Security services, you should remember that different drivers support different authentication methods, while generic driver does not support Security Services at all (see section Security services of chapter “Shared File Systems” in OpenStack Administrator Guide).
You can create a share in a network, list shares, and show information for, update, and delete a specified share. You can also create snapshots of shares (see Share snapshots of chapter “Shared File Systems” in OpenStack Administrator Guide).
There are default and specific share types that allow you to filter or choose back-ends before you create a share. Functions and behaviour of share type is similar to Block Storage volume type (see Share types of chapter “Shared File Systems” in OpenStack Administrator Guide).
To help users keep and restore their data, Shared File Systems service provides a mechanism to create and operate snapshots (see Share snapshots of chapter “Shared File Systems” in OpenStack Administrator Guide).
A security service stores configuration information for clients for authentication and authorization. Inside Manila a share network can be associated with up to three security types (for detailed information see Security services of chapter “Shared File Systems” in OpenStack Administrator Guide):
- Microsoft Active Directory
Shared File Systems service differs from the principles implemented in Block Storage. Shared File Systems service can work in two modes:
- Without interaction with share networks, in so called “no share servers” mode.
- Interacting with share networks.
Networking service is used by the Shared File Systems service to directly operate with share servers. For switching interaction with Networking service on, create a share specifying a share network. To use “share servers” mode even being out of OpenStack, a network plugin called StandaloneNetworkPlugin is used. In this case, provide network information in the configuration: IP range, network type, and segmentation ID. Also you can add security services to a share network (see section “Networking” of chapter “Shared File Systems” in OpenStack Administrator Guide).
The main idea of consistency groups is to enable you to create snapshots at the exact same point in time from multiple file system shares. Those snapshots can be then used for restoring all shares that were associated with the consistency group (see section “Consistency groups” of chapter “Shared File Systems” in OpenStack Administrator Guide).
Shared File System storage allows administrators to set limits and quotas for specific tenants and users. Limits are the resource limitations that are allowed for each tenant or user. Limits consist of:
- Rate limits
- Absolute limits
Rate limits control the frequency at which users can issue specific API requests. Rate limits are configured by administrators in a config file. Also, administrator can specify quotas also known as max values of absolute limits per tenant. Whereas users can see only the amount of their consumed resources. Administrator can specify rate limits or quotas for the following resources:
- Max amount of space available for all shares
- Max number of shares
- Max number of shared networks
- Max number of share snapshots
- Max total amount of all snapshots
- Type and number of API calls that can be made in a specific time interval
User can see his rate limits and absolute limits by running commands manila rate-limits and manila absolute-limits respectively. For more details on limits and quotas see Quotas and limits of “Share management” section of OpenStack Administrator Guide document.
This section lists several of the most important Use Cases that demonstrate the main functions and abilities of Shared File Systems service:
- Create share
- Operating with a share
- Manage access to shares
- Create snapshots
- Create a share network
- Manage a share network
In this section, we examine the process of creating a simple share. It consists of several steps:
- Check if there is an appropriate share type defined in the Shared File Systems service
- If such a share type does not exist, an Admin should create it using manila type-create command before other users are able to use it
- Using a share network is optional. However if you need one, check if there is an appropriate network defined in Shared File Systems service by using manila share-network-list command. For the information on creating a share network, see Create a Share Network below in this chapter.
- Create a public share using manila create.
- Make sure that the share has been created successfully and is ready to use (check the share status and see the share export location)
Below is the same whole procedure described step by step and in more detail.
By default, there are no share types defined in Shared File Systems service, so you can check if a required one has been already created:
If the share types list is empty or does not contain a type you need, create the required share type using this command:
This command will create a public share with the following parameters:
name = netapp1,
spec_driver_handles_share_servers = False
You can now create a public share with my_share_net network, default share type, NFS shared file systems protocol, and 1 GB size:
To confirm that creation has been successful, see the share in the share list:
Check the share status and see the share export location. After creation, the share status should become
is_public defines the level of visibility for the share: whether other tenants can or cannot see the share. By default, the share is private. Now you can mount the created share like a remote file system and use it for your purposes.
Currently, you have a share and would like to control access to this share for other users. For this, you have to perform a number of steps and operations. Before getting to manage access to the share, pay attention to the following important parameters. To grant or deny access to a share, specify one of these supported share access levels:
rw: read and write (RW) access. This is the default value.
ro:read-only (RO) access.
Additionally, you should also specify one of these supported authentication methods:
ip: authenticates an instance through its IP address. A valid format is XX.XX.XX.XX orXX.XX.XX.XX/XX. For example 0.0.0.0/0.
cert: authenticates an instance through a TLS certificate. Specify the TLS identity as the IDENTKEY. A valid value is any string up to 64 characters long in the common name (CN) of the certificate. The meaning of a string depends on its interpretation.
user: authenticates by a specified user or group name. A valid value is an alphanumeric string that can contain some special characters and is from 4 to 32 characters long.
Allow access to the share with IP access type and 10.254.0.4 IP address:
Mount the Share:
Then check if the share mounted successfully and according to the specified access rules:
There are several other useful operations you would perform when working with shares.
To change the name of a share, or update its description, or level of visibility for other tenants, use this command:
Check the attributes of the updated Share1:
Sometimes a share may appear and then hang in an erroneous or a transitional state. Unprivileged users do not have the appropriate access rights to correct this situation. However, having cloud administrator’s permissions, you can reset the share’s state by using
command to reset share state, where state indicates which state to assign the share to. Options include:
available, error, creating, deleting, error_deleting states.
check the share’s status:
If you do not need a share any more, you can delete it using manila delete share_name_or_ID command like:
Sometimes it appears that a share hangs in one of transitional states (i.e.
creating, deleting, managing, unmanaging, extending, and shrinking). In that case, to delete it, you need manila force-delete share_name_or_ID command and administrative permissions to run it:
The Shared File Systems service provides a mechanism of snapshots to help users to restore their own data. To create a snapshot, use manila snapshot-create command like:
Then, if needed, update the name and description of the created snapshot:
To make sure that the snapshot is available, run:
To control a share network, Shared File Systems service requires interaction with Networking service to manage share servers on its own. If the selected driver runs in a mode that requires such kind of interaction, you need to specify the share network when a share is created. For the information on share creation, see Create Share earlier in this chapter. Initially, check the existing share networks type list by:
If share network list is empty or does not contain a required network, just create, for example, a share network with a private network and subnetwork.
network_type share network attributes are automatically set to the values determined by the network provider.
Then check if the network became created by requesting the networks list once again:
Finally, to create a share that uses this share network, get to Create Share use case described earlier in this chapter.
There is a pair of useful commands that help manipulate share networks. To start, check the network list:
If you configured the back-end with
driver_handles_share_servers = True (with the share servers) and had already some operations in the Shared File Systems service, you can see
manila_service_network in the neutron list of networks. This network was created by the share driver for internal usage.
You also can see detailed information about the share network including
network_type, segmentation_id fields:
You also can add and remove the security services to the share network.
Instances are the running virtual machines within an OpenStack cloud. This section deals with how to work with them and their underlying images, their network properties, and how they are represented in the database.
To launch an instance, you need to select an image, a flavor, and a name. The name needn’t be unique, but your life will be simpler if it is because many tools will use the name in place of the UUID so long as the name is unique. You can start an instance from the dashboard from the Launch Instance button on the Instances page or by selecting the Launch action next to an image or a snapshot on the Images page.
On the command line, do this:
There are a number of optional items that can be specified. You should read the rest of this section before trying to start an instance, but this is the base command that later details are layered upon.
To delete instances from the dashboard, select the Delete Instance action next to the instance on the Instances page.
From the command line, do this:
It is important to note that powering off an instance does not terminate it in the OpenStack sense.
Instance Boot Failures
If an instance fails to start and immediately moves to an error state, there are a few different ways to track down what has gone wrong. Some of these can be done with normal user access, while others require access to your log server or compute nodes.
The simplest reasons for nodes to fail to launch are quota violations or the scheduler being unable to find a suitable compute node on which to run the instance. In these cases, the error is apparent when you run a openstack server show on the faulted instance:
In this case, looking at the
fault message shows
NoValidHost, indicating that the scheduler was unable to match the instance requirements.
If openstack server show does not sufficiently explain the failure, searching for the instance UUID in the
nova-compute.log on the compute node it was scheduled on or the
nova-scheduler.log on your scheduler hosts is a good place to start looking for lower-level problems.
Using openstack server show as an admin user will show the compute node the instance was scheduled on as
hostId. If the instance failed during scheduling, this field is blank.
Using Instance-Specific Data
There are two main types of instance-specific data: metadata and user data.
For Compute, instance metadata is a collection of key-value pairs associated with an instance. Compute reads and writes to these key-value pairs any time during the instance lifetime, from inside and outside the instance, when the end user uses the Compute API to do so. However, you cannot query the instance-associated key-value pairs with the metadata service that is compatible with the Amazon EC2 metadata service.
For an example of instance metadata, users can generate and register SSH keys using the openstack keypair create command:
This creates a key named
mykey, which you can associate with instances. The file
mykey.pem is the private key, which should be saved to a secure location because it allows root access to instances the
mykey key is associated with.
Use this command to register an existing key with OpenStack:
To associate a key with an instance on boot, add
--key-name mykey to your command line. For example:
When booting a server, you can also add arbitrary metadata so that you can more easily identify it among other running instances. Use the
--property option with a key-value pair, where you can make up the string for both the key and the value. For example, you could add a description and also the creator of the server:
When viewing the server information, you can see the metadata included on the metadata line:
Instance user data
user-data key is a special key in the metadata service that holds a file that cloud-aware applications within the guest instance can access. For example, cloudinit is an open source package from Ubuntu, but available in most distributions, that handles early initialization of a cloud instance that makes use of this user data.
This user data can be put in a file on your local system and then passed in at instance creation with the flag
To understand the difference between user data and metadata, realize that user data is created before an instance is started. User data is accessible from within the instance when it is running. User data can be used to store configuration, a script, or anything the tenant wants.
Arbitrary local files can also be placed into the instance file system at creation time by using the
--file <dst-path=src-path> option. You may store up to five files.
For example, let’s say you have a special
authorized_keys file named special_authorized_keysfile that for some reason you want to put on the instance instead of using the regular SSH key injection. In this case, you can use the following command:
Associating Security Groups
Security groups, as discussed earlier, are typically required to allow network traffic to an instance, unless the default security group for a project has been modified to be more permissive.
Adding security groups is typically done on instance boot. When launching from the dashboard, you do this on the Access & Security tab of the Launch Instance dialog. When launching from the command line, append
--security-groups with a comma-separated list of security groups.
It is also possible to add and remove security groups when an instance is running. Currently this is only available through the command-line tools. Here is an example:
Where floating IPs are configured in a deployment, each project will have a limited number of floating IPs controlled by a quota. However, these need to be allocated to the project from the central pool prior to their use—usually by the administrator of the project. To allocate a floating IP to a project, use the Allocate IP To Project button on the Floating IPs tab of the Access & Security page of the dashboard. The command line can also be used:
Once allocated, a floating IP can be assigned to running instances from the dashboard either by selecting Associate from the actions drop-down next to the IP on the Floating IPs tab of the Access & Security page or by making this selection next to the instance you want to associate it with on the Instances page. The inverse action, Dissociate Floating IP, is available from the Floating IPs tab of the Access & Security page and from the Instances page.
To associate or disassociate a floating IP with a server from the command line, use the following commands:
Attaching Block Storage
You can attach block storage to instances from the dashboard on the Volumes page. Click the Manage Attachments action next to the volume you want to attach.
To perform this action from command line, run the following command:
You can also specify block deviceblock device mapping at instance boot time through the nova command-line client with this option set:
The block device mapping format is
- A device name where the volume is attached in the system at
- The ID of the volume to boot from, as shown in the output of openstack volume list
snap, which means that the volume was created from a snapshot, or anything other than
snap(a blank string is valid). In the preceding example, the volume was not created from a snapshot, so we leave this field blank in our following example.
- size (GB)
- The size of the volume in gigabytes. It is safe to leave this blank and have the Compute Service infer the size.
- A boolean to indicate whether the volume should be deleted when the instance is terminated. True can be specified as
1. False can be specified as
The following command will boot a new instance and attach a volume at the same time. The volume of ID 13 will be attached as
/dev/vdc. It is not a snapshot, does not specify a size, and will not be deleted when the instance is terminated:
If you have previously prepared block storage with a bootable file system image, it is even possible to boot from persistent block storage. The following command boots an image from the specified volume. It is similar to the previous command, but the image is omitted and the volume is now attached as
Read more detailed instructions for launching an instance from a bootable volume in the OpenStack End User Guide.
To boot normally from an image and attach block storage, map to a device other than vda. You can find instructions for launching an instance and attaching a volume to the instance and for copying the image to the attached volume in the OpenStack End User Guide.
The OpenStack snapshot mechanism allows you to create new images from running instances. This is very convenient for upgrading base images or for taking a published image and customizing it for local use. To snapshot a running instance to an image using the CLI, do this:
The dashboard interface for snapshots can be confusing because the snapshots and images are displayed in the Images page. However, an instance snapshot is an image. The only difference between an image that you upload directly to the Image Service and an image that you create by snapshot is that an image created by snapshot has additional properties in the glance database. These properties are found in the
image_properties table and include:
||<uuid of instance that was snapshotted>|
||<uuid of original image of instance that was snapshotted>|
Live snapshots is a feature that allows users to snapshot the running virtual machines without pausing them. These snapshots are simply disk-only snapshots. Snapshotting an instance can now be performed with no downtime (assuming QEMU 1.3+ and libvirt 1.0+ are used).
Ensuring Snapshots of Linux Guests Are Consistent
The following section is from Sébastien Han’s OpenStack: Perform Consistent Snapshots blog entry.
A snapshot captures the state of the file system, but not the state of the memory. Therefore, to ensure your snapshot contains the data that you want, before your snapshot you need to ensure that:
- Running programs have written their contents to disk
- The file system does not have any “dirty” buffers: where programs have issued the command to write to disk, but the operating system has not yet done the write
To ensure that important services have written their contents to disk (such as databases), we recommend that you read the documentation for those applications to determine what commands to issue to have them sync their contents to disk. If you are unsure how to do this, the safest approach is to simply stop these running services normally.
To deal with the “dirty” buffer issue, we recommend using the sync command before snapshotting:
sync writes dirty buffers (buffered blocks that have been modified but not written yet to the disk block) to disk.
sync is not enough to ensure that the file system is consistent. We recommend that you use the
fsfreeze tool, which halts new access to the file system, and create a stable image on disk that is suitable for snapshotting. The
fsfreeze tool supports several file systems, including ext3, ext4, and XFS. If your virtual machine instance is running on Ubuntu, install the util-linux package to get
If your operating system doesn’t have a version of
fsfreeze available, you can use
xfs_freeze instead, which is available on Ubuntu in the xfsprogs package. Despite the “xfs” in the name, xfs_freeze also works on ext3 and ext4 if you are using a Linux kernel version 2.6.29 or greater, since it works at the virtual file system (VFS) level starting at 2.6.29. The xfs_freeze version supports the same command-line arguments as
Consider the example where you want to take a snapshot of a persistent block storage volume, detected by the guest operating system as
/dev/vdb and mounted on
/mnt. The fsfreeze command accepts two arguments:
|-f||Freeze the system|
|-u||Thaw (unfreeze) the system|
To freeze the volume in preparation for snapshotting, you would do the following, as root, inside the instance:
You must mount the file system before you run the fsfreeze command.
When the fsfreeze -f command is issued, all ongoing transactions in the file system are allowed to complete, new write system calls are halted, and other calls that modify the file system are halted. Most importantly, all dirty data, metadata, and log information are written to disk.
Once the volume has been frozen, do not attempt to read from or write to the volume, as these operations hang. The operating system stops every I/O operation and any I/O attempts are delayed until the file system has been unfrozen.
Once you have issued the fsfreeze command, it is safe to perform the snapshot. For example, if the volume of your instance was named
mon-volume and you wanted to snapshot it to an image named
mon-snapshot, you could now run the following:
When the snapshot is done, you can thaw the file system with the following command, as root, inside of the instance:
If you want to back up the root file system, you can’t simply run the preceding command because it will freeze the prompt. Instead, run the following one-liner, as root, inside the instance:
After this command it is common practice to call openstack image create from your workstation, and once done press enter in your instance shell to unfreeze it. Obviously you could automate this, but at least it will let you properly synchronize.
Ensuring Snapshots of Windows Guests Are Consistent
Obtaining consistent snapshots of Windows VMs is conceptually similar to obtaining consistent snapshots of Linux VMs, although it requires additional utilities to coordinate with a Windows-only subsystem designed to facilitate consistent backups.
Windows XP and later releases include a Volume Shadow Copy Service (VSS) which provides a framework so that compliant applications can be consistently backed up on a live filesystem. To use this framework, a VSS requestor is run that signals to the VSS service that a consistent backup is needed. The VSS service notifies compliant applications (called VSS writers) to quiesce their data activity. The VSS service then tells the copy provider to create a snapshot. Once the snapshot has been made, the VSS service unfreezes VSS writers and normal I/O activity resumes.
QEMU provides a guest agent that can be run in guests running on KVM hypervisors. This guest agent, on Windows VMs, coordinates with the Windows VSS service to facilitate a workflow which ensures consistent snapshots. This feature requires at least QEMU 1.7. The relevant guest agent commands are:
- Write out “dirty” buffers to disk, similar to the Linux
- Suspend I/O to the disks, similar to the Linux
- Resume I/O to the disks, similar to the Linux
To obtain snapshots of a Windows VM these commands can be scripted in sequence: flush the filesystems, freeze the filesystems, snapshot the filesystems, then unfreeze the filesystems. As with scripting similar workflows against Linux VMs, care must be used when writing such a script to ensure error handling is thorough and filesystems will not be left in a frozen state.
Instances in the Database
While instance information is stored in a number of database tables, the table you most likely need to look at in relation to user instances is the instances table.
The instances table carries most of the information related to both running and deleted instances. It has a bewildering array of fields; for an exhaustive list, look at the database. These are the most useful fields for operators looking to form queries:
deletedfield is set to
1if the instance has been deleted and
NULLif it has not been deleted. This field is important for excluding deleted instances from your queries.
uuidfield is the UUID of the instance and is used throughout other tables in the database as a foreign key. This ID is also reported in logs, the dashboard, and command-line tools to uniquely identify an instance.
- A collection of foreign keys are available to find relations to the instance. The most useful of these —
project_idare the UUIDs of the user who launched the instance and the project it was launched in.
hostfield tells which compute node is hosting the instance.
hostnamefield holds the name of the instance when it is launched. The display-name is initially the same as hostname but can be reset using the nova rename command.
A number of time-related fields are useful for tracking when state changes happened on an instance: