OpsGuide/User-Facing Operations

This guide is for OpenStack operators and does not seek to be an exhaustive reference for users, but as an operator, you should have a basic understanding of how to use the cloud facilities. This chapter looks at OpenStack from a basic user perspective, which helps you understand your users’ needs and determine, when you get a trouble ticket, whether it is a user issue or a service issue. The main concepts covered are images, flavors, security groups, block storage, shared file system storage, and instances.

Images
OpenStack images can often be thought of as “virtual machine templates.” Images can also be standard installation media such as ISO images. Essentially, they contain bootable file systems that are used to launch instances.

Adding Images
Several pre-made images exist and can easily be imported into the Image service. A common image to add is the CirrOS image, which is very small and used for testing purposes. To add this image, simply do:

The openstack image create command provides a large set of options for working with your image. For example, the  option is useful for images that require root disks of a certain size (for example, large Windows images). To view these options, run:

Run the following command to view the properties of existing images:



Adding Signed Images
To provide a chain of trust from an end user to the Image service, and the Image service to Compute, an end user can import signed images that can be initially verified in the Image service, and later verified in the Compute service. Appropriate Image service properties need to be set to enable this signature feature.

The following are steps needed to create the signature used for the signed images:

  Retrieve image for upload   Use private key to create a signature of the image

Generate signature of image and convert it to a base64 representation:

Create context
 * 

Encode certificate in DER format 

Upload Certificate in DER format to Castellan 

Upload Image to Image service, with Signature Metadata



 Verify the Keystone URL

  Signature verification will occur when Compute boots the signed image

Sharing Images Between Projects
In a multi-tenant cloud environment, users sometimes want to share their personal images or snapshots with other projects. This can be done on the command line with the  tool by the owner of the image.

To share an image or snapshot with another project, do the following:



 Obtain the UUID of the image: </li>

 Obtain the UUID of the project with which you want to share your image, let’s call it target project. Unfortunately, non-admin users are unable to use the openstack command to do this. The easiest solution is to obtain the UUID either from an administrator of the cloud or from a user located in the target project. </li>

 Once you have both pieces of information, run the openstack image add project command:

For example: </li>  You now need to act in the target project scope.

To accept the sharing, you need to update the member status: For example: Project  will now have access to image. Tip You can explicitly ask for pending member status to view shared images not yet accepted: </li></ol>

Deleting Images
To delete an image, just execute:

Other CLI Options
A full set of options can be found using:

or the Command-Line Interface Reference.

The Image service and the Database
The only thing the Image service does not store in a database is the image itself. The Image service database has two main tables:



Working directly with the database and SQL queries can provide you with custom lists and reports of images. Technically, you can update properties about images through the database, although this is not generally recommended.

Example Image service Database Queries
One interesting example is modifying the table of images and the owner of that image. This can be easily done if you simply display the unique ID of the owner. This example goes one step further and displays the readable name of the owner:

Another example is displaying all properties for a certain image:

Flavors
Virtual hardware templates are called “flavors” in OpenStack, defining sizes for RAM, disk, number of cores, and so on. The default install provides five flavors.

These are configurable by admin users (the rights may also be delegated to other users by redefining the access controls for  in   on the   server). To get the list of available flavors on your system, run:

The openstack flavor create command allows authorized users to create new flavors. Additional flavor manipulation commands can be shown with the following command:

Flavors define a number of parameters, resulting in the user having a choice of what type of virtual machine to run—just like they would have if they were purchasing a physical server. Table. Flavor parameters lists the elements that can be set. Note in particular, which can be used to define free-form characteristics, giving a lot of flexibility beyond just the size of RAM, CPU, and Disk.

Private Flavors
A user might need a custom flavor that is uniquely tuned for a project she is working on. For example, the user might require 128 GB of memory. If you create a new flavor as described above, the user would have access to the custom flavor, but so would all other tenants in your cloud. Sometimes this sharing isn’t desirable. In this scenario, allowing all users to have access to a flavor with 128 GB of memory might cause your cloud to reach full capacity very quickly. To prevent this, you can restrict access to the custom flavor using the nova flavor-access-add command:

To view a flavor’s access list, do the following:

It’s also helpful to allocate a specific numeric range for custom and private flavors. On UNIX-based systems, nonsystem accounts usually have a UID starting at 500. A similar approach can be taken with custom flavors. This helps you easily identify which flavors are custom, private, and public for the entire cloud.

How Do I Modify an Existing Flavor?
The OpenStack dashboard simulates the ability to modify a flavor by deleting an existing flavor and creating a new one with the same name.

Security Groups
A common new-user issue with OpenStack is failing to set an appropriate security group when launching an instance. As a result, the user is unable to contact the instance on the network.

Security groups are sets of IP filter rules that are applied to an instance’s networking. They are project specific, and project members can edit the default rules for their group and add new rules sets. All projects have a “default” security group, which is applied to instances that have no other security group defined. Unless changed, this security group denies all incoming traffic.

End-User Configuration of Security Groups
Security groups for the current project can be found on the OpenStack dashboard under Access &amp; Security. To see details of an existing group, select the Edit Security Group action for that security group. Obviously, modifying existing groups can be done from this edit interface. There is a Create Security Group button on the main Access &amp; Security page for creating new groups. We discuss the terms used in these fields when we explain the command-line equivalents.

Setting with openstack command

If your environment is using Neutron, you can configure security groups settings using the openstack command. Get a list of security groups for the project you are acting in, by using following command:

To view the details of a security group:

These rules are all “allow” type rules, as the default is deny. This example shows the full port range for all protocols allowed from all IPs. This section describes the most common security group rule parameters:


 * direction
 * The direction in which the security group rule is applied. Valid values are  or.


 * remote_ip_prefix
 * This attribute value matches the specified IP prefix as the source IP address of the IP packet.


 * protocol
 * The protocol that is matched by the security group rule. Valid values are,  ,  ,  , and.


 * port_range_min
 * The minimum port number in the range that is matched by the security group rule. If the protocol is TCP or UDP, this value must be less than or equal to the  attribute value. If the protocol is ICMP or ICMPv6, this value must be an ICMP or ICMPv6 type, respectively.


 * port_range_max
 * The maximum port number in the range that is matched by the security group rule. The  attribute constrains the   attribute. If the protocol is ICMP or ICMPv6, this value must be an ICMP or ICMPv6 type, respectively.


 * ethertype
 * Must be  or , and addresses represented in CIDR must match the ingress or egress rules.

When adding a new security group, you should pick a descriptive but brief name. This name shows up in brief descriptions of the instances that use it where the longer description field often does not. Seeing that an instance is using security group  is much easier to understand than   or.

This example creates a security group that allows web traffic anywhere on the Internet. We’ll call this group, which is clear and reasonably concise, encapsulating what is allowed and from where. From the command line, do:

Immediately after create, the security group has only an allow egress rule. To make it do what we want, we need to add some rules:

Despite only outputting the newly added rule, this operation is additive:

The inverse operation is called openstack security group rule delete, specifying security-group-rule ID. Whole security groups can be removed with openstack security group delete.

To create security group rules for a cluster of instances, use RemoteGroups.

RemoteGroups are a dynamic way of defining the CIDR of allowed sources. The user specifies a RemoteGroup (security group name) and then all the users’ other instances using the specified RemoteGroup are selected dynamically. This dynamic selection alleviates the need for individual rules to allow each new member of the cluster.

The code is similar to the above example of openstack security group rule create. To use RemoteGroup, specify  instead of. For example:

The “cluster” rule allows SSH access from any other instance that uses the  group.

Block Storage
OpenStack volumes are persistent block-storage devices that may be attached and detached from instances, but they can be attached to only one instance at a time. Similar to an external hard drive, they do not provide shared storage in the way a network file system or object store does. It is left to the operating system in the instance to put a file system on the block device and mount it, or not.

As with other removable disk technology, it is important that the operating system is not trying to make use of the disk before removing it. On Linux instances, this typically involves unmounting any file systems mounted from the volume. The OpenStack volume service cannot tell whether it is safe to remove volumes from an instance, so it does what it is told. If a user tells the volume service to detach a volume from an instance while it is being written to, you can expect some level of file system corruption as well as faults from whatever process within the instance was using the device.

There is nothing OpenStack-specific in being aware of the steps needed to access block devices from within the instance operating system, potentially formatting them for first use and being cautious when removing them. What is specific is how to create new volumes and attach and detach them from instances. These operations can all be done from the Volumes page of the dashboard or by using the  command-line client.

To add new volumes, you need only a volume size in gigabytes. Either put these into the Create Volume web form or use the command line:

This creates a 10 GB volume. To list existing volumes and the instances they are connected to, if any:

OpenStack Block Storage also allows creating snapshots of volumes. Remember that this is a block-level snapshot that is crash consistent, so it is best if the volume is not connected to an instance when the snapshot is taken and second best if the volume is not in use on the instance it is attached to. If the volume is under heavy use, the snapshot may have an inconsistent file system. In fact, by default, the volume service does not take a snapshot of a volume that is attached to an image, though it can be forced to. To take a volume snapshot, either select Create Snapshot from the actions column next to the volume name on the dashboard Volumes page, or run this from the command line:

Block Storage Creation Failures
If a user tries to create a volume and the volume immediately goes into an error state, the best way to troubleshoot is to grep the cinder log files for the volume’s UUID. First try the log files on the cloud controller, and then try the storage node where the volume was attempted to be created:

Shared File Systems Service
Similar to Block Storage, the Shared File System is a persistent storage, called share, that can be used in multi-tenant environments. Users create and mount a share as a remote file system on any machine that allows mounting shares, and has network access to share exporter. This share can then be used for storing, sharing, and exchanging files. The default configuration of the Shared File Systems service depends on the back-end driver the admin chooses when starting the Shared File Systems service. For more information about existing back-end drivers, see Share Backends of Shared File Systems service Developer Guide. For example, in case of OpenStack Block Storage based back-end is used, the Shared File Systems service cares about everything, including VMs, networking, keypairs, and security groups. Other configurations require more detailed knowledge of shares functionality to set up and tune specific parameters and modes of shares functioning.

Shares are a remote mountable file system, so users can mount a share to multiple hosts, and have it accessed from multiple hosts by multiple users at a time. With the Shared File Systems service, you can perform a large number of operations with shares:


 * Create, update, delete, and force-delete shares
 * Change access rules for shares, reset share state
 * Specify quotas for existing users or tenants
 * Create share networks
 * Define new share types
 * Perform operations with share snapshots: create, change name, create a share from a snapshot, delete
 * Operate with consistency groups
 * Use security services

For more information on share management see Share management of chapter “Shared File Systems” in OpenStack Administrator Guide. As to Security services, you should remember that different drivers support different authentication methods, while generic driver does not support Security Services at all (see section Security services of chapter “Shared File Systems” in OpenStack Administrator Guide).

You can create a share in a network, list shares, and show information for, update, and delete a specified share. You can also create snapshots of shares (see Share snapshots of chapter “Shared File Systems” in OpenStack Administrator Guide).

There are default and specific share types that allow you to filter or choose back-ends before you create a share. Functions and behaviour of share type is similar to Block Storage volume type (see Share types of chapter “Shared File Systems” in OpenStack Administrator Guide).

To help users keep and restore their data, Shared File Systems service provides a mechanism to create and operate snapshots (see Share snapshots of chapter “Shared File Systems” in OpenStack Administrator Guide).

A security service stores configuration information for clients for authentication and authorization. Inside Manila a share network can be associated with up to three security types (for detailed information see Security services of chapter “Shared File Systems” in OpenStack Administrator Guide):


 * LDAP
 * Kerberos
 * Microsoft Active Directory

Shared File Systems service differs from the principles implemented in Block Storage. Shared File Systems service can work in two modes:


 * Without interaction with share networks, in so called “no share servers” mode.
 * Interacting with share networks.

Networking service is used by the Shared File Systems service to directly operate with share servers. For switching interaction with Networking service on, create a share specifying a share network. To use “share servers” mode even being out of OpenStack, a network plugin called StandaloneNetworkPlugin is used. In this case, provide network information in the configuration: IP range, network type, and segmentation ID. Also you can add security services to a share network (see section “Networking” of chapter “Shared File Systems” in OpenStack Administrator Guide).

The main idea of consistency groups is to enable you to create snapshots at the exact same point in time from multiple file system shares. Those snapshots can be then used for restoring all shares that were associated with the consistency group (see section “Consistency groups” of chapter “Shared File Systems” in OpenStack Administrator Guide).

Shared File System storage allows administrators to set limits and quotas for specific tenants and users. Limits are the resource limitations that are allowed for each tenant or user. Limits consist of:


 * Rate limits
 * Absolute limits

Rate limits control the frequency at which users can issue specific API requests. Rate limits are configured by administrators in a config file. Also, administrator can specify quotas also known as max values of absolute limits per tenant. Whereas users can see only the amount of their consumed resources. Administrator can specify rate limits or quotas for the following resources:


 * Max amount of space available for all shares
 * Max number of shares
 * Max number of shared networks
 * Max number of share snapshots
 * Max total amount of all snapshots
 * Type and number of API calls that can be made in a specific time interval

User can see his rate limits and absolute limits by running commands manila rate-limits and manila absolute-limits respectively. For more details on limits and quotas see Quotas and limits of “Share management” section of OpenStack Administrator Guide document.

This section lists several of the most important Use Cases that demonstrate the main functions and abilities of Shared File Systems service:


 * Create share
 * Operating with a share
 * Manage access to shares
 * Create snapshots
 * Create a share network
 * Manage a share network

Create Share
In this section, we examine the process of creating a simple share. It consists of several steps:


 * Check if there is an appropriate share type defined in the Shared File Systems service
 * If such a share type does not exist, an Admin should create it using manila type-create command before other users are able to use it
 * Using a share network is optional. However if you need one, check if there is an appropriate network defined in Shared File Systems service by using manila share-network-list command. For the information on creating a share network, see Create a Share Network below in this chapter.
 * Create a public share using manila create.
 * Make sure that the share has been created successfully and is ready to use (check the share status and see the share export location)

Below is the same whole procedure described step by step and in more detail.

By default, there are no share types defined in Shared File Systems service, so you can check if a required one has been already created:

If the share types list is empty or does not contain a type you need, create the required share type using this command:

This command will create a public share with the following parameters: ,

You can now create a public share with my_share_net network, default share type, NFS shared file systems protocol, and 1 GB size:

To confirm that creation has been successful, see the share in the share list:

Check the share status and see the share export location. After creation, the share status should become :

The value  defines the level of visibility for the share: whether other tenants can or cannot see the share. By default, the share is private. Now you can mount the created share like a remote file system and use it for your purposes.

Manage Access To Shares
Currently, you have a share and would like to control access to this share for other users. For this, you have to perform a number of steps and operations. Before getting to manage access to the share, pay attention to the following important parameters. To grant or deny access to a share, specify one of these supported share access levels:


 * : read and write (RW) access. This is the default value.
 * read-only (RO) access.

Additionally, you should also specify one of these supported authentication methods:


 * : authenticates an instance through its IP address. A valid format is XX.XX.XX.XX orXX.XX.XX.XX/XX. For example 0.0.0.0/0.
 * : authenticates an instance through a TLS certificate. Specify the TLS identity as the IDENTKEY. A valid value is any string up to 64 characters long in the common name (CN) of the certificate. The meaning of a string depends on its interpretation.
 * : authenticates by a specified user or group name. A valid value is an alphanumeric string that can contain some special characters and is from 4 to 32 characters long.

Allow access to the share with IP access type and 10.254.0.4 IP address:

Mount the Share:

Then check if the share mounted successfully and according to the specified access rules:

Manage Shares
There are several other useful operations you would perform when working with shares.

Update Share
To change the name of a share, or update its description, or level of visibility for other tenants, use this command:

Check the attributes of the updated Share1:

Reset Share State
Sometimes a share may appear and then hang in an erroneous or a transitional state. Unprivileged users do not have the appropriate access rights to correct this situation. However, having cloud administrator’s permissions, you can reset the share’s state by using

command to reset share state, where state indicates which state to assign the share to. Options include:  states.

After running

check the share’s status:

Delete Share
If you do not need a share any more, you can delete it using manila delete share_name_or_ID command like:

Sometimes it appears that a share hangs in one of transitional states (i.e. ). In that case, to delete it, you need manila force-delete share_name_or_ID command and administrative permissions to run it:

Create Snapshots
The Shared File Systems service provides a mechanism of snapshots to help users to restore their own data. To create a snapshot, use manila snapshot-create command like:

Then, if needed, update the name and description of the created snapshot:

To make sure that the snapshot is available, run:

Create a Share Network
To control a share network, Shared File Systems service requires interaction with Networking service to manage share servers on its own. If the selected driver runs in a mode that requires such kind of interaction, you need to specify the share network when a share is created. For the information on share creation, see Create Share earlier in this chapter. Initially, check the existing share networks type list by:

If share network list is empty or does not contain a required network, just create, for example, a share network with a private network and subnetwork.

The,  ,  , and   share network attributes are automatically set to the values determined by the network provider.

Then check if the network became created by requesting the networks list once again:

Finally, to create a share that uses this share network, get to Create Share use case described earlier in this chapter.

Manage a Share Network
There is a pair of useful commands that help manipulate share networks. To start, check the network list:

If you configured the back-end with  (with the share servers) and had already some operations in the Shared File Systems service, you can see   in the neutron list of networks. This network was created by the share driver for internal usage.

You also can see detailed information about the share network including  fields:

You also can add and remove the security services to the share network.

Instances
Instances are the running virtual machines within an OpenStack cloud. This section deals with how to work with them and their underlying images, their network properties, and how they are represented in the database.

Starting Instances
To launch an instance, you need to select an image, a flavor, and a name. The name needn’t be unique, but your life will be simpler if it is because many tools will use the name in place of the UUID so long as the name is unique. You can start an instance from the dashboard from the Launch Instance button on the Instances page or by selecting the Launch action next to an image or a snapshot on the Images page.

On the command line, do this:

There are a number of optional items that can be specified. You should read the rest of this section before trying to start an instance, but this is the base command that later details are layered upon.

To delete instances from the dashboard, select the Delete Instance action next to the instance on the Instances page.

From the command line, do this:

It is important to note that powering off an instance does not terminate it in the OpenStack sense.

Instance Boot Failures
If an instance fails to start and immediately moves to an error state, there are a few different ways to track down what has gone wrong. Some of these can be done with normal user access, while others require access to your log server or compute nodes.

The simplest reasons for nodes to fail to launch are quota violations or the scheduler being unable to find a suitable compute node on which to run the instance. In these cases, the error is apparent when you run a openstack server show on the faulted instance:

In this case, looking at the  message shows , indicating that the scheduler was unable to match the instance requirements.

If openstack server show does not sufficiently explain the failure, searching for the instance UUID in the  on the compute node it was scheduled on or the   on your scheduler hosts is a good place to start looking for lower-level problems.

Using openstack server show as an admin user will show the compute node the instance was scheduled on as. If the instance failed during scheduling, this field is blank.

Using Instance-Specific Data
There are two main types of instance-specific data: metadata and user data.

Instance metadata
For Compute, instance metadata is a collection of key-value pairs associated with an instance. Compute reads and writes to these key-value pairs any time during the instance lifetime, from inside and outside the instance, when the end user uses the Compute API to do so. However, you cannot query the instance-associated key-value pairs with the metadata service that is compatible with the Amazon EC2 metadata service.

For an example of instance metadata, users can generate and register SSH keys using the openstack keypair create command:

This creates a key named, which you can associate with instances. The file  is the private key, which should be saved to a secure location because it allows root access to instances the   key is associated with.

Use this command to register an existing key with OpenStack:

To associate a key with an instance on boot, add  to your command line. For example:

When booting a server, you can also add arbitrary metadata so that you can more easily identify it among other running instances. Use the  option with a key-value pair, where you can make up the string for both the key and the value. For example, you could add a description and also the creator of the server:

When viewing the server information, you can see the metadata included on the metadata line:

Instance user data
The  key is a special key in the metadata service that holds a file that cloud-aware applications within the guest instance can access. For example, cloudinit is an open source package from Ubuntu, but available in most distributions, that handles early initialization of a cloud instance that makes use of this user data.

This user data can be put in a file on your local system and then passed in at instance creation with the flag.

For example

To understand the difference between user data and metadata, realize that user data is created before an instance is started. User data is accessible from within the instance when it is running. User data can be used to store configuration, a script, or anything the tenant wants.

File injection
Arbitrary local files can also be placed into the instance file system at creation time by using the  option. You may store up to five files.

For example, let’s say you have a special  file named special_authorized_keysfile that for some reason you want to put on the instance instead of using the regular SSH key injection. In this case, you can use the following command:

Associating Security Groups
Security groups, as discussed earlier, are typically required to allow network traffic to an instance, unless the default security group for a project has been modified to be more permissive.

Adding security groups is typically done on instance boot. When launching from the dashboard, you do this on the Access &amp; Security tab of the Launch Instance dialog. When launching from the command line, append  with a comma-separated list of security groups.

It is also possible to add and remove security groups when an instance is running. Currently this is only available through the command-line tools. Here is an example:

Floating IPs
Where floating IPs are configured in a deployment, each project will have a limited number of floating IPs controlled by a quota. However, these need to be allocated to the project from the central pool prior to their use—usually by the administrator of the project. To allocate a floating IP to a project, use the Allocate IP To Project button on the Floating IPs tab of the Access &amp; Security page of the dashboard. The command line can also be used:

Once allocated, a floating IP can be assigned to running instances from the dashboard either by selecting Associate from the actions drop-down next to the IP on the Floating IPs tab of the Access &amp; Security page or by making this selection next to the instance you want to associate it with on the Instances page. The inverse action, Dissociate Floating IP, is available from the Floating IPs tab of the Access &amp; Security page and from the Instances page.

To associate or disassociate a floating IP with a server from the command line, use the following commands:

Attaching Block Storage
You can attach block storage to instances from the dashboard on the Volumes page. Click the Manage Attachments action next to the volume you want to attach.

To perform this action from command line, run the following command:

You can also specify block deviceblock device mapping at instance boot time through the nova command-line client with this option set:

The block device mapping format is, where:


 * dev-name
 * A device name where the volume is attached in the system at


 * id
 * The ID of the volume to boot from, as shown in the output of openstack volume list


 * type
 * Either, which means that the volume was created from a snapshot, or anything other than   (a blank string is valid). In the preceding example, the volume was not created from a snapshot, so we leave this field blank in our following example.


 * size (GB)
 * The size of the volume in gigabytes. It is safe to leave this blank and have the Compute Service infer the size.


 * delete-on-terminate
 * A boolean to indicate whether the volume should be deleted when the instance is terminated. True can be specified as  or  . False can be specified as   or.

The following command will boot a new instance and attach a volume at the same time. The volume of ID 13 will be attached as. It is not a snapshot, does not specify a size, and will not be deleted when the instance is terminated:

If you have previously prepared block storage with a bootable file system image, it is even possible to boot from persistent block storage. The following command boots an image from the specified volume. It is similar to the previous command, but the image is omitted and the volume is now attached as :

Read more detailed instructions for launching an instance from a bootable volume in the OpenStack End User Guide.

To boot normally from an image and attach block storage, map to a device other than vda. You can find instructions for launching an instance and attaching a volume to the instance and for copying the image to the attached volume in the OpenStack End User Guide.

Taking Snapshots
The OpenStack snapshot mechanism allows you to create new images from running instances. This is very convenient for upgrading base images or for taking a published image and customizing it for local use. To snapshot a running instance to an image using the CLI, do this:

The dashboard interface for snapshots can be confusing because the snapshots and images are displayed in the Images page. However, an instance snapshot is an image. The only difference between an image that you upload directly to the Image Service and an image that you create by snapshot is that an image created by snapshot has additional properties in the glance database. These properties are found in the  table and include:

Live Snapshots
Live snapshots is a feature that allows users to snapshot the running virtual machines without pausing them. These snapshots are simply disk-only snapshots. Snapshotting an instance can now be performed with no downtime (assuming QEMU 1.3+ and libvirt 1.0+ are used).

Ensuring Snapshots of Linux Guests Are Consistent

The following section is from Sébastien Han’s OpenStack: Perform Consistent Snapshots blog entry.

A snapshot captures the state of the file system, but not the state of the memory. Therefore, to ensure your snapshot contains the data that you want, before your snapshot you need to ensure that:


 * Running programs have written their contents to disk
 * The file system does not have any “dirty” buffers: where programs have issued the command to write to disk, but the operating system has not yet done the write

To ensure that important services have written their contents to disk (such as databases), we recommend that you read the documentation for those applications to determine what commands to issue to have them sync their contents to disk. If you are unsure how to do this, the safest approach is to simply stop these running services normally.

To deal with the “dirty” buffer issue, we recommend using the sync command before snapshotting:

Running  writes dirty buffers (buffered blocks that have been modified but not written yet to the disk block) to disk.

Just running  is not enough to ensure that the file system is consistent. We recommend that you use the  tool, which halts new access to the file system, and create a stable image on disk that is suitable for snapshotting. The  tool supports several file systems, including ext3, ext4, and XFS. If your virtual machine instance is running on Ubuntu, install the util-linux package to get :

If your operating system doesn’t have a version of  available, you can use   instead, which is available on Ubuntu in the xfsprogs package. Despite the “xfs” in the name, xfs_freeze also works on ext3 and ext4 if you are using a Linux kernel version 2.6.29 or greater, since it works at the virtual file system (VFS) level starting at 2.6.29. The xfs_freeze version supports the same command-line arguments as.

Consider the example where you want to take a snapshot of a persistent block storage volume, detected by the guest operating system as  and mounted on. The fsfreeze command accepts two arguments:

To freeze the volume in preparation for snapshotting, you would do the following, as root, inside the instance:

You must mount the file system before you run the fsfreeze command.

When the fsfreeze -f command is issued, all ongoing transactions in the file system are allowed to complete, new write system calls are halted, and other calls that modify the file system are halted. Most importantly, all dirty data, metadata, and log information are written to disk.

Once the volume has been frozen, do not attempt to read from or write to the volume, as these operations hang. The operating system stops every I/O operation and any I/O attempts are delayed until the file system has been unfrozen.

Once you have issued the fsfreeze command, it is safe to perform the snapshot. For example, if the volume of your instance was named  and you wanted to snapshot it to an image named , you could now run the following:

When the snapshot is done, you can thaw the file system with the following command, as root, inside of the instance:

If you want to back up the root file system, you can’t simply run the preceding command because it will freeze the prompt. Instead, run the following one-liner, as root, inside the instance:

After this command it is common practice to call openstack image create from your workstation, and once done press enter in your instance shell to unfreeze it. Obviously you could automate this, but at least it will let you properly synchronize.

Ensuring Snapshots of Windows Guests Are Consistent

Obtaining consistent snapshots of Windows VMs is conceptually similar to obtaining consistent snapshots of Linux VMs, although it requires additional utilities to coordinate with a Windows-only subsystem designed to facilitate consistent backups.

Windows XP and later releases include a Volume Shadow Copy Service (VSS) which provides a framework so that compliant applications can be consistently backed up on a live filesystem. To use this framework, a VSS requestor is run that signals to the VSS service that a consistent backup is needed. The VSS service notifies compliant applications (called VSS writers) to quiesce their data activity. The VSS service then tells the copy provider to create a snapshot. Once the snapshot has been made, the VSS service unfreezes VSS writers and normal I/O activity resumes.

QEMU provides a guest agent that can be run in guests running on KVM hypervisors. This guest agent, on Windows VMs, coordinates with the Windows VSS service to facilitate a workflow which ensures consistent snapshots. This feature requires at least QEMU 1.7. The relevant guest agent commands are:


 * guest-file-flush
 * Write out “dirty” buffers to disk, similar to the Linux  operation.


 * guest-fsfreeze
 * Suspend I/O to the disks, similar to the Linux  operation.


 * guest-fsfreeze-thaw
 * Resume I/O to the disks, similar to the Linux  operation.

To obtain snapshots of a Windows VM these commands can be scripted in sequence: flush the filesystems, freeze the filesystems, snapshot the filesystems, then unfreeze the filesystems. As with scripting similar workflows against Linux VMs, care must be used when writing such a script to ensure error handling is thorough and filesystems will not be left in a frozen state.

Instances in the Database
While instance information is stored in a number of database tables, the table you most likely need to look at in relation to user instances is the instances table.

The instances table carries most of the information related to both running and deleted instances. It has a bewildering array of fields; for an exhaustive list, look at the database. These are the most useful fields for operators looking to form queries:


 * The  field is set to   if the instance has been deleted and   if it has not been deleted. This field is important for excluding deleted instances from your queries.
 * The  field is the UUID of the instance and is used throughout other tables in the database as a foreign key. This ID is also reported in logs, the dashboard, and command-line tools to uniquely identify an instance.
 * A collection of foreign keys are available to find relations to the instance. The most useful of these —  and   are the UUIDs of the user who launched the instance and the project it was launched in.
 * The  field tells which compute node is hosting the instance.
 * The  field holds the name of the instance when it is launched. The display-name is initially the same as hostname but can be reset using the nova rename command.

A number of time-related fields are useful for tracking when state changes happened on an instance:



Good Luck
This section was intended as a brief introduction to some of the most useful of many OpenStack commands. For an exhaustive list, please refer to the OpenStack Administrator Guide. We hope your users remain happy and recognize your hard work! (For more hard work, turn the page to the next chapter, where we discuss the system-facing operations: maintenance, failures and debugging.)