OSSN/OSSN-0007

Summary
When using the KVM hypervisor with libvirt on OpenStack Compute nodes, live migration of instances from one Compute server to another requires that the libvirt daemon is configured for remote network connectivity. The libvirt daemon configuration recommended in the OpenStack Configuration Reference manual configures libvirtd to listen for incoming TCP connections on all network interfaces without requiring any authentication or using any encryption. This insecure configuration allows for anyone with network access to the libvirt daemon TCP port on OpenStack Compute nodes to control the hypervisor through the libvirt API.

Affected Services / Software
Nova, Compute, KVM, libvirt, Grizzly, Havana, Icehouse

Discussion
The default configuration of the libvirt daemon is to not allow remote access. Live migration of running instances between OpenStack Compute nodes requires libvirt daemon remote access between OpenStack Compute nodes.

The libvirt daemon should not be configured to allow unauthenticated remote access. The libvirt daemon has a choice of 4 secure options for remote access over TCP. These options are:


 * SSH tunnel to libvirtd's UNIX socket
 * libvirtd TCP socket, with GSSAPI/Kerberos for auth+data encryption
 * libvirtd TCP socket, with TLS for encryption and x.509 client certificates for authentication
 * libvirtd TCP socket, with TLS for encryption and Kerberos for authentication

It is not necessary for the libvirt daemon to listen for remote TCP connections on all interfaces. Remote network connectivity to the libvirt daemon should be restricted as much as possible. Remote access is only needed between the OpenStack Compute nodes, so the libvirt daemon only needs to listen for remote TCP connections on the interface that is used for this communication. A firewall can be configured to lock down access to the TCP port that the libvirt daemon listens on, but this does not sufficiently protect access to the libvirt API. Other processes on a remote OpenStack Compute node might have network access, but should not be authorized to remotely control the hypervisor on another OpenStack Compute node.

Recommended Actions
If you are using the KVM hypervisor with libvirt on OpenStack Compute nodes, you should review your libvirt daemon configuration to ensure that it is not allowing unauthenticated remote access.

Remote access to the libvirt daemon via TCP is configured by the "listen_tls", "listen_tcp", and "auth_tcp" configuration directives. By default, these directives are all commented out. This results in remote access via TCP being disabled.

If you do not need remote libvirt daemon access, you should ensure that the following configuration directives are set as follows in the /etc/libvirt/libvirtd.conf configuration file. Commenting out these directives will have the same effect, as these values match the internal defaults:

listen_tls = 1 listen_tcp = 0 auth_tcp = "sasl"

If you need to allow remote access to the libvirt daemon between OpenStack Compute nodes for live migration, you should ensure that authentication is required. Additionally, you should consider enabling TLS to allow remote connections to be encrypted.

The following libvirt daemon configuration directives will allow for unencrypted remote connections that use SASL for authentication:

listen_tls = 0 listen_tcp = 1 auth_tcp = "sasl"

If you want to require TLS encrypted remote connections, you will have to obtain X.509 certificates and configure the libvirt daemon to use them to use TLS. Details on this configuration are in the libvirt daemon documentation. Once the certificates are configured, you should set the following libvirt daemon configuration directives:

listen_tls = 1 listen_tcp = 0 auth_tls = "none"

When using TLS, setting the "auth_tls" configuration directive to "none" uses X.509 client certificates for authentication. You can additionally require SASL authentication by setting the following libvirt daemon configuration directives:

listen_tls = 1 listen_tcp = 0 auth_tls = "sasl"

When using TLS, it is also necessary to configure the OpenStack Compute nodes to use a non-default URI for live migration. This is done by setting the following configuration directive in /etc/nova/nova.conf:

live_migration_uri=qemu+tls://%s/system

For more details on libvirt daemon remote URI formats, please see the following libvirt daemon documentation:


 * http://libvirt.org/remote.html#Remote_URI_reference

For details on configuring SASL authentication and X.509 certificates for the libvirt daemon, please consult the libvirt daemon documentation at the following locations:


 * http://libvirt.org/remote.html
 * http://libvirt.org/auth.html

When configuring the libvirt daemon for authentication, it is also important to configure authorization to restrict remote access to your OpenStack Compute nodes. For example, if you don't configure authorization, any X.509 client certificate that is signed by your issuing CA will be allowed access. When using SASL/GSSAPI for Kerberos authentication, any client with a valid TGT will be granted access. Lack of authorization can allow unintended remote access. The libvirt daemon documentation should be consulted for details on configuring authorization.

In addition to requiring authentication for remote access to the libvirt daemon on your OpenStack Compute nodes, it is also recommended to restrict network access such that connectivity is only allowed between the Compute nodes.

The first thing that should be done is to restrict the network interfaces that the libvirt daemon listens on for remote connections. By default, the libvirt daemon listens on all interfaces when remote access is enabled. This can be restricted by setting the following configuration directive in /etc/libvirt/libvirtd.conf:

listen_addr = 

Migration in the libvirt daemon also uses a range of ephemeral ports by default. The connections to these ephemeral ports are not authenticated or encrypted. It is possible to tunnel the migration traffic over the regular libvirt daemon remote access port, which will use the authentication and encryption settings that you have defined for that port. It is recommended that you do this for the additional security that it provides. To enable tunnelling of the migration traffic, you must tell your OpenStack Compute nodes to set the VIR_MIGRATE_TUNNELLED flag for live migration. This is done by setting the following directive in /etc/nova/nova.conf:

live_migration_flag=VIR_MIGRATE_UNDEFINE_SOURCE, VIR_MIGRATE_PEER2PEER, VIR_MIGRATE_TUNNELLED

The tunnelling of migration traffic described above does not apply to live block migration. Live block migration is currently only possible over ephemeral ports.

If you choose to use the ephemeral migration ports, there are a few things that you should be aware of. Unfortunately, there is no way to restrict the network interfaces that these ephemeral ports will listen on in libvirt versions prior to 1.1.4. If you are running version 1.1.4 or later of the libvirt daemon, you can set the following directive in /etc/libvirt/qemu.conf to specify what interfaces are used for the ephemeral migration ports:

migration_address = 

It is also recommended to configure the firewall on each OpenStack Compute node to only allow other Compute nodes to access the ports that are used for remote access to the libvirt daemon. By default, this is port 16514 for TLS and 16509 for unencrypted TCP.

Additionally, migration over ephemeral ports uses a port range of 49152-49215. You will need to allow your OpenStack Compute nodes to communicate with each other over these ports if you choose not to tunnel the migration traffic over the libvirt daemon remote access port.

You can check what ports you have configured for the libvirt daemon by looking at the following configuration directives:

tls_port (libvirtd.conf) tcp_port (libvirtd.conf) migration_port_min (qemu.conf) migration_port_max (qemu.conf)

If you are running a version of the libvirt daemon older than 1.1.4 and you want to perform live block migration, you will need to allow your OpenStack Compute nodes to communicate over port range 5900-49151. If you are running version 1.1.4 or later of the libvirt daemon, the regular ephemeral migration port range is used for live block migration.

Please consult the documentation for your firewall software for instructions on configuring the appropriate firewall rules.

Contacts / References

 * Author: Nathan Kinder, Red Hat
 * This OSSN: https://wiki.openstack.org/wiki/OSSN/OSSN-0007
 * Original LaunchPad Bug : https://bugs.launchpad.net/openstack-manuals/+bug/1287194
 * OpenStack Security ML : openstack-security@lists.openstack.org
 * OpenStack Security Group : https://launchpad.net/~openstack-ossg