Using live migration feature

Environments

OS: Ubuntu lucid 10.04/10.10 for both instances and host.
NFS: The NOVA-INST-DIR/instances directory of nova-computes have to be mounted same shared storage(tested using NFS)

  "NOVA-INST-DIR/instance" is expected that vm image is put on to. see nova.compute.manager for the default value)

Instances : Instance can be migrated with ISCSI/AoE based volumes.
Hypervisor: KVM(with libvirt( tested version of Ubuntu10.04/10.10)

Sample Nova Installation

Prepare 3 Ubuntu lucid 10.04/10.10 host at least, lets say, HostA, HostB, HostC
nova-api/nova-network/nova-volume/nova-objectstore/nova-scheduler(and other daemon) are running on HostA.
nova-compute is running on both HostB and HostC.
NOVA-INST-DIR is same at HostA/HostB/HostC. ("NOVA-INST-DIR" shows top of install dir)
HostA export NOVA-INST-DIR/instances, HostB and HostC mount it.
detail description is below.

Pre-requisite settings

(a) /etc/hosts setting at HostA/HostB/HostC

Make sure 3 Hosts can name-resolution with each other. ping with each other is better way to test.

ping HostA
ping HostB
ping HostC

(b) NFS settings at HostA. add the below desctiption to /etc/exports.

NOVA-INST-DIR/instances HostA/255.255.0.0(rw,sync,fsid=0,no_root_squash

Change "255.255.0.0" appropriate netmask, which should include HostB/HostC. Then restart nfs server.

/etc/init.d/nfs-kernel-server restart
/etc/init.d/idmapd restart

Also, at any compute nodes, add below line to /etc/fstab:

HostA:/ DIR nfs4 defaults 0 0

where a.b.c.d is nfs server's ipaddress and "DIR" must be same as 'instances_path'( seenova.compute.manager for the default value)

Then try to mount at compute node. Check exported directory is successfully mounted.

mount -a -v

If fail, try this at any hosts.

iptables -F

Also, check file/daemon permissions. We expect any nova daemons are running as root.

root@openstack2-api:/opt/nova-2010.4# ps -ef | grep nova
root 5948 5904 9 11:29 pts/4 00:00:00 python /opt/nova-2010.4//bin/nova-api
root 5952 5908 6 11:29 pts/5 00:00:00 python /opt/nova-2010.4//bin/nova-objectstore
... (snip)

"instances/" directory can be seen from server side:

root@openstack:~# ls -ld nova-install-dir/instances/
drwxr-xr-x 2 root root 4096 2010-12-07 14:34 nova-install-dir/instances/

Also, client(compute node) side:

# ls -ld nova-install-dir/instances/
drwxr-xr-x 2 root root 4096 2010-12-07 14:34 nova-install-dir/instances/

(b) libvirt settings In default configuration, this feature use simple tcp protocol(qemu+tcp://).To use this protocol, below configuration is necessary.

Modify /etc/libvirt/libvirt.conf :

before : #listen_tls = 0
after : listen_tls = 0

before : #listen_tcp = 1
after : listen_tcp = 1

Adding following line to /etc/libvirt/libvirt.conf :

auth_tcp = "none"

Modify /etc/init/libvirt-bin.conf

 before : exec /usr/sbin/libvirtd -d
 after : exec /usr/sbin/libvirtd -d -l

Modify modify /etc/default/libvirt-bin

before :libvirtd_opts=" -d"
after :libvirtd_opts=" -d -l"

then, restart libvirt

stop libvirt-bin && start libvirt-bin ps -ef | grep libvirt

Make sure you get the below result.

root@openstack2:/opt/nova-2010.2# ps -ef | grep libvirt
root 1145 1 0 Nov27 ? 00:00:03 /usr/sbin/libvirtd -d -l

if you would like to use qemu+ssh or other protocol, change "live_migration_uri" flag. by adding "--live_migration_uri" to /etc/nova/nova.conf and make appropriate settings described at http://libvirt.org/.

Usage

To get a list of physical hosts,
- nova-manage host list
To get a available pysical resource of each host,
- nova-manage host shows hostname
- An example result is below. The 1st line shows total amount of resource that the specified host has. The 2nd and latter lines show usage resource per project. This command is created because admins can decide which host should be a destination of live migration.

HOST PROJECT cpu mem(mb) disk(gb)
openstack2-c2 16 32232 878 > openstack2-c2 admin 1 2048 20

For live migration,
- nova-manage instances live_migration ec2-id(i-xxxx) destination-host-name.
- Once this command is executed, admins will check the status through euca-describe-instances. The status is changed from 'running' to 'migrating', and changed to 'running' when live migration finishes. Note that it depends on an environment how long it takes to live migration finishes. If it finishes too fast, admins cannot see 'migrating' status.

# euca-describe-instances
Reservation:r-qlg3favp
RESERVATION r-qlg3favp adminINSTANCE i-2ah453 ami-tiny 12.1.0.14 10.0.0.3
migrating testkey (admin, openstack2-c2) 0 m1.small 2010-11-28 16:09:16

When live migration finishes successfully, admin can check the part of euca-describe-instances which shows physical node information. ( only when euca-describe-instances is executed by admin user ) Admins also can check live migration source compute node logfile which may show a log.

Live migration i-xxxx to DESTHOST finishes successfully.

Trouble shooting

When live migration fails somehow, error messages are shown at:
- scheduler logfile
- source compute node logfile
- dest compute node logfile

Obsolete:LiveMigrationUsage