Obsolete:LiveMigrationUsage

= Using live migration feature =

Overview of this feature

 * OS: Ubuntu lucid 10.04/10.10 (both instances and host).
 * Shared strorage: The NOVA-INST-DIR/instances directory of nova-computes have to be mounted same shared storage(tested using NFS)
 * Instances : Instance can be migrated with ISCSI/AoE based volumes.
 * Hypervisor: KVM with libvirt
 * (NOTE1:) "NOVA-INST-DIR/instance" is expected that vm image is put on to. see "flags.instances_path" in nova.compute.manager for the default value
 * (NOTE2:) this feature is admin only, since nova-manage is necessary.

Sample Nova Installation before starting

 * Prepare 3 Ubuntu lucid 10.04/10.10 host at least, lets say, HostA, HostB, HostC
 * nova-api/nova-network/nova-volume/nova-objectstore/nova-scheduler(and other daemon) are running on HostA.
 * nova-compute is running on both HostB and HostC.
 * All nova daemon runs as root.
 * HostA export NOVA-INST-DIR/instances, HostB and HostC mount it.
 * To avoid any confusion, NOVA-INST-DIR is same at HostA/HostB/HostC("NOVA-INST-DIR" shows top of install dir).
 * HostA export NOVA-INST-DIR/instances, HostB and HostC mount it.
 * detail description is below.

Pre-requisite settings
(a) /etc/hosts setting at HostA/HostB/HostC

Make sure 3 Hosts can name-resolution with each other. ping with each other is better way to test.

ping HostA ping HostB ping HostC

(b) NFS settings at HostA. add the below desctiption to /etc/exports.

NOVA-INST-DIR/instances HostA/255.255.0.0(rw,sync,fsid=0,no_root_squash

Change "255.255.0.0" appropriate netmask, which should include HostB/HostC. Then restart nfs server.

/etc/init.d/nfs-kernel-server restart /etc/init.d/idmapd restart

Also, at any compute nodes, add below line to /etc/fstab:

HostA:/ DIR nfs4 defaults 0 0

Then try to mount at compute node. Check exported directory is successfully mounted.

mount -a -v

If fail, try this at any hosts.

iptables -F

Also, check file/daemon permissions. We expect any nova daemons are running as root.

root@openstack2-api:/opt/nova-2010.4# ps -ef | grep nova root 5948 5904 9 11:29 pts/4 00:00:00 python /opt/nova-2010.4//bin/nova-api root 5952 5908 6 11:29 pts/5 00:00:00 python /opt/nova-2010.4//bin/nova-objectstore ... (snip)

"NOVA-INST-DIR/instances/" directory can be seen at HostA:

root@openstack:~# ls -ld NOVA-INST-DIR/instances/ drwxr-xr-x 2 root root 4096 2010-12-07 14:34 nova-install-dir/instances/

Also, at HostB and HostC:

drwxr-xr-x 2 root root 4096 2010-12-07 14:34 nova-install-dir/instances/
 * 1) ls -ld NOVA-INST-DIR/instances/

Filesystem          1K-blocks      Used Available Use% Mounted on /dev/sda1            921514972   4180880 870523828   1% / none                 16498340      1228  16497112   1% /dev none                 16502856         0  16502856   0% /dev/shm none                 16502856       368  16502488   1% /var/run none                 16502856         0  16502856   0% /var/lock none                 16502856         0  16502856   0% /lib/init/rw HostA:       921515008 101921792 772783104  12% /opt  ( <--- this line is important.)
 * 1) df -k

(b) libvirt settings (Necessary to use simple tcp (qemu+tcp://)).

Modify /etc/libvirt/libvirt.conf :

before : #listen_tls = 0 after : listen_tls = 0

before : #listen_tcp = 1 after : listen_tcp = 1

Adding following line to /etc/libvirt/libvirt.conf :

auth_tcp = "none"

Modify /etc/init/libvirt-bin.conf

before : exec /usr/sbin/libvirtd -d after : exec /usr/sbin/libvirtd -d -l

Modify modify /etc/default/libvirt-bin

before :libvirtd_opts=" -d" after :libvirtd_opts=" -d -l"

then, restart libvirt


 * 1) stop libvirt-bin && start libvirt-bin ps -ef | grep libvirt

Make sure you get the below result.

root 1145 1 0 Nov27 ? 00:00:03 /usr/sbin/libvirtd -d -l
 * 1) /opt/nova-2010.2# ps -ef | grep libvirt

if you would like to use qemu+ssh, change "live_migration_uri" flag at nova.vir.libvirt_conn and make appropriate settings described at http://libvirt.org/.

Usage
If a physical server is going to under maintenance, he may want to know any instances running onto it.

root@openstack2-api:/opt/live-migration# euca-describe-instances Reservation:r-2raqmabo RESERVATION  r-2raqmabo    admin   default INSTANCE     i-00000003    ami-ubuntu-lucid      a.b.c.d  e.f.g.h      running     testkey (admin, HostB)  0             m1.small     2011-02-15 07:28:32   nova
 * 1) euca-describe-instance

"HostB" is physical server name. then he know which instance should be migrated.

Next, he may look for the destination host that instances are migrated to.

HostA nova-scheduler enabled :-) None HostA nova-volume enabled  :-) None HostA nova-network enabled :-) None HostB nova-compute enabled  :-) None HostC nova-compute enabled :-) None
 * 1) nova-manage service list

Now he knows HostB and HostC is available.

but HostC may be lack of resource(cpu/memory/hdd..) so, checking.

HOST            PROJECT     cpu   mem(mb) disk(gb) HostC(total)                 16     32232     878 HostC(used)                  13     21284     442 HostC           p1            5     10240     150 HostC           p2            5     10240     150 .....
 * 1) nova-manage service updateresource HostC
 * 2) nova-manage service describeresource HostC

Remember that updateresource first, then describeresource. Otherwise, Host(used) is not updated.


 * cpu : the number of vcpu
 * memory_mb : total amount of memory (MB)
 * memory_mb : total amount of NOVA-INST-DIR/instances(GB)
 * 1st list shows total amount of resource physical server has.
 * 2nd list shows current used resource.
 * 3rd line and under is used resource per project.

Remember that any value of 1st and 2nd line is obtained from os/libvirt. But 3rd line sums the value of instance_types. This means instances "dont use" resources described 3rd line, just "can use at maximum".

Now, admin checks Host(used) line at first, if there is enough resource available, HostC is good candidate for live migration destination. But if sums HostC-p1 and HostC-p2 line is over HostC(total), HostC is not good candidate.

OK, now its the time to live migration.

Migration of i-00000001 initiated. Check its progress using euca-describe-instances.
 * 1) nova-manage instance live_migration i-00000003 HostC

Few seconds later, confirm HostB->HostC below.

root@openstack2-api:/opt/live-migration# euca-describe-instances Reservation:r-2raqmabo RESERVATION  r-2raqmabo    admin   default INSTANCE     i-00000003    ami-ubuntu-lucid      a.b.c.d  e.f.g.h      running     testkey (admin, HostC)  0             m1.small     2011-02-15 07:28:32   nova
 * 1) euca-describe-instance

If that still shows HostB, maybe some problem occurs. see Trouble shooting section.

Trouble shooting
When live migration fails somehow, error messages are shown at:


 * scheduler logfile
 * source compute node logfile
 * dest compute node logfile