XenServer/Development

XenServer Development

#!wiki red/solid
Note: you can now use DevStack to get started with XenServer.
For more details see: [[XenXCPAndXenServer]]

<<TableOfContents()>>

Getting Started Notes

XenServer requires a nova-compute domU to run on the hypervisor in order to inject images into the VHDs. Nova-compute is the worker for the Hypervisor. It pulls jobs and applies them to the hypervisor it resides on. It's required to run on the hypervisor because it utilizes /sys/hypervisor.

Prepare Nova

Install dependencies

sudo apt-get install python-software-properties
sudo add-apt-repository ppa:nova-core/trunk
sudo apt-get update

sudo apt-get install git-core mysql-server build-essential rabbitmq-server unzip swig screen parted curl euca2ools python-pip python-dev python-mysqldb libxml2 libxslt-dev

sudo pip install xenapi

Setup development branch

mkdir ~/openstack && cd ~/openstack

git clone https://github.com/openstack/nova.git
sudo pip install -r nova/tools/pip-requires

You may run into issues when when running the sudo pip install -r nova/tools/pip-requires command with an error similar to this on Ubuntu 10.04.2 LTS:

<backtrace>
VersionConflict: (M2Crypto 0.20.1 (/usr/lib/pymodules/python2.6), Requirement.parse('M2Crypto==0.20.2'))

Storing complete log in ./pip-log.txt

This is because euca2ools and cloud-utils install an older version of M2Crypto. You can solve this by installing the required version:

wget http://repo.vexxhost.com/openstack/m2crypto/python-m2crypto_0.20.2-1_amd64.deb
sudo dpkg -i python-m2crypto_0.20.2-1_amd64.deb

You can rerun the sudo pip install -r nova/tools/pip-requires command and it will install with no problems afterwards

Install Glance

Note: The glance-scrubber.conf file may be missing from some packages, refer to https://bugs.launchpad.net/ubuntu/+source/glance/+bug/816972.

https://github.com/openstack/glance.git
cd glance
sudo cp etc/glance-*.conf ~
# set filesystem_store_datadir to somewhere writable (/home/name/openstack/images/glance perhaps) in glance-api.conf
# set sql_connection to MySQL (mysql://root:password@127.0.0.1/glance) in glance-registry.conf and glance-scrubber.conf
sudo python setup.py install

Setup MySQL

mysql -u root -p -e "create database nova; create database glance;"

Configure Nova flags

cd nova

# xenapi_inject_image == false if using an ovf type glance file.  otherwise set to true to inject the bits into the image

cat > ../nova.conf << EOF
--verbose
--nodaemon
--sql_connection=mysql://root:<password>@127.0.0.1/nova
--network_manager=nova.network.manager.FlatManager
--image_service=nova.image.glance.GlanceImageService
--flat_network_bridge=xenbr0
--connection_type=xenapi
--xenapi_connection_url=https://<XenServer IP>
--xenapi_connection_username=root
--xenapi_connection_password=password
--reboot_timeout=600
--rescue_timeout=86400
--resize_confirm_window=86400
--allow_admin_api
--enabled_apis=osapi_compute
--allow_resize_to_same_host
--xenapi_generate_swap
--firewall_driver=nova.virt.xenapi.firewall.Dom0IptablesFirewallDriver
EOF

# Add the following to your flagfile if you're using Ubuntu Maverick
cat >> ../nova.conf << EOF
--xenapi_remap_vbd_dev
EOF

sudo mkdir /etc/nova
sudo ln -s ~/openstack/nova.conf /etc/nova/

Ensure tests pass

./run_tests.sh -x -N

Test XenAPI

./bin/nova-manage shell python
import XenAPI
import nova.virt.xenapi_conn
nova.virt.xenapi_conn.XenAPI = XenAPI
x = nova.virt.xenapi_conn.XenAPIConnection("https://<XenServer IP>", "root", "password")
x.list_instances()

Setup Nova

cd nova/nova/CA && ./genrootca.sh && cd ../..

NAME=<name>

./bin/nova-manage db sync
./bin/nova-manage user admin $NAME
./bin/nova-manage project create openstack $NAME
./bin/nova-manage network create private 10.0.0.0/8 1 64 [0 0 <IPv6 Prefix>::/<Mask> <IPv6 Gateway>]
./bin/nova-manage project zip openstack $NAME

unzip nova.zip
. novarc

Enable Legacy Auth

vim etc/nova/api-paste.ini
# set [pipeline:openstack_compute_api_v2]
# pipeline = faultwrap noauth ratelimit osapi_compute_app_v2
# to
# [pipeline:openstack_compute_api_v2]
# pipeline = faultwrap auth ratelimit osapi_compute_app_v2

Run Nova

screen  # each process gets its own window

./bin/nova-api
./bin/nova-compute
./bin/nova-network
./bin/nova-scheduler

Setup Glance

sudo mkdir /var/log/glance
sudo glance-manage db_sync

Run Glance

sudo glance-control all start

Prepare XenServer

Install Nova plugins

cd ~/openstack/nova/plugins/xenserver/xenapi 
scp -r etc/ root@<XenServer IP>:/

ssh root@<XenServer IP>

chmod a+x /etc/xapi.d/plugins/*
sed -i -e "s/enabled=0/enabled=1/" /etc/yum.repos.d/CentOS-Base.repo
yum install parted
mkdir /boot/guest
sed -i -e "s/enabled=1/enabled=0/" /etc/yum.repos.d/CentOS-Base.repo

Configure SR storage

ssh root@<XenServer IP> xe sr-list  # look for the 'Local storage' SR
ssh root@<XenServer IP> xe sr-param-set uuid=<SR uuid> other-config:i18n-key=local-storage

<Provisional message: placeholder update when Nova E4 releases> From Nova Essex 4, a new flag called 'sr_matching_filter' has been introduced (which defaults to 'other-config:i18n-key=local-storage'). This filter is used for finding the SR on which to install guest instances. The default value is the Local Storage in default XenServer/XCP installations as detailed above. However, if you want to select an SR with different matching criteria, this flag can be set to 'other-config:my_favorite_sr=true'. This means that XenAPI will look for an SR whose other-config contains the key-value pair (my_favorite_sr, true). On the other hand, to fall back on the Default SR, as displayed by XenCenter and as returned by xenapi.pool.get_default_SR, this flag can be set to to 'default-sr:true'.

Instance Management

Prepare Images

mkdir -p ~/openstack/images && cd ~/openstack/images
mkdir glance

AMI Images

mkdir lucid_ami && cd lucid_ami

wget -q -O - http://173.203.107.207/ubuntu-lucid.tar | tar xSv

glance add name=ramdisk disk_format=ari container_format=ari is_public=True < initrd.img-2.6.32-23-server
glance add name=kernel disk_format=aki container_format=aki is_public=True < vmlinuz-2.6.32-23-server
glance add name=lucid_ami disk_format=ami container_format=ami is_public=True ramdisk_id=<ramdisk ID> kernel_id=<kernel ID> < ubuntu-lucid.img

OVF Images

Linux

mkdir linux_ovf && cd linux_ovf

glance add name=lucid_ovf disk_format=vhd container_format=ovf is_public=True < lucid.ova

Windows

mkdir windows && cd windows

glance add name=windows disk_format=vhd container_format=ovf type=raw os_type=windows is_public=True < windows.ova

Run Instance

ssh-keygen -f nova_key && chmod 600 nova_key
euca-add-keypair nova_key > nova_key.priv

nova flavor-list
nova image-list
nova boot test --flavor <flavor ID> --image <image ID>
nova list

Connect to instance console

VNC

ssh root@<XenServer IP> netstat -tuanp | grep vncterm
ssh -L 590N:localhost:590N root@<XenServer IP>
vnc://localhost:590N

XenConsole

ssh root@<XenServer IP>
list_domains
ps ax | grep vnc | grep <domain ID>
kill -9 <VNC PIDs>
/usr/lib/xen/bin/xenconsole <domain ID>

Install XenTools

Make xs-tools.iso available to the instance in XenCenter, OpenXenManager or the CLI (not documented here) before proceeding.

ssh root@<instance IP>
mount /dev/xvdb /mnt
cd /mnt/Linux
dpkg -i ./xe-guest-utilities_5.5.0-464_amd64.deb
update-rc.d -f xe-linux-distribution remove
update-rc.d xe-linux-distribution defaults
reboot

Install OpenXenManager on OS X

#!wiki note
The rev48 "packaged" tarball [[http://sourceforge.net/projects/openxenmanager/forums/forum/1159068/topic/4529278|may be broken]] -- SVN trunk worked for me, with the Homebrew dependency installation method I've documented below. ''--ChesMartin''

1. Download: http://xensemaking.com/openxenmanager/
2. Compile pygtk:

sudo port install py26-gtk

If you're a Homebrew user, you're unfortunately in for a more work due in part to some dependencies not being available as formula yet. You can try this procedure for building PyGTK and dependencies.

1.#3 Open a X11 shell and go to the openxenmanager directory

1. Run OpenXenManager:

python window.py

The project README also mentions python-gtk-vnc and rrdtool for graphs -- neither path here takes those into account so you'll need to look into them if you wish to have those features working.